1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ============================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ============================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU 133 - ``verde`` 134 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU 135 - ``oland`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 ``gfx705`` ``amdgcn`` APU *TBA* 170 171 .. TODO:: 172 173 Add product 174 names. 175 176 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 177 ----------------------------------------------------------------------------------------------------------- 178 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 179 [on] - Pro A6-8500B 180 - A8-8600P 181 - Pro A8-8600B 182 - FX-8800P 183 - Pro A12-8800B 184 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 185 [on] - Pro A10-8700B 186 - A10-8780P 187 \ ``amdgcn`` APU - xnack - A10-9600P 188 [on] - A10-9630P 189 - A12-9700P 190 - A12-9730P 191 - FX-9800P 192 - FX-9830P 193 \ ``amdgcn`` APU - xnack - E2-9010 194 [on] - A6-9210 195 - A9-9410 196 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - Radeon R285 197 - ``tonga`` [off] - Radeon R9 380 198 - Radeon R9 385 199 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 200 [off] - Radeon R9 Fury 201 - Radeon R9 FuryX 202 - Radeon Pro Duo 203 - FirePro S9300x2 204 - Radeon Instinct MI8 205 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 206 [off] - Radeon RX 480 207 - Radeon Instinct MI6 208 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 209 [off] 210 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 211 [off] - FirePro S7100 212 - FirePro W7100 213 - Mobile FirePro 214 M7170 215 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack *TBA* 216 [on] 217 .. TODO:: 218 219 Add product 220 names. 221 222 **GCN GFX9** [AMD-GCN-GFX9]_ 223 ----------------------------------------------------------------------------------------------------------- 224 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 225 [off] Frontier Edition 226 - Radeon RX Vega 56 227 - Radeon RX Vega 64 228 - Radeon RX Vega 64 229 Liquid 230 - Radeon Instinct MI25 231 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 232 [on] - Ryzen 5 2400G 233 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 234 [off] 235 .. TODO:: 236 237 Add product 238 names. 239 240 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 241 [off] - Radeon Instinct MI60 242 - sram-ecc - Radeon VII 243 [off] - Radeon Pro VII 244 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 245 [off] 246 - sram-ecc 247 [on] 248 .. TODO:: 249 250 Add product 251 names. 252 253 ``gfx909`` ``amdgcn`` APU - xnack *TBA* 254 [off] 255 .. TODO:: 256 257 Add product 258 names. 259 260 ``gfx90c`` ``amdgcn`` APU - xnack - Ryzen 7 4700G 261 [on] - Ryzen 7 4700GE 262 - Ryzen 7 4700G 263 - Ryzen 7 4700GE 264 - Ryzen 5 4600G 265 - Ryzen 5 4600GE 266 - Ryzen 3 4300G 267 - Ryzen 3 4300GE 268 - Ryzen Pro 4000G 269 - Ryzen 7 Pro 4700G 270 - Ryzen 7 Pro 4750GE 271 - Ryzen 5 Pro 4650G 272 - Ryzen 5 Pro 4650GE 273 - Ryzen 3 Pro 4350G 274 - Ryzen 3 Pro 4350GE 275 276 **GCN GFX10** [AMD-GCN-GFX10]_ 277 ----------------------------------------------------------------------------------------------------------- 278 ``gfx1010`` ``amdgcn`` dGPU - xnack - Radeon RX 5700 279 [off] - Radeon RX 5700 XT 280 - wavefrontsize64 - Radeon Pro 5600 XT 281 [off] - Radeon Pro 5600M 282 - cumode 283 [off] 284 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 285 [off] 286 - wavefrontsize64 287 [off] 288 - cumode 289 [off] 290 .. TODO:: 291 292 Add product 293 names. 294 295 ``gfx1012`` ``amdgcn`` dGPU - xnack - Radeon RX 5500 296 [off] - Radeon RX 5500 XT 297 - wavefrontsize64 298 [off] 299 - cumode 300 [off] 301 ``gfx1030`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 302 [off] 303 - cumode 304 [off] 305 .. TODO:: 306 307 Add product 308 names. 309 310 ``gfx1031`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 311 [off] 312 - cumode 313 [off] 314 .. TODO:: 315 316 Add product 317 names. 318 319 ``gfx1032`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 320 [off] 321 - cumode 322 [off] 323 .. TODO:: 324 Add product 325 names. 326 ``gfx1033`` ``amdgcn`` APU - wavefrontsize64 *TBA* 327 [off] 328 - cumode 329 [off] 330 .. TODO:: 331 Add product 332 names. 333 334 =========== =============== ============ ===== ============================= ======= ====================== 335 336.. _amdgpu-target-features: 337 338Target Features 339--------------- 340 341Target features control how code is generated to support certain 342processor specific features. Not all target features are supported by 343all processors. The runtime must ensure that the features supported by 344the device used to execute the code match the features enabled when 345generating the code. A mismatch of features may result in incorrect 346execution, or a reduction in performance. 347 348The target features supported by each processor, and the default value 349used if not specified explicitly, is listed in 350:ref:`amdgpu-processor-table`. 351 352Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 353target features. 354 355For example: 356 357``-mxnack`` 358 Enable the ``xnack`` feature. 359``-mno-xnack`` 360 Disable the ``xnack`` feature. 361 362 .. table:: AMDGPU Target Features 363 :name: amdgpu-target-feature-table 364 365 ====================== ================================================== 366 Target Feature Description 367 ====================== ================================================== 368 -m[no-]xnack Enable/disable generating code that has 369 memory clauses that are compatible with 370 having XNACK replay enabled. 371 372 This is used for demand paging and page 373 migration. If XNACK replay is enabled in 374 the device, then if a page fault occurs 375 the code may execute incorrectly if the 376 ``xnack`` feature is not enabled. Executing 377 code that has the feature enabled on a 378 device that does not have XNACK replay 379 enabled will execute correctly but may 380 be less performant than code with the 381 feature disabled. 382 383 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 384 ECC is enabled/disabled. 385 386 -m[no-]wavefrontsize64 Control the default wavefront size used when 387 generating code for kernels. When disabled 388 native wavefront size 32 is used, when enabled 389 wavefront size 64 is used. 390 391 -m[no-]cumode Control the default wavefront execution mode used 392 when generating code for kernels. When disabled 393 native WGP wavefront execution mode is used, 394 when enabled CU wavefront execution mode is used 395 (see :ref:`amdgpu-amdhsa-memory-model`). 396 ====================== ================================================== 397 398.. _amdgpu-address-spaces: 399 400Address Spaces 401-------------- 402 403The AMDGPU architecture supports a number of memory address spaces. The address 404space names use the OpenCL standard names, with some additions. 405 406The AMDGPU address spaces correspond to target architecture specific LLVM 407address space numbers used in LLVM IR. 408 409The AMDGPU address spaces are described in 410:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 411supported for the ``amdgcn`` target. 412 413 .. table:: AMDGPU Address Spaces 414 :name: amdgpu-address-spaces-table 415 416 ================================= =============== =========== ================ ======= ============================ 417 .. 64-Bit Process Address Space 418 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 419 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 420 Space Number Name Name Size 421 ================================= =============== =========== ================ ======= ============================ 422 Generic 0 flat flat 64 0x0000000000000000 423 Global 1 global global 64 0x0000000000000000 424 Region 2 N/A GDS 32 *not implemented for AMDHSA* 425 Local 3 group LDS 32 0xFFFFFFFF 426 Constant 4 constant *same as global* 64 0x0000000000000000 427 Private 5 private scratch 32 0xFFFFFFFF 428 Constant 32-bit 6 *TODO* 0x00000000 429 Buffer Fat Pointer (experimental) 7 *TODO* 430 ================================= =============== =========== ================ ======= ============================ 431 432**Generic** 433 The generic address space uses the hardware flat address support available in 434 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 435 local apertures), that are outside the range of addressable global memory, to 436 map from a flat address to a private or local address. 437 438 FLAT instructions can take a flat address and access global, private 439 (scratch), and group (LDS) memory depending on if the address is within one 440 of the aperture ranges. Flat access to scratch requires hardware aperture 441 setup and setup in the kernel prologue (see 442 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 443 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 444 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 445 446 To convert between a private or group address space address (termed a segment 447 address) and a flat address the base address of the corresponding aperture 448 can be used. For GFX7-GFX8 these are available in the 449 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 450 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 451 GFX9-GFX10 the aperture base addresses are directly available as inline 452 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 453 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 454 aligned to 2^32 which makes it easier to convert from flat to segment or 455 segment to flat. 456 457 A global address space address has the same value when used as a flat address 458 so no conversion is needed. 459 460**Global and Constant** 461 The global and constant address spaces both use global virtual addresses, 462 which are the same virtual address space used by the CPU. However, some 463 virtual addresses may only be accessible to the CPU, some only accessible 464 by the GPU, and some by both. 465 466 Using the constant address space indicates that the data will not change 467 during the execution of the kernel. This allows scalar read instructions to 468 be used. As the constant address space could only be modified on the host 469 side, a generic pointer loaded from the constant address space is safe to be 470 assumed as a global pointer since only the device global memory is visible 471 and managed on the host side. The vector and scalar L1 caches are invalidated 472 of volatile data before each kernel dispatch execution to allow constant 473 memory to change values between kernel dispatches. 474 475**Region** 476 The region address space uses the hardware Global Data Store (GDS). All 477 wavefronts executing on the same device will access the same memory for any 478 given region address. However, the same region address accessed by wavefronts 479 executing on different devices will access different memory. It is higher 480 performance than global memory. It is allocated by the runtime. The data 481 store (DS) instructions can be used to access it. 482 483**Local** 484 The local address space uses the hardware Local Data Store (LDS) which is 485 automatically allocated when the hardware creates the wavefronts of a 486 work-group, and freed when all the wavefronts of a work-group have 487 terminated. All wavefronts belonging to the same work-group will access the 488 same memory for any given local address. However, the same local address 489 accessed by wavefronts belonging to different work-groups will access 490 different memory. It is higher performance than global memory. The data store 491 (DS) instructions can be used to access it. 492 493**Private** 494 The private address space uses the hardware scratch memory support which 495 automatically allocates memory when it creates a wavefront and frees it when 496 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 497 given private address will be different to the memory accessed by another lane 498 of the same or different wavefront for the same private address. 499 500 If a kernel dispatch uses scratch, then the hardware allocates memory from a 501 pool of backing memory allocated by the runtime for each wavefront. The lanes 502 of the wavefront access this using dword (4 byte) interleaving. The mapping 503 used from private address to backing memory address is: 504 505 ``wavefront-scratch-base + 506 ((private-address / 4) * wavefront-size * 4) + 507 (wavefront-lane-id * 4) + (private-address % 4)`` 508 509 If each lane of a wavefront accesses the same private address, the 510 interleaving results in adjacent dwords being accessed and hence requires 511 fewer cache lines to be fetched. 512 513 There are different ways that the wavefront scratch base address is 514 determined by a wavefront (see 515 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 516 517 Scratch memory can be accessed in an interleaved manner using buffer 518 instructions with the scratch buffer descriptor and per wavefront scratch 519 offset, by the scratch instructions, or by flat instructions. Multi-dword 520 access is not supported except by flat and scratch instructions in 521 GFX9-GFX10. 522 523**Constant 32-bit** 524 *TODO* 525 526**Buffer Fat Pointer** 527 The buffer fat pointer is an experimental address space that is currently 528 unsupported in the backend. It exposes a non-integral pointer that is in 529 the future intended to support the modelling of 128-bit buffer descriptors 530 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 531 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 532 model the buffer descriptors used heavily in graphics workloads targeting 533 the backend. 534 535.. _amdgpu-memory-scopes: 536 537Memory Scopes 538------------- 539 540This section provides LLVM memory synchronization scopes supported by the AMDGPU 541backend memory model when the target triple OS is ``amdhsa`` (see 542:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 543 544The memory model supported is based on the HSA memory model [HSA]_ which is 545based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 546relation is transitive over the synchronizes-with relation independent of scope 547and synchronizes-with allows the memory scope instances to be inclusive (see 548table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 549 550This is different to the OpenCL [OpenCL]_ memory model which does not have scope 551inclusion and requires the memory scopes to exactly match. However, this 552is conservatively correct for OpenCL. 553 554 .. table:: AMDHSA LLVM Sync Scopes 555 :name: amdgpu-amdhsa-llvm-sync-scopes-table 556 557 ======================= =================================================== 558 LLVM Sync Scope Description 559 ======================= =================================================== 560 *none* The default: ``system``. 561 562 Synchronizes with, and participates in modification 563 and seq_cst total orderings with, other operations 564 (except image operations) for all address spaces 565 (except private, or generic that accesses private) 566 provided the other operation's sync scope is: 567 568 - ``system``. 569 - ``agent`` and executed by a thread on the same 570 agent. 571 - ``workgroup`` and executed by a thread in the 572 same work-group. 573 - ``wavefront`` and executed by a thread in the 574 same wavefront. 575 576 ``agent`` Synchronizes with, and participates in modification 577 and seq_cst total orderings with, other operations 578 (except image operations) for all address spaces 579 (except private, or generic that accesses private) 580 provided the other operation's sync scope is: 581 582 - ``system`` or ``agent`` and executed by a thread 583 on the same agent. 584 - ``workgroup`` and executed by a thread in the 585 same work-group. 586 - ``wavefront`` and executed by a thread in the 587 same wavefront. 588 589 ``workgroup`` Synchronizes with, and participates in modification 590 and seq_cst total orderings with, other operations 591 (except image operations) for all address spaces 592 (except private, or generic that accesses private) 593 provided the other operation's sync scope is: 594 595 - ``system``, ``agent`` or ``workgroup`` and 596 executed by a thread in the same work-group. 597 - ``wavefront`` and executed by a thread in the 598 same wavefront. 599 600 ``wavefront`` Synchronizes with, and participates in modification 601 and seq_cst total orderings with, other operations 602 (except image operations) for all address spaces 603 (except private, or generic that accesses private) 604 provided the other operation's sync scope is: 605 606 - ``system``, ``agent``, ``workgroup`` or 607 ``wavefront`` and executed by a thread in the 608 same wavefront. 609 610 ``singlethread`` Only synchronizes with and participates in 611 modification and seq_cst total orderings with, 612 other operations (except image operations) running 613 in the same thread for all address spaces (for 614 example, in signal handlers). 615 616 ``one-as`` Same as ``system`` but only synchronizes with other 617 operations within the same address space. 618 619 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 620 operations within the same address space. 621 622 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 623 other operations within the same address space. 624 625 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 626 other operations within the same address space. 627 628 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 629 other operations within the same address space. 630 ======================= =================================================== 631 632LLVM IR Intrinsics 633------------------ 634 635The AMDGPU backend implements the following LLVM IR intrinsics. 636 637*This section is WIP.* 638 639.. TODO:: 640 641 List AMDGPU intrinsics. 642 643LLVM IR Attributes 644------------------ 645 646The AMDGPU backend supports the following LLVM IR attributes. 647 648 .. table:: AMDGPU LLVM IR Attributes 649 :name: amdgpu-llvm-ir-attributes-table 650 651 ======================================= ========================================================== 652 LLVM Attribute Description 653 ======================================= ========================================================== 654 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 655 will be specified when the kernel is dispatched. Generated 656 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 657 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 658 argument block size for the implicit arguments. This 659 varies by OS and language (for OpenCL see 660 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 661 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 662 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 663 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 664 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 665 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 666 execution unit. Generated by the ``amdgpu_waves_per_eu`` 667 CLANG attribute [CLANG-ATTR]_. 668 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 669 mode register to be set on entry. Overrides the default for 670 the calling convention. 671 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 672 the mode register to be set on entry. Overrides the default 673 for the calling convention. 674 ======================================= ========================================================== 675 676.. _amdgpu-elf-code-object: 677 678ELF Code Object 679=============== 680 681The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 682can be linked by ``lld`` to produce a standard ELF shared code object which can 683be loaded and executed on an AMDGPU target. 684 685.. _amdgpu-elf-header: 686 687Header 688------ 689 690The AMDGPU backend uses the following ELF header: 691 692 .. table:: AMDGPU ELF Header 693 :name: amdgpu-elf-header-table 694 695 ========================== =============================== 696 Field Value 697 ========================== =============================== 698 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 699 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 700 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 701 - ``ELFOSABI_AMDGPU_HSA`` 702 - ``ELFOSABI_AMDGPU_PAL`` 703 - ``ELFOSABI_AMDGPU_MESA3D`` 704 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 705 - ``ELFABIVERSION_AMDGPU_PAL`` 706 - ``ELFABIVERSION_AMDGPU_MESA3D`` 707 ``e_type`` - ``ET_REL`` 708 - ``ET_DYN`` 709 ``e_machine`` ``EM_AMDGPU`` 710 ``e_entry`` 0 711 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 712 ========================== =============================== 713 714.. 715 716 .. table:: AMDGPU ELF Header Enumeration Values 717 :name: amdgpu-elf-header-enumeration-values-table 718 719 =============================== ===== 720 Name Value 721 =============================== ===== 722 ``EM_AMDGPU`` 224 723 ``ELFOSABI_NONE`` 0 724 ``ELFOSABI_AMDGPU_HSA`` 64 725 ``ELFOSABI_AMDGPU_PAL`` 65 726 ``ELFOSABI_AMDGPU_MESA3D`` 66 727 ``ELFABIVERSION_AMDGPU_HSA`` 1 728 ``ELFABIVERSION_AMDGPU_PAL`` 0 729 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 730 =============================== ===== 731 732``e_ident[EI_CLASS]`` 733 The ELF class is: 734 735 * ``ELFCLASS32`` for ``r600`` architecture. 736 737 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 738 process address space applications. 739 740``e_ident[EI_DATA]`` 741 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 742 743``e_ident[EI_OSABI]`` 744 One of the following AMDGPU target architecture specific OS ABIs 745 (see :ref:`amdgpu-os-table`): 746 747 * ``ELFOSABI_NONE`` for *unknown* OS. 748 749 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 750 751 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 752 753 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 754 755``e_ident[EI_ABIVERSION]`` 756 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 757 object conforms: 758 759 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 760 runtime ABI. 761 762 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 763 runtime ABI. 764 765 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 766 3D runtime ABI. 767 768``e_type`` 769 Can be one of the following values: 770 771 772 ``ET_REL`` 773 The type produced by the AMDGPU backend compiler as it is relocatable code 774 object. 775 776 ``ET_DYN`` 777 The type produced by the linker as it is a shared code object. 778 779 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 780 781``e_machine`` 782 The value ``EM_AMDGPU`` is used for the machine for all processors supported 783 by the ``r600`` and ``amdgcn`` architectures (see 784 :ref:`amdgpu-processor-table`). The specific processor is specified in the 785 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 786 :ref:`amdgpu-elf-header-e_flags-table`). 787 788``e_entry`` 789 The entry point is 0 as the entry points for individual kernels must be 790 selected in order to invoke them through AQL packets. 791 792``e_flags`` 793 The AMDGPU backend uses the following ELF header flags: 794 795 .. table:: AMDGPU ELF Header ``e_flags`` 796 :name: amdgpu-elf-header-e_flags-table 797 798 ================================= ========== ============================= 799 Name Value Description 800 ================================= ========== ============================= 801 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 802 -------------------------------------------- ----------------------------- 803 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 804 mask for 805 ``EF_AMDGPU_MACH_xxx`` values 806 defined in 807 :ref:`amdgpu-ef-amdgpu-mach-table`. 808 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 809 target feature is 810 enabled for all code 811 contained in the code object. 812 If the processor 813 does not support the 814 ``xnack`` target 815 feature then must 816 be 0. 817 See 818 :ref:`amdgpu-target-features`. 819 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 820 target feature is 821 enabled for all code 822 contained in the code object. 823 If the processor 824 does not support the 825 ``sram-ecc`` target 826 feature then must 827 be 0. 828 See 829 :ref:`amdgpu-target-features`. 830 ================================= ========== ============================= 831 832 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 833 :name: amdgpu-ef-amdgpu-mach-table 834 835 ==================================== ========== ============================= 836 Name Value Description (see 837 :ref:`amdgpu-processor-table`) 838 ==================================== ========== ============================= 839 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 840 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 841 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 842 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 843 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 844 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 845 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 846 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 847 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 848 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 849 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 850 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 851 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 852 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 853 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 854 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 855 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 856 *reserved* 0x011 - Reserved for ``r600`` 857 0x01f architecture processors. 858 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 859 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 860 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 861 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 862 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 863 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 864 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 865 *reserved* 0x027 Reserved. 866 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 867 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 868 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 869 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 870 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 871 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 872 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 873 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 874 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 875 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 876 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 877 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 878 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 879 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 880 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 881 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 882 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 883 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 884 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 885 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 886 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 887 ==================================== ========== ============================= 888 889Sections 890-------- 891 892An AMDGPU target ELF code object has the standard ELF sections which include: 893 894 .. table:: AMDGPU ELF Sections 895 :name: amdgpu-elf-sections-table 896 897 ================== ================ ================================= 898 Name Type Attributes 899 ================== ================ ================================= 900 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 901 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 902 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 903 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 904 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 905 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 906 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 907 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 908 ``.note`` ``SHT_NOTE`` *none* 909 ``.rela``\ *name* ``SHT_RELA`` *none* 910 ``.rela.dyn`` ``SHT_RELA`` *none* 911 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 912 ``.shstrtab`` ``SHT_STRTAB`` *none* 913 ``.strtab`` ``SHT_STRTAB`` *none* 914 ``.symtab`` ``SHT_SYMTAB`` *none* 915 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 916 ================== ================ ================================= 917 918These sections have their standard meanings (see [ELF]_) and are only generated 919if needed. 920 921``.debug``\ *\** 922 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 923 information on the DWARF produced by the AMDGPU backend. 924 925``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 926 The standard sections used by a dynamic loader. 927 928``.note`` 929 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 930 backend. 931 932``.rela``\ *name*, ``.rela.dyn`` 933 For relocatable code objects, *name* is the name of the section that the 934 relocation records apply. For example, ``.rela.text`` is the section name for 935 relocation records associated with the ``.text`` section. 936 937 For linked shared code objects, ``.rela.dyn`` contains all the relocation 938 records from each of the relocatable code object's ``.rela``\ *name* sections. 939 940 See :ref:`amdgpu-relocation-records` for the relocation records supported by 941 the AMDGPU backend. 942 943``.text`` 944 The executable machine code for the kernels and functions they call. Generated 945 as position independent code. See :ref:`amdgpu-code-conventions` for 946 information on conventions used in the isa generation. 947 948.. _amdgpu-note-records: 949 950Note Records 951------------ 952 953The AMDGPU backend code object contains ELF note records in the ``.note`` 954section. The set of generated notes and their semantics depend on the code 955object version; see :ref:`amdgpu-note-records-v2` and 956:ref:`amdgpu-note-records-v3`. 957 958As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 959must be generated after the ``name`` field to ensure the ``desc`` field is 4 960byte aligned. In addition, minimal zero-byte padding must be generated to 961ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 962field of the ``.note`` section must be at least 4 to indicate at least 8 byte 963alignment. 964 965.. _amdgpu-note-records-v2: 966 967Code Object V2 Note Records (--amdhsa-code-object-version=2) 968~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 969 970.. warning:: Code Object V2 is not the default code object version emitted by 971 this version of LLVM. For a description of the notes generated with the 972 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 973 974The AMDGPU backend code object uses the following ELF note record in the 975``.note`` section when compiling for Code Object V2 976(--amdhsa-code-object-version=2). 977 978Additional note records may be present, but any which are not documented here 979are deprecated and should not be used. 980 981 .. table:: AMDGPU Code Object V2 ELF Note Records 982 :name: amdgpu-elf-note-records-table-v2 983 984 ===== ============================== ====================================== 985 Name Type Description 986 ===== ============================== ====================================== 987 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 988 ===== ============================== ====================================== 989 990.. 991 992 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 993 :name: amdgpu-elf-note-record-enumeration-values-table-v2 994 995 ============================== ===== 996 Name Value 997 ============================== ===== 998 *reserved* 0-9 999 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 1000 *reserved* 11 1001 ============================== ===== 1002 1003``NT_AMD_AMDGPU_HSA_METADATA`` 1004 Specifies extensible metadata associated with the code objects executed on HSA 1005 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 1006 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1007 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 1008 object metadata string. 1009 1010.. _amdgpu-note-records-v3: 1011 1012Code Object V3 Note Records (--amdhsa-code-object-version=3) 1013~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1014 1015The AMDGPU backend code object uses the following ELF note record in the 1016``.note`` section when compiling for Code Object V3 1017(--amdhsa-code-object-version=3). 1018 1019Additional note records may be present, but any which are not documented here 1020are deprecated and should not be used. 1021 1022 .. table:: AMDGPU Code Object V3 ELF Note Records 1023 :name: amdgpu-elf-note-records-table-v3 1024 1025 ======== ============================== ====================================== 1026 Name Type Description 1027 ======== ============================== ====================================== 1028 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1029 binary format. 1030 ======== ============================== ====================================== 1031 1032.. 1033 1034 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 1035 :name: amdgpu-elf-note-record-enumeration-values-table-v3 1036 1037 ============================== ===== 1038 Name Value 1039 ============================== ===== 1040 *reserved* 0-31 1041 ``NT_AMDGPU_METADATA`` 32 1042 ============================== ===== 1043 1044``NT_AMDGPU_METADATA`` 1045 Specifies extensible metadata associated with an AMDGPU code 1046 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 1047 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 1048 map keys defined for the ``amdhsa`` OS. 1049 1050.. _amdgpu-symbols: 1051 1052Symbols 1053------- 1054 1055Symbols include the following: 1056 1057 .. table:: AMDGPU ELF Symbols 1058 :name: amdgpu-elf-symbols-table 1059 1060 ===================== ================== ================ ================== 1061 Name Type Section Description 1062 ===================== ================== ================ ================== 1063 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1064 - ``.rodata`` 1065 - ``.bss`` 1066 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1067 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1068 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1069 ===================== ================== ================ ================== 1070 1071Global variable 1072 Global variables both used and defined by the compilation unit. 1073 1074 If the symbol is defined in the compilation unit then it is allocated in the 1075 appropriate section according to if it has initialized data or is readonly. 1076 1077 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1078 will resolve relocations using the definition provided by another code object 1079 or explicitly defined by the runtime. 1080 1081 If the symbol resides in local/group memory (LDS) then its section is the 1082 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1083 ``st_value`` field describes alignment requirements as it does for common 1084 symbols. 1085 1086 .. TODO:: 1087 1088 Add description of linked shared object symbols. Seems undefined symbols 1089 are marked as STT_NOTYPE. 1090 1091Kernel descriptor 1092 Every HSA kernel has an associated kernel descriptor. It is the address of the 1093 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1094 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1095 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1096 1097Kernel entry point 1098 Every HSA kernel also has a symbol for its machine code entry point. 1099 1100.. _amdgpu-relocation-records: 1101 1102Relocation Records 1103------------------ 1104 1105AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1106relocatable fields are: 1107 1108``word32`` 1109 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1110 alignment. These values use the same byte order as other word values in the 1111 AMDGPU architecture. 1112 1113``word64`` 1114 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1115 alignment. These values use the same byte order as other word values in the 1116 AMDGPU architecture. 1117 1118Following notations are used for specifying relocation calculations: 1119 1120**A** 1121 Represents the addend used to compute the value of the relocatable field. 1122 1123**G** 1124 Represents the offset into the global offset table at which the relocation 1125 entry's symbol will reside during execution. 1126 1127**GOT** 1128 Represents the address of the global offset table. 1129 1130**P** 1131 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1132 of the storage unit being relocated (computed using ``r_offset``). 1133 1134**S** 1135 Represents the value of the symbol whose index resides in the relocation 1136 entry. Relocations not using this must specify a symbol index of 1137 ``STN_UNDEF``. 1138 1139**B** 1140 Represents the base address of a loaded executable or shared object which is 1141 the difference between the ELF address and the actual load address. 1142 Relocations using this are only valid in executable or shared objects. 1143 1144The following relocation types are supported: 1145 1146 .. table:: AMDGPU ELF Relocation Records 1147 :name: amdgpu-elf-relocation-records-table 1148 1149 ========================== ======= ===== ========== ============================== 1150 Relocation Type Kind Value Field Calculation 1151 ========================== ======= ===== ========== ============================== 1152 ``R_AMDGPU_NONE`` 0 *none* *none* 1153 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1154 Dynamic 1155 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1156 Dynamic 1157 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1158 Dynamic 1159 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1160 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1161 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1162 Dynamic 1163 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1164 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1165 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1166 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1167 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1168 *reserved* 12 1169 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1170 ========================== ======= ===== ========== ============================== 1171 1172``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1173the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1174 1175There is no current OS loader support for 32-bit programs and so 1176``R_AMDGPU_ABS32`` is not used. 1177 1178.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1179 1180Loaded Code Object Path Uniform Resource Identifier (URI) 1181--------------------------------------------------------- 1182 1183The AMD GPU code object loader represents the path of the ELF shared object from 1184which the code object was loaded as a textual Unifom Resource Identifier (URI). 1185Note that the code object is the in memory loaded relocated form of the ELF 1186shared object. Multiple code objects may be loaded at different memory 1187addresses in the same process from the same ELF shared object. 1188 1189The loaded code object path URI syntax is defined by the following BNF syntax: 1190 1191.. code:: 1192 1193 code_object_uri ::== file_uri | memory_uri 1194 file_uri ::== "file://" file_path [ range_specifier ] 1195 memory_uri ::== "memory://" process_id range_specifier 1196 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1197 file_path ::== URI_ENCODED_OS_FILE_PATH 1198 process_id ::== DECIMAL_NUMBER 1199 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1200 1201**number** 1202 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1203 and octal values by "0". 1204 1205**file_path** 1206 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1207 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1208 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1209 the path are separated by "/". 1210 1211**offset** 1212 Is a 0-based byte offset to the start of the code object. For a file URI, it 1213 is from the start of the file specified by the ``file_path``, and if omitted 1214 defaults to 0. For a memory URI, it is the memory address and is required. 1215 1216**size** 1217 Is the number of bytes in the code object. For a file URI, if omitted it 1218 defaults to the size of the file. It is required for a memory URI. 1219 1220**process_id** 1221 Is the identity of the process owning the memory. For Linux it is the C 1222 unsigned integral decimal literal for the process ID (PID). 1223 1224For example: 1225 1226.. code:: 1227 1228 file:///dir1/dir2/file1 1229 file:///dir3/dir4/file2#offset=0x2000&size=3000 1230 memory://1234#offset=0x20000&size=3000 1231 1232.. _amdgpu-dwarf-debug-information: 1233 1234DWARF Debug Information 1235======================= 1236 1237.. warning:: 1238 1239 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1240 is not currently fully implemented and is subject to change. 1241 1242AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1243:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1244object executable code and data to the source language constructs. It can be 1245used by tools such as debuggers and profilers. It uses features defined in 1246:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1247DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1248 1249This section defines the AMDGPU target architecture specific DWARF mappings. 1250 1251.. _amdgpu-dwarf-register-identifier: 1252 1253Register Identifier 1254------------------- 1255 1256This section defines the AMDGPU target architecture register numbers used in 1257DWARF operation expressions (see DWARF Version 5 section 2.5 and 1258:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1259instructions (see DWARF Version 5 section 6.4 and 1260:ref:`amdgpu-dwarf-call-frame-information`). 1261 1262A single code object can contain code for kernels that have different wavefront 1263sizes. The vector registers and some scalar registers are based on the wavefront 1264size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1265simplifies the consumer of the DWARF so that each register has a fixed size, 1266rather than being dynamic according to the wavefront size mode. Similarly, 1267distinct DWARF registers are defined for those registers that vary in size 1268according to the process address size. This allows a consumer to treat a 1269specific AMDGPU processor as a single architecture regardless of how it is 1270configured at run time. The compiler explicitly specifies the DWARF registers 1271that match the mode in which the code it is generating will be executed. 1272 1273DWARF registers are encoded as numbers, which are mapped to architecture 1274registers. The mapping for AMDGPU is defined in 1275:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1276mapping. 1277 1278.. table:: AMDGPU DWARF Register Mapping 1279 :name: amdgpu-dwarf-register-mapping-table 1280 1281 ============== ================= ======== ================================== 1282 DWARF Register AMDGPU Register Bit Size Description 1283 ============== ================= ======== ================================== 1284 0 PC_32 32 Program Counter (PC) when 1285 executing in a 32-bit process 1286 address space. Used in the CFI to 1287 describe the PC of the calling 1288 frame. 1289 1 EXEC_MASK_32 32 Execution Mask Register when 1290 executing in wavefront 32 mode. 1291 2-15 *Reserved* *Reserved for highly accessed 1292 registers using DWARF shortcut.* 1293 16 PC_64 64 Program Counter (PC) when 1294 executing in a 64-bit process 1295 address space. Used in the CFI to 1296 describe the PC of the calling 1297 frame. 1298 17 EXEC_MASK_64 64 Execution Mask Register when 1299 executing in wavefront 64 mode. 1300 18-31 *Reserved* *Reserved for highly accessed 1301 registers using DWARF shortcut.* 1302 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1303 Registers. 1304 96-127 *Reserved* *Reserved for frequently accessed 1305 registers using DWARF 1-byte ULEB.* 1306 128 STATUS 32 Status Register. 1307 129-511 *Reserved* *Reserved for future Scalar 1308 Architectural Registers.* 1309 512 VCC_32 32 Vector Condition Code Register 1310 when executing in wavefront 32 1311 mode. 1312 513-1023 *Reserved* *Reserved for future Vector 1313 Architectural Registers when 1314 executing in wavefront 32 mode.* 1315 768 VCC_64 64 Vector Condition Code Register 1316 when executing in wavefront 64 1317 mode. 1318 769-1023 *Reserved* *Reserved for future Vector 1319 Architectural Registers when 1320 executing in wavefront 64 mode.* 1321 1024-1087 *Reserved* *Reserved for padding.* 1322 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1323 1130-1535 *Reserved* *Reserved for future Scalar 1324 General Purpose Registers.* 1325 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1326 when executing in wavefront 32 1327 mode. 1328 1792-2047 *Reserved* *Reserved for future Vector 1329 General Purpose Registers when 1330 executing in wavefront 32 mode.* 1331 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1332 when executing in wavefront 32 1333 mode. 1334 2304-2559 *Reserved* *Reserved for future Vector 1335 Accumulation Registers when 1336 executing in wavefront 32 mode.* 1337 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1338 when executing in wavefront 64 1339 mode. 1340 2816-3071 *Reserved* *Reserved for future Vector 1341 General Purpose Registers when 1342 executing in wavefront 64 mode.* 1343 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1344 when executing in wavefront 64 1345 mode. 1346 3328-3583 *Reserved* *Reserved for future Vector 1347 Accumulation Registers when 1348 executing in wavefront 64 mode.* 1349 ============== ================= ======== ================================== 1350 1351The vector registers are represented as the full size for the wavefront. They 1352are organized as consecutive dwords (32-bits), one per lane, with the dword at 1353the least significant bit position corresponding to lane 0 and so forth. DWARF 1354location expressions involving the ``DW_OP_LLVM_offset`` and 1355``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1356register corresponding to the lane that is executing the current thread of 1357execution in languages that are implemented using a SIMD or SIMT execution 1358model. 1359 1360If the wavefront size is 32 lanes then the wavefront 32 mode register 1361definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1362mode register definitions are used. Some AMDGPU targets support executing in 1363both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1364to the wavefront mode of the generated code will be used. 1365 1366If code is generated to execute in a 32-bit process address space, then the 136732-bit process address space register definitions are used. If code is generated 1368to execute in a 64-bit process address space, then the 64-bit process address 1369space register definitions are used. The ``amdgcn`` target only supports the 137064-bit process address space. 1371 1372.. _amdgpu-dwarf-address-class-identifier: 1373 1374Address Class Identifier 1375------------------------ 1376 1377The DWARF address class represents the source language memory space. See DWARF 1378Version 5 section 2.12 which is updated by the *DWARF Extensions For 1379Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1380 1381The DWARF address class mapping used for AMDGPU is defined in 1382:ref:`amdgpu-dwarf-address-class-mapping-table`. 1383 1384.. table:: AMDGPU DWARF Address Class Mapping 1385 :name: amdgpu-dwarf-address-class-mapping-table 1386 1387 ========================= ====== ================= 1388 DWARF AMDGPU 1389 -------------------------------- ----------------- 1390 Address Class Name Value Address Space 1391 ========================= ====== ================= 1392 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1393 ``DW_ADDR_LLVM_global`` 0x0001 Global 1394 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1395 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1396 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1397 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1398 ========================= ====== ================= 1399 1400The DWARF address class values defined in the *DWARF Extensions For 1401Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1402 1403In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1404available for use for the AMD extension for access to the hardware GDS memory 1405which is scratchpad memory allocated per device. 1406 1407For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1408address class of ``DW_ADDR_none`` is used. 1409 1410See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1411mapping of DWARF address classes to DWARF address spaces, including address size 1412and NULL value. 1413 1414.. _amdgpu-dwarf-address-space-identifier: 1415 1416Address Space Identifier 1417------------------------ 1418 1419DWARF address spaces correspond to target architecture specific linear 1420addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1421For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1422 1423The DWARF address space mapping used for AMDGPU is defined in 1424:ref:`amdgpu-dwarf-address-space-mapping-table`. 1425 1426.. table:: AMDGPU DWARF Address Space Mapping 1427 :name: amdgpu-dwarf-address-space-mapping-table 1428 1429 ======================================= ===== ======= ======== ================= ======================= 1430 DWARF AMDGPU Notes 1431 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1432 Address Space Name Value Address Bit Size Address Space 1433 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1434 .. 64-bit 32-bit 1435 process process 1436 address address 1437 space space 1438 ======================================= ===== ======= ======== ================= ======================= 1439 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1440 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1441 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1442 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1443 *Reserved* 0x04 1444 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1445 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1446 ======================================= ===== ======= ======== ================= ======================= 1447 1448See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1449including address size and NULL value. 1450 1451The ``DW_ASPACE_none`` address space is the default target architecture address 1452space used in DWARF operations that do not specify an address space. It 1453therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1454related operations can refer to addresses in the program code. 1455 1456The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1457specify the flat address space. If the address corresponds to an address in the 1458local address space, then it corresponds to the wavefront that is executing the 1459focused thread of execution. If the address corresponds to an address in the 1460private address space, then it corresponds to the lane that is executing the 1461focused thread of execution for languages that are implemented using a SIMD or 1462SIMT execution model. 1463 1464.. note:: 1465 1466 CUDA-like languages such as HIP that do not have address spaces in the 1467 language type system, but do allow variables to be allocated in different 1468 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1469 address space in the DWARF expression operations as the default address space 1470 is the global address space. 1471 1472The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1473specify the local address space corresponding to the wavefront that is executing 1474the focused thread of execution. 1475 1476The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1477to specify the private address space corresponding to the lane that is executing 1478the focused thread of execution for languages that are implemented using a SIMD 1479or SIMT execution model. 1480 1481The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1482to specify the unswizzled private address space corresponding to the wavefront 1483that is executing the focused thread of execution. The wavefront view of private 1484memory is the per wavefront unswizzled backing memory layout defined in 1485:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1486location for the backing memory of the wavefront (namely the address is not 1487offset by ``wavefront-scratch-base``). The following formula can be used to 1488convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1489``DW_ASPACE_AMDGPU_private_wave`` address: 1490 1491:: 1492 1493 private-address-wavefront = 1494 ((private-address-lane / 4) * wavefront-size * 4) + 1495 (wavefront-lane-id * 4) + (private-address-lane % 4) 1496 1497If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1498of the dwords for each lane starting with lane 0 is required, then this 1499simplifies to: 1500 1501:: 1502 1503 private-address-wavefront = 1504 private-address-lane * wavefront-size 1505 1506A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1507complete spilled vector register back into a complete vector register in the 1508CFI. The frame pointer can be a private lane address which is dword aligned, 1509which can be shifted to multiply by the wavefront size, and then used to form a 1510private wavefront address that gives a location for a contiguous set of dwords, 1511one per lane, where the vector register dwords are spilled. The compiler knows 1512the wavefront size since it generates the code. Note that the type of the 1513address may have to be converted as the size of a 1514``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1515``DW_ASPACE_AMDGPU_private_wave`` address. 1516 1517.. _amdgpu-dwarf-lane-identifier: 1518 1519Lane identifier 1520--------------- 1521 1522DWARF lane identifies specify a target architecture lane position for hardware 1523that executes in a SIMD or SIMT manner, and on which a source language maps its 1524threads of execution onto those lanes. The DWARF lane identifier is pushed by 1525the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1526section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1527section :ref:`amdgpu-dwarf-operation-expressions`. 1528 1529For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1530wavefront. It is numbered from 0 to the wavefront size minus 1. 1531 1532Operation Expressions 1533--------------------- 1534 1535DWARF expressions are used to compute program values and the locations of 1536program objects. See DWARF Version 5 section 2.5 and 1537:ref:`amdgpu-dwarf-operation-expressions`. 1538 1539DWARF location descriptions describe how to access storage which includes memory 1540and registers. When accessing storage on AMDGPU, bytes are ordered with least 1541significant bytes first, and bits are ordered within bytes with least 1542significant bits first. 1543 1544For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1545unwinding vector registers that are spilled under the execution mask to memory: 1546the zero-single location description is the vector register, and the one-single 1547location description is the spilled memory location description. The 1548``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1549memory location description. 1550 1551In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1552``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1553controlled by the execution mask. An undefined location description together 1554with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1555to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1556 1557Debugger Information Entry Attributes 1558------------------------------------- 1559 1560This section describes how certain debugger information entry attributes are 1561used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1562by *DWARF Extensions For Heterogeneous Debugging* section 1563:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1564 1565.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1566 1567``DW_AT_LLVM_lane_pc`` 1568~~~~~~~~~~~~~~~~~~~~~~ 1569 1570For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1571location of the separate lanes of a SIMT thread. 1572 1573If the lane is an active lane then this will be the same as the current program 1574location. 1575 1576If the lane is inactive, but was active on entry to the subprogram, then this is 1577the program location in the subprogram at which execution of the lane is 1578conceptual positioned. 1579 1580If the lane was not active on entry to the subprogram, then this will be the 1581undefined location. A client debugger can check if the lane is part of a valid 1582work-group by checking that the lane is in the range of the associated 1583work-group within the grid, accounting for partial work-groups. If it is not, 1584then the debugger can omit any information for the lane. Otherwise, the debugger 1585may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1586calling subprogram until it finds a non-undefined location. Conceptually the 1587lane only has the call frames that it has a non-undefined 1588``DW_AT_LLVM_lane_pc``. 1589 1590The following example illustrates how the AMDGPU backend can generate a DWARF 1591location list expression for the nested ``IF/THEN/ELSE`` structures of the 1592following subprogram pseudo code for a target with 64 lanes per wavefront. 1593 1594.. code:: 1595 :number-lines: 1596 1597 SUBPROGRAM X 1598 BEGIN 1599 a; 1600 IF (c1) THEN 1601 b; 1602 IF (c2) THEN 1603 c; 1604 ELSE 1605 d; 1606 ENDIF 1607 e; 1608 ELSE 1609 f; 1610 ENDIF 1611 g; 1612 END 1613 1614The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1615execution mask (``EXEC``) to linearize the control flow. The condition is 1616evaluated to make a mask of the lanes for which the condition evaluates to true. 1617First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1618logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1619``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1620the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1621region the ``EXEC`` mask is restored to the value it had at the beginning of the 1622region. This is shown below. Other approaches are possible, but the basic 1623concept is the same. 1624 1625.. code:: 1626 :number-lines: 1627 1628 $lex_start: 1629 a; 1630 %1 = EXEC 1631 %2 = c1 1632 $lex_1_start: 1633 EXEC = %1 & %2 1634 $if_1_then: 1635 b; 1636 %3 = EXEC 1637 %4 = c2 1638 $lex_1_1_start: 1639 EXEC = %3 & %4 1640 $lex_1_1_then: 1641 c; 1642 EXEC = ~EXEC & %3 1643 $lex_1_1_else: 1644 d; 1645 EXEC = %3 1646 $lex_1_1_end: 1647 e; 1648 EXEC = ~EXEC & %1 1649 $lex_1_else: 1650 f; 1651 EXEC = %1 1652 $lex_1_end: 1653 g; 1654 $lex_end: 1655 1656To create the DWARF location list expression that defines the location 1657description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1658pseudo instruction can be used to annotate the linearized control flow. This can 1659be done by defining an artificial variable for the lane PC. The DWARF location 1660list expression created for it is used as the value of the 1661``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1662 1663A DWARF procedure is defined for each well nested structured control flow region 1664which provides the conceptual lane program location for a lane if it is not 1665active (namely it is divergent). The DWARF operation expression for each region 1666conceptually inherits the value of the immediately enclosing region and modifies 1667it according to the semantics of the region. 1668 1669For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1670the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1671region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1672region since the ``THEN`` region has completed. 1673 1674The lane PC artificial variable is assigned at each region transition. It uses 1675the immediately enclosing region's DWARF procedure to compute the program 1676location for each lane assuming they are divergent, and then modifies the result 1677by inserting the current program location for each lane that the ``EXEC`` mask 1678indicates is active. 1679 1680By having separate DWARF procedures for each region, they can be reused to 1681define the value for any nested region. This reduces the total size of the DWARF 1682operation expressions. 1683 1684The following provides an example using pseudo LLVM MIR. 1685 1686.. code:: 1687 :number-lines: 1688 1689 $lex_start: 1690 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1691 DW_AT_name = "__uint64"; 1692 DW_AT_byte_size = 8; 1693 DW_AT_encoding = DW_ATE_unsigned; 1694 ]; 1695 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1696 DW_AT_name = "__active_lane_pc"; 1697 DW_AT_location = [ 1698 DW_OP_regx PC; 1699 DW_OP_LLVM_extend 64, 64; 1700 DW_OP_regval_type EXEC, %uint_64; 1701 DW_OP_LLVM_select_bit_piece 64, 64; 1702 ]; 1703 ]; 1704 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1705 DW_AT_name = "__divergent_lane_pc"; 1706 DW_AT_location = [ 1707 DW_OP_LLVM_undefined; 1708 DW_OP_LLVM_extend 64, 64; 1709 ]; 1710 ]; 1711 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1712 DW_OP_call_ref %__divergent_lane_pc; 1713 DW_OP_call_ref %__active_lane_pc; 1714 ]; 1715 a; 1716 %1 = EXEC; 1717 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1718 %2 = c1; 1719 $lex_1_start: 1720 EXEC = %1 & %2; 1721 $lex_1_then: 1722 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1723 DW_AT_name = "__divergent_lane_pc_1_then"; 1724 DW_AT_location = DIExpression[ 1725 DW_OP_call_ref %__divergent_lane_pc; 1726 DW_OP_addrx &lex_1_start; 1727 DW_OP_stack_value; 1728 DW_OP_LLVM_extend 64, 64; 1729 DW_OP_call_ref %__lex_1_save_exec; 1730 DW_OP_deref_type 64, %__uint_64; 1731 DW_OP_LLVM_select_bit_piece 64, 64; 1732 ]; 1733 ]; 1734 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1735 DW_OP_call_ref %__divergent_lane_pc_1_then; 1736 DW_OP_call_ref %__active_lane_pc; 1737 ]; 1738 b; 1739 %3 = EXEC; 1740 DBG_VALUE %3, %__lex_1_1_save_exec; 1741 %4 = c2; 1742 $lex_1_1_start: 1743 EXEC = %3 & %4; 1744 $lex_1_1_then: 1745 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1746 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1747 DW_AT_location = DIExpression[ 1748 DW_OP_call_ref %__divergent_lane_pc_1_then; 1749 DW_OP_addrx &lex_1_1_start; 1750 DW_OP_stack_value; 1751 DW_OP_LLVM_extend 64, 64; 1752 DW_OP_call_ref %__lex_1_1_save_exec; 1753 DW_OP_deref_type 64, %__uint_64; 1754 DW_OP_LLVM_select_bit_piece 64, 64; 1755 ]; 1756 ]; 1757 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1758 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1759 DW_OP_call_ref %__active_lane_pc; 1760 ]; 1761 c; 1762 EXEC = ~EXEC & %3; 1763 $lex_1_1_else: 1764 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1765 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1766 DW_AT_location = DIExpression[ 1767 DW_OP_call_ref %__divergent_lane_pc_1_then; 1768 DW_OP_addrx &lex_1_1_end; 1769 DW_OP_stack_value; 1770 DW_OP_LLVM_extend 64, 64; 1771 DW_OP_call_ref %__lex_1_1_save_exec; 1772 DW_OP_deref_type 64, %__uint_64; 1773 DW_OP_LLVM_select_bit_piece 64, 64; 1774 ]; 1775 ]; 1776 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1777 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1778 DW_OP_call_ref %__active_lane_pc; 1779 ]; 1780 d; 1781 EXEC = %3; 1782 $lex_1_1_end: 1783 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1784 DW_OP_call_ref %__divergent_lane_pc; 1785 DW_OP_call_ref %__active_lane_pc; 1786 ]; 1787 e; 1788 EXEC = ~EXEC & %1; 1789 $lex_1_else: 1790 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1791 DW_AT_name = "__divergent_lane_pc_1_else"; 1792 DW_AT_location = DIExpression[ 1793 DW_OP_call_ref %__divergent_lane_pc; 1794 DW_OP_addrx &lex_1_end; 1795 DW_OP_stack_value; 1796 DW_OP_LLVM_extend 64, 64; 1797 DW_OP_call_ref %__lex_1_save_exec; 1798 DW_OP_deref_type 64, %__uint_64; 1799 DW_OP_LLVM_select_bit_piece 64, 64; 1800 ]; 1801 ]; 1802 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1803 DW_OP_call_ref %__divergent_lane_pc_1_else; 1804 DW_OP_call_ref %__active_lane_pc; 1805 ]; 1806 f; 1807 EXEC = %1; 1808 $lex_1_end: 1809 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1810 DW_OP_call_ref %__divergent_lane_pc; 1811 DW_OP_call_ref %__active_lane_pc; 1812 ]; 1813 g; 1814 $lex_end: 1815 1816The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1817that are active, with the current program location. 1818 1819Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1820the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1821instruction, location list entries will be created that describe where the 1822artificial variables are allocated at any given program location. The compiler 1823may allocate them to registers or spill them to memory. 1824 1825The DWARF procedures for each region use the values of the saved execution mask 1826artificial variables to only update the lanes that are active on entry to the 1827region. All other lanes retain the value of the enclosing region where they were 1828last active. If they were not active on entry to the subprogram, then will have 1829the undefined location description. 1830 1831Other structured control flow regions can be handled similarly. For example, 1832loops would set the divergent program location for the region at the end of the 1833loop. Any lanes active will be in the loop, and any lanes not active must have 1834exited the loop. 1835 1836An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1837``IF/THEN/ELSE`` regions. 1838 1839The DWARF procedures can use the active lane artificial variable described in 1840:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1841``EXEC`` mask in order to support whole or quad wavefront mode. 1842 1843.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1844 1845``DW_AT_LLVM_active_lane`` 1846~~~~~~~~~~~~~~~~~~~~~~~~~~ 1847 1848The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1849entry is used to specify the lanes that are conceptually active for a SIMT 1850thread. 1851 1852The execution mask may be modified to implement whole or quad wavefront mode 1853operations. For example, all lanes may need to temporarily be made active to 1854execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1855update it to enable the necessary lanes, perform the operations, and then 1856restore the ``EXEC`` mask from the saved value. While executing the whole 1857wavefront region, the conceptual execution mask is the saved value, not the 1858``EXEC`` value. 1859 1860This is handled by defining an artificial variable for the active lane mask. The 1861active lane mask artificial variable would be the actual ``EXEC`` mask for 1862normal regions, and the saved execution mask for regions where the mask is 1863temporarily updated. The location list expression created for this artificial 1864variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1865attribute. 1866 1867``DW_AT_LLVM_augmentation`` 1868~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1869 1870For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1871debugger information entry has the following value for the augmentation string: 1872 1873:: 1874 1875 [amdgpu:v0.0] 1876 1877The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1878extensions used in the DWARF of the compilation unit. The version number 1879conforms to [SEMVER]_. 1880 1881Call Frame Information 1882---------------------- 1883 1884DWARF Call Frame Information (CFI) describes how a consumer can virtually 1885*unwind* call frames in a running process or core dump. See DWARF Version 5 1886section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1887 1888For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1889 18901. ``augmentation`` string contains the following null-terminated UTF-8 string: 1891 1892 :: 1893 1894 [amd:v0.0] 1895 1896 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1897 extensions used in this CIE or to the FDEs that use it. The version number 1898 conforms to [SEMVER]_. 1899 19002. ``address_size`` for the ``Global`` address space is defined in 1901 :ref:`amdgpu-dwarf-address-space-identifier`. 1902 19033. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1904 19054. ``code_alignment_factor`` is 4 bytes. 1906 1907 .. TODO:: 1908 1909 Add to :ref:`amdgpu-processor-table` table. 1910 19115. ``data_alignment_factor`` is 4 bytes. 1912 1913 .. TODO:: 1914 1915 Add to :ref:`amdgpu-processor-table` table. 1916 19176. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1918 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1919 19207. ``initial_instructions`` Since a subprogram X with fewer registers can be 1921 called from subprogram Y that has more allocated, X will not change any of 1922 the extra registers as it cannot access them. Therefore, the default rule 1923 for all columns is ``same value``. 1924 1925For AMDGPU the register number follows the numbering defined in 1926:ref:`amdgpu-dwarf-register-identifier`. 1927 1928For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1929the return address to get the address of a byte within the call site 1930instructions. See DWARF Version 5 section 6.4.4. 1931 1932Accelerated Access 1933------------------ 1934 1935See DWARF Version 5 section 6.1. 1936 1937Lookup By Name Section Header 1938~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1939 1940See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1941 1942For AMDGPU the lookup by name section header table: 1943 1944``augmentation_string_size`` (uword) 1945 1946 Set to the length of the ``augmentation_string`` value which is always a 1947 multiple of 4. 1948 1949``augmentation_string`` (sequence of UTF-8 characters) 1950 1951 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1952 1953 :: 1954 1955 [amdgpu:v0.0] 1956 1957 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1958 extensions used in the DWARF of this index. The version number conforms to 1959 [SEMVER]_. 1960 1961 .. note:: 1962 1963 This is different to the DWARF Version 5 definition that requires the first 1964 4 characters to be the vendor ID. But this is consistent with the other 1965 augmentation strings and does allow multiple vendor contributions. However, 1966 backwards compatibility may be more desirable. 1967 1968Lookup By Address Section Header 1969~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1970 1971See DWARF Version 5 section 6.1.2. 1972 1973For AMDGPU the lookup by address section header table: 1974 1975``address_size`` (ubyte) 1976 1977 Match the address size for the ``Global`` address space defined in 1978 :ref:`amdgpu-dwarf-address-space-identifier`. 1979 1980``segment_selector_size`` (ubyte) 1981 1982 AMDGPU does not use a segment selector so this is 0. The entries in the 1983 ``.debug_aranges`` do not have a segment selector. 1984 1985Line Number Information 1986----------------------- 1987 1988See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1989 1990AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1991The instruction set must be obtained from the ELF file header ``e_flags`` field 1992in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1993<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1994 1995.. TODO:: 1996 1997 Should the ``isa`` state machine register be used to indicate if the code is 1998 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1999 2000For AMDGPU the line number program header fields have the following values (see 2001DWARF Version 5 section 6.2.4): 2002 2003``address_size`` (ubyte) 2004 Matches the address size for the ``Global`` address space defined in 2005 :ref:`amdgpu-dwarf-address-space-identifier`. 2006 2007``segment_selector_size`` (ubyte) 2008 AMDGPU does not use a segment selector so this is 0. 2009 2010``minimum_instruction_length`` (ubyte) 2011 For GFX9-GFX10 this is 4. 2012 2013``maximum_operations_per_instruction`` (ubyte) 2014 For GFX9-GFX10 this is 1. 2015 2016Source text for online-compiled programs (for example, those compiled by the 2017OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2018See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2019Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2020<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2021 2022The Clang option used to control source embedding in AMDGPU is defined in 2023:ref:`amdgpu-clang-debug-options-table`. 2024 2025 .. table:: AMDGPU Clang Debug Options 2026 :name: amdgpu-clang-debug-options-table 2027 2028 ==================== ================================================== 2029 Debug Flag Description 2030 ==================== ================================================== 2031 -g[no-]embed-source Enable/disable embedding source text in DWARF 2032 debug sections. Useful for environments where 2033 source cannot be written to disk, such as 2034 when performing online compilation. 2035 ==================== ================================================== 2036 2037For example: 2038 2039``-gembed-source`` 2040 Enable the embedded source. 2041 2042``-gno-embed-source`` 2043 Disable the embedded source. 2044 204532-Bit and 64-Bit DWARF Formats 2046------------------------------- 2047 2048See DWARF Version 5 section 7.4 and 2049:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2050 2051For AMDGPU: 2052 2053* For the ``amdgcn`` target architecture only the 64-bit process address space 2054 is supported. 2055 2056* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2057 the 32-bit DWARF format. 2058 2059Unit Headers 2060------------ 2061 2062For AMDGPU the following values apply for each of the unit headers described in 2063DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2064 2065``address_size`` (ubyte) 2066 Matches the address size for the ``Global`` address space defined in 2067 :ref:`amdgpu-dwarf-address-space-identifier`. 2068 2069.. _amdgpu-code-conventions: 2070 2071Code Conventions 2072================ 2073 2074This section provides code conventions used for each supported target triple OS 2075(see :ref:`amdgpu-target-triples`). 2076 2077AMDHSA 2078------ 2079 2080This section provides code conventions used when the target triple OS is 2081``amdhsa`` (see :ref:`amdgpu-target-triples`). 2082 2083.. _amdgpu-amdhsa-code-object-target-identification: 2084 2085Code Object Target Identification 2086~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2087 2088The AMDHSA OS uses the following syntax to specify the code object 2089target as a single string: 2090 2091 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 2092 2093Where: 2094 2095 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 2096 are the same as the *Target Triple* (see 2097 :ref:`amdgpu-target-triples`). 2098 2099 - ``<Processor>`` is the same as the *Processor* (see 2100 :ref:`amdgpu-processors`). 2101 2102 - ``<Target Features>`` is a list of the enabled *Target Features* 2103 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 2104 apply to *Processor*. The list must be in the same order as listed 2105 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 2106 Features* must be included in the list if they are enabled even if 2107 that is the default for *Processor*. 2108 2109For example: 2110 2111 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 2112 2113.. _amdgpu-amdhsa-code-object-metadata: 2114 2115Code Object Metadata 2116~~~~~~~~~~~~~~~~~~~~ 2117 2118The code object metadata specifies extensible metadata associated with the code 2119objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 2120[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 2121object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 2122:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2123 2124Code object metadata is specified in a note record (see 2125:ref:`amdgpu-note-records`) and is required when the target triple OS is 2126``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2127information necessary to support the ROCM kernel queries. For example, the 2128segment sizes needed in a dispatch packet. In addition, a high-level language 2129runtime may require other information to be included. For example, the AMD 2130OpenCL runtime records kernel argument information. 2131 2132.. _amdgpu-amdhsa-code-object-metadata-v2: 2133 2134Code Object V2 Metadata (--amdhsa-code-object-version=2) 2135++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2136 2137.. warning:: Code Object V2 is not the default code object version emitted by 2138 this version of LLVM. For a description of the metadata generated with the 2139 default configuration (Code Object V3) see 2140 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2141 2142Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 2143record (see :ref:`amdgpu-note-records-v2`). 2144 2145The metadata is specified as a YAML formatted string (see [YAML]_ and 2146:doc:`YamlIO`). 2147 2148.. TODO:: 2149 2150 Is the string null terminated? It probably should not if YAML allows it to 2151 contain null characters, otherwise it should be. 2152 2153The metadata is represented as a single YAML document comprised of the mapping 2154defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2155referenced tables. 2156 2157For boolean values, the string values of ``false`` and ``true`` are used for 2158false and true respectively. 2159 2160Additional information can be added to the mappings. To avoid conflicts, any 2161non-AMD key names should be prefixed by "*vendor-name*.". 2162 2163 .. table:: AMDHSA Code Object V2 Metadata Map 2164 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2165 2166 ========== ============== ========= ======================================= 2167 String Key Value Type Required? Description 2168 ========== ============== ========= ======================================= 2169 "Version" sequence of Required - The first integer is the major 2170 2 integers version. Currently 1. 2171 - The second integer is the minor 2172 version. Currently 0. 2173 "Printf" sequence of Each string is encoded information 2174 strings about a printf function call. The 2175 encoded information is organized as 2176 fields separated by colon (':'): 2177 2178 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2179 2180 where: 2181 2182 ``ID`` 2183 A 32-bit integer as a unique id for 2184 each printf function call 2185 2186 ``N`` 2187 A 32-bit integer equal to the number 2188 of arguments of printf function call 2189 minus 1 2190 2191 ``S[i]`` (where i = 0, 1, ... , N-1) 2192 32-bit integers for the size in bytes 2193 of the i-th FormatString argument of 2194 the printf function call 2195 2196 FormatString 2197 The format string passed to the 2198 printf function call. 2199 "Kernels" sequence of Required Sequence of the mappings for each 2200 mapping kernel in the code object. See 2201 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2202 for the definition of the mapping. 2203 ========== ============== ========= ======================================= 2204 2205.. 2206 2207 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2208 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2209 2210 ================= ============== ========= ================================ 2211 String Key Value Type Required? Description 2212 ================= ============== ========= ================================ 2213 "Name" string Required Source name of the kernel. 2214 "SymbolName" string Required Name of the kernel 2215 descriptor ELF symbol. 2216 "Language" string Source language of the kernel. 2217 Values include: 2218 2219 - "OpenCL C" 2220 - "OpenCL C++" 2221 - "HCC" 2222 - "OpenMP" 2223 2224 "LanguageVersion" sequence of - The first integer is the major 2225 2 integers version. 2226 - The second integer is the 2227 minor version. 2228 "Attrs" mapping Mapping of kernel attributes. 2229 See 2230 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2231 for the mapping definition. 2232 "Args" sequence of Sequence of mappings of the 2233 mapping kernel arguments. See 2234 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2235 for the definition of the mapping. 2236 "CodeProps" mapping Mapping of properties related to 2237 the kernel code. See 2238 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2239 for the mapping definition. 2240 ================= ============== ========= ================================ 2241 2242.. 2243 2244 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2245 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2246 2247 =================== ============== ========= ============================== 2248 String Key Value Type Required? Description 2249 =================== ============== ========= ============================== 2250 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2251 3 integers must be >=1 and the dispatch 2252 work-group size X, Y, Z must 2253 correspond to the specified 2254 values. Defaults to 0, 0, 0. 2255 2256 Corresponds to the OpenCL 2257 ``reqd_work_group_size`` 2258 attribute. 2259 "WorkGroupSizeHint" sequence of The dispatch work-group size 2260 3 integers X, Y, Z is likely to be the 2261 specified values. 2262 2263 Corresponds to the OpenCL 2264 ``work_group_size_hint`` 2265 attribute. 2266 "VecTypeHint" string The name of a scalar or vector 2267 type. 2268 2269 Corresponds to the OpenCL 2270 ``vec_type_hint`` attribute. 2271 2272 "RuntimeHandle" string The external symbol name 2273 associated with a kernel. 2274 OpenCL runtime allocates a 2275 global buffer for the symbol 2276 and saves the kernel's address 2277 to it, which is used for 2278 device side enqueueing. Only 2279 available for device side 2280 enqueued kernels. 2281 =================== ============== ========= ============================== 2282 2283.. 2284 2285 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2286 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2287 2288 ================= ============== ========= ================================ 2289 String Key Value Type Required? Description 2290 ================= ============== ========= ================================ 2291 "Name" string Kernel argument name. 2292 "TypeName" string Kernel argument type name. 2293 "Size" integer Required Kernel argument size in bytes. 2294 "Align" integer Required Kernel argument alignment in 2295 bytes. Must be a power of two. 2296 "ValueKind" string Required Kernel argument kind that 2297 specifies how to set up the 2298 corresponding argument. 2299 Values include: 2300 2301 "ByValue" 2302 The argument is copied 2303 directly into the kernarg. 2304 2305 "GlobalBuffer" 2306 A global address space pointer 2307 to the buffer data is passed 2308 in the kernarg. 2309 2310 "DynamicSharedPointer" 2311 A group address space pointer 2312 to dynamically allocated LDS 2313 is passed in the kernarg. 2314 2315 "Sampler" 2316 A global address space 2317 pointer to a S# is passed in 2318 the kernarg. 2319 2320 "Image" 2321 A global address space 2322 pointer to a T# is passed in 2323 the kernarg. 2324 2325 "Pipe" 2326 A global address space pointer 2327 to an OpenCL pipe is passed in 2328 the kernarg. 2329 2330 "Queue" 2331 A global address space pointer 2332 to an OpenCL device enqueue 2333 queue is passed in the 2334 kernarg. 2335 2336 "HiddenGlobalOffsetX" 2337 The OpenCL grid dispatch 2338 global offset for the X 2339 dimension is passed in the 2340 kernarg. 2341 2342 "HiddenGlobalOffsetY" 2343 The OpenCL grid dispatch 2344 global offset for the Y 2345 dimension is passed in the 2346 kernarg. 2347 2348 "HiddenGlobalOffsetZ" 2349 The OpenCL grid dispatch 2350 global offset for the Z 2351 dimension is passed in the 2352 kernarg. 2353 2354 "HiddenNone" 2355 An argument that is not used 2356 by the kernel. Space needs to 2357 be left for it, but it does 2358 not need to be set up. 2359 2360 "HiddenPrintfBuffer" 2361 A global address space pointer 2362 to the runtime printf buffer 2363 is passed in kernarg. 2364 2365 "HiddenHostcallBuffer" 2366 A global address space pointer 2367 to the runtime hostcall buffer 2368 is passed in kernarg. 2369 2370 "HiddenDefaultQueue" 2371 A global address space pointer 2372 to the OpenCL device enqueue 2373 queue that should be used by 2374 the kernel by default is 2375 passed in the kernarg. 2376 2377 "HiddenCompletionAction" 2378 A global address space pointer 2379 to help link enqueued kernels into 2380 the ancestor tree for determining 2381 when the parent kernel has finished. 2382 2383 "HiddenMultiGridSyncArg" 2384 A global address space pointer for 2385 multi-grid synchronization is 2386 passed in the kernarg. 2387 2388 "ValueType" string Unused and deprecated. This should no longer 2389 be emitted, but is accepted for compatibility. 2390 2391 2392 "PointeeAlign" integer Alignment in bytes of pointee 2393 type for pointer type kernel 2394 argument. Must be a power 2395 of 2. Only present if 2396 "ValueKind" is 2397 "DynamicSharedPointer". 2398 "AddrSpaceQual" string Kernel argument address space 2399 qualifier. Only present if 2400 "ValueKind" is "GlobalBuffer" or 2401 "DynamicSharedPointer". Values 2402 are: 2403 2404 - "Private" 2405 - "Global" 2406 - "Constant" 2407 - "Local" 2408 - "Generic" 2409 - "Region" 2410 2411 .. TODO:: 2412 2413 Is GlobalBuffer only Global 2414 or Constant? Is 2415 DynamicSharedPointer always 2416 Local? Can HCC allow Generic? 2417 How can Private or Region 2418 ever happen? 2419 2420 "AccQual" string Kernel argument access 2421 qualifier. Only present if 2422 "ValueKind" is "Image" or 2423 "Pipe". Values 2424 are: 2425 2426 - "ReadOnly" 2427 - "WriteOnly" 2428 - "ReadWrite" 2429 2430 .. TODO:: 2431 2432 Does this apply to 2433 GlobalBuffer? 2434 2435 "ActualAccQual" string The actual memory accesses 2436 performed by the kernel on the 2437 kernel argument. Only present if 2438 "ValueKind" is "GlobalBuffer", 2439 "Image", or "Pipe". This may be 2440 more restrictive than indicated 2441 by "AccQual" to reflect what the 2442 kernel actual does. If not 2443 present then the runtime must 2444 assume what is implied by 2445 "AccQual" and "IsConst". Values 2446 are: 2447 2448 - "ReadOnly" 2449 - "WriteOnly" 2450 - "ReadWrite" 2451 2452 "IsConst" boolean Indicates if the kernel argument 2453 is const qualified. Only present 2454 if "ValueKind" is 2455 "GlobalBuffer". 2456 2457 "IsRestrict" boolean Indicates if the kernel argument 2458 is restrict qualified. Only 2459 present if "ValueKind" is 2460 "GlobalBuffer". 2461 2462 "IsVolatile" boolean Indicates if the kernel argument 2463 is volatile qualified. Only 2464 present if "ValueKind" is 2465 "GlobalBuffer". 2466 2467 "IsPipe" boolean Indicates if the kernel argument 2468 is pipe qualified. Only present 2469 if "ValueKind" is "Pipe". 2470 2471 .. TODO:: 2472 2473 Can GlobalBuffer be pipe 2474 qualified? 2475 2476 ================= ============== ========= ================================ 2477 2478.. 2479 2480 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2481 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2482 2483 ============================ ============== ========= ===================== 2484 String Key Value Type Required? Description 2485 ============================ ============== ========= ===================== 2486 "KernargSegmentSize" integer Required The size in bytes of 2487 the kernarg segment 2488 that holds the values 2489 of the arguments to 2490 the kernel. 2491 "GroupSegmentFixedSize" integer Required The amount of group 2492 segment memory 2493 required by a 2494 work-group in 2495 bytes. This does not 2496 include any 2497 dynamically allocated 2498 group segment memory 2499 that may be added 2500 when the kernel is 2501 dispatched. 2502 "PrivateSegmentFixedSize" integer Required The amount of fixed 2503 private address space 2504 memory required for a 2505 work-item in 2506 bytes. If the kernel 2507 uses a dynamic call 2508 stack then additional 2509 space must be added 2510 to this value for the 2511 call stack. 2512 "KernargSegmentAlign" integer Required The maximum byte 2513 alignment of 2514 arguments in the 2515 kernarg segment. Must 2516 be a power of 2. 2517 "WavefrontSize" integer Required Wavefront size. Must 2518 be a power of 2. 2519 "NumSGPRs" integer Required Number of scalar 2520 registers used by a 2521 wavefront for 2522 GFX6-GFX10. This 2523 includes the special 2524 SGPRs for VCC, Flat 2525 Scratch (GFX7-GFX10) 2526 and XNACK (for 2527 GFX8-GFX10). It does 2528 not include the 16 2529 SGPR added if a trap 2530 handler is 2531 enabled. It is not 2532 rounded up to the 2533 allocation 2534 granularity. 2535 "NumVGPRs" integer Required Number of vector 2536 registers used by 2537 each work-item for 2538 GFX6-GFX10 2539 "MaxFlatWorkGroupSize" integer Required Maximum flat 2540 work-group size 2541 supported by the 2542 kernel in work-items. 2543 Must be >=1 and 2544 consistent with 2545 ReqdWorkGroupSize if 2546 not 0, 0, 0. 2547 "NumSpilledSGPRs" integer Number of stores from 2548 a scalar register to 2549 a register allocator 2550 created spill 2551 location. 2552 "NumSpilledVGPRs" integer Number of stores from 2553 a vector register to 2554 a register allocator 2555 created spill 2556 location. 2557 ============================ ============== ========= ===================== 2558 2559.. _amdgpu-amdhsa-code-object-metadata-v3: 2560 2561Code Object V3 Metadata (--amdhsa-code-object-version=3) 2562++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2563 2564Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2565(see :ref:`amdgpu-note-records-v3`). 2566 2567The metadata is represented as Message Pack formatted binary data (see 2568[MsgPack]_). The top level is a Message Pack map that includes the 2569keys defined in table 2570:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2571tables. 2572 2573Additional information can be added to the maps. To avoid conflicts, 2574any key names should be prefixed by "*vendor-name*." where 2575``vendor-name`` can be the name of the vendor and specific vendor 2576tool that generates the information. The prefix is abbreviated to 2577simply "." when it appears within a map that has been added by the 2578same *vendor-name*. 2579 2580 .. table:: AMDHSA Code Object V3 Metadata Map 2581 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2582 2583 ================= ============== ========= ======================================= 2584 String Key Value Type Required? Description 2585 ================= ============== ========= ======================================= 2586 "amdhsa.version" sequence of Required - The first integer is the major 2587 2 integers version. Currently 1. 2588 - The second integer is the minor 2589 version. Currently 0. 2590 "amdhsa.printf" sequence of Each string is encoded information 2591 strings about a printf function call. The 2592 encoded information is organized as 2593 fields separated by colon (':'): 2594 2595 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2596 2597 where: 2598 2599 ``ID`` 2600 A 32-bit integer as a unique id for 2601 each printf function call 2602 2603 ``N`` 2604 A 32-bit integer equal to the number 2605 of arguments of printf function call 2606 minus 1 2607 2608 ``S[i]`` (where i = 0, 1, ... , N-1) 2609 32-bit integers for the size in bytes 2610 of the i-th FormatString argument of 2611 the printf function call 2612 2613 FormatString 2614 The format string passed to the 2615 printf function call. 2616 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2617 map kernel in the code object. See 2618 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2619 for the definition of the keys included 2620 in that map. 2621 ================= ============== ========= ======================================= 2622 2623.. 2624 2625 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2626 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2627 2628 =================================== ============== ========= ================================ 2629 String Key Value Type Required? Description 2630 =================================== ============== ========= ================================ 2631 ".name" string Required Source name of the kernel. 2632 ".symbol" string Required Name of the kernel 2633 descriptor ELF symbol. 2634 ".language" string Source language of the kernel. 2635 Values include: 2636 2637 - "OpenCL C" 2638 - "OpenCL C++" 2639 - "HCC" 2640 - "HIP" 2641 - "OpenMP" 2642 - "Assembler" 2643 2644 ".language_version" sequence of - The first integer is the major 2645 2 integers version. 2646 - The second integer is the 2647 minor version. 2648 ".args" sequence of Sequence of maps of the 2649 map kernel arguments. See 2650 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2651 for the definition of the keys 2652 included in that map. 2653 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2654 3 integers must be >=1 and the dispatch 2655 work-group size X, Y, Z must 2656 correspond to the specified 2657 values. Defaults to 0, 0, 0. 2658 2659 Corresponds to the OpenCL 2660 ``reqd_work_group_size`` 2661 attribute. 2662 ".workgroup_size_hint" sequence of The dispatch work-group size 2663 3 integers X, Y, Z is likely to be the 2664 specified values. 2665 2666 Corresponds to the OpenCL 2667 ``work_group_size_hint`` 2668 attribute. 2669 ".vec_type_hint" string The name of a scalar or vector 2670 type. 2671 2672 Corresponds to the OpenCL 2673 ``vec_type_hint`` attribute. 2674 2675 ".device_enqueue_symbol" string The external symbol name 2676 associated with a kernel. 2677 OpenCL runtime allocates a 2678 global buffer for the symbol 2679 and saves the kernel's address 2680 to it, which is used for 2681 device side enqueueing. Only 2682 available for device side 2683 enqueued kernels. 2684 ".kernarg_segment_size" integer Required The size in bytes of 2685 the kernarg segment 2686 that holds the values 2687 of the arguments to 2688 the kernel. 2689 ".group_segment_fixed_size" integer Required The amount of group 2690 segment memory 2691 required by a 2692 work-group in 2693 bytes. This does not 2694 include any 2695 dynamically allocated 2696 group segment memory 2697 that may be added 2698 when the kernel is 2699 dispatched. 2700 ".private_segment_fixed_size" integer Required The amount of fixed 2701 private address space 2702 memory required for a 2703 work-item in 2704 bytes. If the kernel 2705 uses a dynamic call 2706 stack then additional 2707 space must be added 2708 to this value for the 2709 call stack. 2710 ".kernarg_segment_align" integer Required The maximum byte 2711 alignment of 2712 arguments in the 2713 kernarg segment. Must 2714 be a power of 2. 2715 ".wavefront_size" integer Required Wavefront size. Must 2716 be a power of 2. 2717 ".sgpr_count" integer Required Number of scalar 2718 registers required by a 2719 wavefront for 2720 GFX6-GFX9. A register 2721 is required if it is 2722 used explicitly, or 2723 if a higher numbered 2724 register is used 2725 explicitly. This 2726 includes the special 2727 SGPRs for VCC, Flat 2728 Scratch (GFX7-GFX9) 2729 and XNACK (for 2730 GFX8-GFX9). It does 2731 not include the 16 2732 SGPR added if a trap 2733 handler is 2734 enabled. It is not 2735 rounded up to the 2736 allocation 2737 granularity. 2738 ".vgpr_count" integer Required Number of vector 2739 registers required by 2740 each work-item for 2741 GFX6-GFX9. A register 2742 is required if it is 2743 used explicitly, or 2744 if a higher numbered 2745 register is used 2746 explicitly. 2747 ".max_flat_workgroup_size" integer Required Maximum flat 2748 work-group size 2749 supported by the 2750 kernel in work-items. 2751 Must be >=1 and 2752 consistent with 2753 ReqdWorkGroupSize if 2754 not 0, 0, 0. 2755 ".sgpr_spill_count" integer Number of stores from 2756 a scalar register to 2757 a register allocator 2758 created spill 2759 location. 2760 ".vgpr_spill_count" integer Number of stores from 2761 a vector register to 2762 a register allocator 2763 created spill 2764 location. 2765 =================================== ============== ========= ================================ 2766 2767.. 2768 2769 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2770 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2771 2772 ====================== ============== ========= ================================ 2773 String Key Value Type Required? Description 2774 ====================== ============== ========= ================================ 2775 ".name" string Kernel argument name. 2776 ".type_name" string Kernel argument type name. 2777 ".size" integer Required Kernel argument size in bytes. 2778 ".offset" integer Required Kernel argument offset in 2779 bytes. The offset must be a 2780 multiple of the alignment 2781 required by the argument. 2782 ".value_kind" string Required Kernel argument kind that 2783 specifies how to set up the 2784 corresponding argument. 2785 Values include: 2786 2787 "by_value" 2788 The argument is copied 2789 directly into the kernarg. 2790 2791 "global_buffer" 2792 A global address space pointer 2793 to the buffer data is passed 2794 in the kernarg. 2795 2796 "dynamic_shared_pointer" 2797 A group address space pointer 2798 to dynamically allocated LDS 2799 is passed in the kernarg. 2800 2801 "sampler" 2802 A global address space 2803 pointer to a S# is passed in 2804 the kernarg. 2805 2806 "image" 2807 A global address space 2808 pointer to a T# is passed in 2809 the kernarg. 2810 2811 "pipe" 2812 A global address space pointer 2813 to an OpenCL pipe is passed in 2814 the kernarg. 2815 2816 "queue" 2817 A global address space pointer 2818 to an OpenCL device enqueue 2819 queue is passed in the 2820 kernarg. 2821 2822 "hidden_global_offset_x" 2823 The OpenCL grid dispatch 2824 global offset for the X 2825 dimension is passed in the 2826 kernarg. 2827 2828 "hidden_global_offset_y" 2829 The OpenCL grid dispatch 2830 global offset for the Y 2831 dimension is passed in the 2832 kernarg. 2833 2834 "hidden_global_offset_z" 2835 The OpenCL grid dispatch 2836 global offset for the Z 2837 dimension is passed in the 2838 kernarg. 2839 2840 "hidden_none" 2841 An argument that is not used 2842 by the kernel. Space needs to 2843 be left for it, but it does 2844 not need to be set up. 2845 2846 "hidden_printf_buffer" 2847 A global address space pointer 2848 to the runtime printf buffer 2849 is passed in kernarg. 2850 2851 "hidden_hostcall_buffer" 2852 A global address space pointer 2853 to the runtime hostcall buffer 2854 is passed in kernarg. 2855 2856 "hidden_default_queue" 2857 A global address space pointer 2858 to the OpenCL device enqueue 2859 queue that should be used by 2860 the kernel by default is 2861 passed in the kernarg. 2862 2863 "hidden_completion_action" 2864 A global address space pointer 2865 to help link enqueued kernels into 2866 the ancestor tree for determining 2867 when the parent kernel has finished. 2868 2869 "hidden_multigrid_sync_arg" 2870 A global address space pointer for 2871 multi-grid synchronization is 2872 passed in the kernarg. 2873 2874 ".value_type" string Unused and deprecated. This should no longer 2875 be emitted, but is accepted for compatibility. 2876 2877 ".pointee_align" integer Alignment in bytes of pointee 2878 type for pointer type kernel 2879 argument. Must be a power 2880 of 2. Only present if 2881 ".value_kind" is 2882 "dynamic_shared_pointer". 2883 ".address_space" string Kernel argument address space 2884 qualifier. Only present if 2885 ".value_kind" is "global_buffer" or 2886 "dynamic_shared_pointer". Values 2887 are: 2888 2889 - "private" 2890 - "global" 2891 - "constant" 2892 - "local" 2893 - "generic" 2894 - "region" 2895 2896 .. TODO:: 2897 2898 Is "global_buffer" only "global" 2899 or "constant"? Is 2900 "dynamic_shared_pointer" always 2901 "local"? Can HCC allow "generic"? 2902 How can "private" or "region" 2903 ever happen? 2904 2905 ".access" string Kernel argument access 2906 qualifier. Only present if 2907 ".value_kind" is "image" or 2908 "pipe". Values 2909 are: 2910 2911 - "read_only" 2912 - "write_only" 2913 - "read_write" 2914 2915 .. TODO:: 2916 2917 Does this apply to 2918 "global_buffer"? 2919 2920 ".actual_access" string The actual memory accesses 2921 performed by the kernel on the 2922 kernel argument. Only present if 2923 ".value_kind" is "global_buffer", 2924 "image", or "pipe". This may be 2925 more restrictive than indicated 2926 by ".access" to reflect what the 2927 kernel actual does. If not 2928 present then the runtime must 2929 assume what is implied by 2930 ".access" and ".is_const" . Values 2931 are: 2932 2933 - "read_only" 2934 - "write_only" 2935 - "read_write" 2936 2937 ".is_const" boolean Indicates if the kernel argument 2938 is const qualified. Only present 2939 if ".value_kind" is 2940 "global_buffer". 2941 2942 ".is_restrict" boolean Indicates if the kernel argument 2943 is restrict qualified. Only 2944 present if ".value_kind" is 2945 "global_buffer". 2946 2947 ".is_volatile" boolean Indicates if the kernel argument 2948 is volatile qualified. Only 2949 present if ".value_kind" is 2950 "global_buffer". 2951 2952 ".is_pipe" boolean Indicates if the kernel argument 2953 is pipe qualified. Only present 2954 if ".value_kind" is "pipe". 2955 2956 .. TODO:: 2957 2958 Can "global_buffer" be pipe 2959 qualified? 2960 2961 ====================== ============== ========= ================================ 2962 2963.. 2964 2965Kernel Dispatch 2966~~~~~~~~~~~~~~~ 2967 2968The HSA architected queuing language (AQL) defines a user space memory interface 2969that can be used to control the dispatch of kernels, in an agent independent 2970way. An agent can have zero or more AQL queues created for it using the ROCm 2971runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the 2972*HSA Platform System Architecture Specification* [HSA]_ for the AQL queue 2973mechanics and packet layouts. 2974 2975The packet processor of a kernel agent is responsible for detecting and 2976dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2977packet processor is implemented by the hardware command processor (CP), 2978asynchronous dispatch controller (ADC) and shader processor input controller 2979(SPI). 2980 2981The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2982mode driver to initialize and register the AQL queue with CP. 2983 2984To dispatch a kernel the following actions are performed. This can occur in the 2985CPU host program, or from an HSA kernel executing on a GPU. 2986 29871. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2988 executed is obtained. 29892. A pointer to the kernel descriptor (see 2990 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2991 It must be for a kernel that is contained in a code object that that was 2992 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2993 associated. 29943. Space is allocated for the kernel arguments using the ROCm runtime allocator 2995 for a memory region with the kernarg property for the kernel agent that will 2996 execute the kernel. It must be at least 16-byte aligned. 29974. Kernel argument values are assigned to the kernel argument memory 2998 allocation. The layout is defined in the *HSA Programmer's Language 2999 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3000 kernel argument memory in the same way constant memory is accessed. (Note 3001 that the HSA specification allows an implementation to copy the kernel 3002 argument contents to another location that is accessed by the kernel.) 30035. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 3004 api uses 64-bit atomic operations to reserve space in the AQL queue for the 3005 packet. The packet must be set up, and the final write must use an atomic 3006 store release to set the packet kind to ensure the packet contents are 3007 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3008 notify the kernel agent that the AQL queue has been updated. These rules, and 3009 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3010 System Architecture Specification* [HSA]_. 30116. A kernel dispatch packet includes information about the actual dispatch, 3012 such as grid and work-group size, together with information from the code 3013 object about the kernel, such as segment sizes. The ROCm runtime queries on 3014 the kernel symbol can be used to obtain the code object values which are 3015 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 30167. CP executes micro-code and is responsible for detecting and setting up the 3017 GPU to execute the wavefronts of a kernel dispatch. 30188. CP ensures that when the a wavefront starts executing the kernel machine 3019 code, the scalar general purpose registers (SGPR) and vector general purpose 3020 registers (VGPR) are set up as required by the machine code. The required 3021 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3022 register state is defined in 3023 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 30249. The prolog of the kernel machine code (see 3025 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3026 before continuing executing the machine code that corresponds to the kernel. 302710. When the kernel dispatch has completed execution, CP signals the completion 3028 signal specified in the kernel dispatch packet if not 0. 3029 3030.. _amdgpu-amdhsa-memory-spaces: 3031 3032Memory Spaces 3033~~~~~~~~~~~~~ 3034 3035The memory space properties are: 3036 3037 .. table:: AMDHSA Memory Spaces 3038 :name: amdgpu-amdhsa-memory-spaces-table 3039 3040 ================= =========== ======== ======= ================== 3041 Memory Space Name HSA Segment Hardware Address NULL Value 3042 Name Name Size 3043 ================= =========== ======== ======= ================== 3044 Private private scratch 32 0x00000000 3045 Local group LDS 32 0xFFFFFFFF 3046 Global global global 64 0x0000000000000000 3047 Constant constant *same as 64 0x0000000000000000 3048 global* 3049 Generic flat flat 64 0x0000000000000000 3050 Region N/A GDS 32 *not implemented 3051 for AMDHSA* 3052 ================= =========== ======== ======= ================== 3053 3054The global and constant memory spaces both use global virtual addresses, which 3055are the same virtual address space used by the CPU. However, some virtual 3056addresses may only be accessible to the CPU, some only accessible by the GPU, 3057and some by both. 3058 3059Using the constant memory space indicates that the data will not change during 3060the execution of the kernel. This allows scalar read instructions to be 3061used. The vector and scalar L1 caches are invalidated of volatile data before 3062each kernel dispatch execution to allow constant memory to change values between 3063kernel dispatches. 3064 3065The local memory space uses the hardware Local Data Store (LDS) which is 3066automatically allocated when the hardware creates work-groups of wavefronts, and 3067freed when all the wavefronts of a work-group have terminated. The data store 3068(DS) instructions can be used to access it. 3069 3070The private memory space uses the hardware scratch memory support. If the kernel 3071uses scratch, then the hardware allocates memory that is accessed using 3072wavefront lane dword (4 byte) interleaving. The mapping used from private 3073address to physical address is: 3074 3075 ``wavefront-scratch-base + 3076 (private-address * wavefront-size * 4) + 3077 (wavefront-lane-id * 4)`` 3078 3079There are different ways that the wavefront scratch base address is determined 3080by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3081memory can be accessed in an interleaved manner using buffer instruction with 3082the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3083instructions, or by flat instructions. If each lane of a wavefront accesses the 3084same private address, the interleaving results in adjacent dwords being accessed 3085and hence requires fewer cache lines to be fetched. Multi-dword access is not 3086supported except by flat and scratch instructions in GFX9-GFX10. 3087 3088The generic address space uses the hardware flat address support available in 3089GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3090local apertures), that are outside the range of addressible global memory, to 3091map from a flat address to a private or local address. 3092 3093FLAT instructions can take a flat address and access global, private (scratch) 3094and group (LDS) memory depending in if the address is within one of the 3095aperture ranges. Flat access to scratch requires hardware aperture setup and 3096setup in the kernel prologue (see 3097:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3098hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3099:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3100 3101To convert between a segment address and a flat address the base address of the 3102apertures address can be used. For GFX7-GFX8 these are available in the 3103:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3104Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3105GFX9-GFX10 the aperture base addresses are directly available as inline constant 3106registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3107address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3108which makes it easier to convert from flat to segment or segment to flat. 3109 3110Image and Samplers 3111~~~~~~~~~~~~~~~~~~ 3112 3113Image and sample handles created by the ROCm runtime are 64-bit addresses of a 3114hardware 32-byte V# and 48 byte S# object respectively. In order to support the 3115HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 3116enumeration values for the queries that are not trivially deducible from the S# 3117representation. 3118 3119HSA Signals 3120~~~~~~~~~~~ 3121 3122HSA signal handles created by the ROCm runtime are 64-bit addresses of a 3123structure allocated in memory accessible from both the CPU and GPU. The 3124structure is defined by the ROCm runtime and subject to change between releases 3125(see [AMD-ROCm-github]_). 3126 3127.. _amdgpu-amdhsa-hsa-aql-queue: 3128 3129HSA AQL Queue 3130~~~~~~~~~~~~~ 3131 3132The HSA AQL queue structure is defined by the ROCm runtime and subject to change 3133between releases (see [AMD-ROCm-github]_). For some processors it contains 3134fields needed to implement certain language features such as the flat address 3135aperture bases. It also contains fields used by CP such as managing the 3136allocation of scratch memory. 3137 3138.. _amdgpu-amdhsa-kernel-descriptor: 3139 3140Kernel Descriptor 3141~~~~~~~~~~~~~~~~~ 3142 3143A kernel descriptor consists of the information needed by CP to initiate the 3144execution of a kernel, including the entry point address of the machine code 3145that implements the kernel. 3146 3147Code Object V3 Kernel Descriptor for GFX6-GFX10 (--amdhsa-code-object-version=3) 3148++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++=+++++++++ 3149 3150CP microcode requires the Kernel descriptor to be allocated on 64-byte 3151alignment. 3152 3153The fields used by CP for code objects before V3 also match those specified in 3154:ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 3155 3156 .. table:: Code Object V3 Kernel Descriptor for GFX6-GFX10 3157 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3 3158 3159 ======= ======= =============================== ============================ 3160 Bits Size Field Name Description 3161 ======= ======= =============================== ============================ 3162 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3163 address space memory 3164 required for a work-group 3165 in bytes. This does not 3166 include any dynamically 3167 allocated local address 3168 space memory that may be 3169 added when the kernel is 3170 dispatched. 3171 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3172 private address space 3173 memory required for a 3174 work-item in bytes. If 3175 is_dynamic_callstack is 1 3176 then additional space must 3177 be added to this value for 3178 the call stack. 3179 127:64 8 bytes Reserved, must be 0. 3180 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3181 negative) from base 3182 address of kernel 3183 descriptor to kernel's 3184 entry point instruction 3185 which must be 256 byte 3186 aligned. 3187 351:272 20 Reserved, must be 0. 3188 bytes 3189 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 3190 Reserved, must be 0. 3191 GFX10 3192 Compute Shader (CS) 3193 program settings used by 3194 CP to set up 3195 ``COMPUTE_PGM_RSRC3`` 3196 configuration 3197 register. See 3198 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3199 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3200 program settings used by 3201 CP to set up 3202 ``COMPUTE_PGM_RSRC1`` 3203 configuration 3204 register. See 3205 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3206 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3207 program settings used by 3208 CP to set up 3209 ``COMPUTE_PGM_RSRC2`` 3210 configuration 3211 register. See 3212 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3213 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3214 _BUFFER SGPR user data registers 3215 (see 3216 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3217 3218 The total number of SGPR 3219 user data registers 3220 requested must not exceed 3221 16 and match value in 3222 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3223 Any requests beyond 16 3224 will be ignored. 3225 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3226 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3227 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3228 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3229 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3230 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3231 _SIZE 3232 457:455 3 bits Reserved, must be 0. 3233 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3234 Reserved, must be 0. 3235 GFX10 3236 - If 0 execute in 3237 wavefront size 64 mode. 3238 - If 1 execute in 3239 native wavefront size 3240 32 mode. 3241 463:459 1 bit Reserved, must be 0. 3242 464 1 bit RESERVED_464 Deprecated, must be 0. 3243 467:465 3 bits Reserved, must be 0. 3244 468 1 bit RESERVED_468 Deprecated, must be 0. 3245 469:471 3 bits Reserved, must be 0. 3246 511:472 5 bytes Reserved, must be 0. 3247 512 **Total size 64 bytes.** 3248 ======= ==================================================================== 3249 3250.. 3251 3252 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3253 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3254 3255 ======= ======= =============================== =========================================================================== 3256 Bits Size Field Name Description 3257 ======= ======= =============================== =========================================================================== 3258 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3259 blocks used by each work-item; 3260 granularity is device 3261 specific: 3262 3263 GFX6-GFX9 3264 - vgprs_used 0..256 3265 - max(0, ceil(vgprs_used / 4) - 1) 3266 GFX10 (wavefront size 64) 3267 - max_vgpr 1..256 3268 - max(0, ceil(vgprs_used / 4) - 1) 3269 GFX10 (wavefront size 32) 3270 - max_vgpr 1..256 3271 - max(0, ceil(vgprs_used / 8) - 1) 3272 3273 Where vgprs_used is defined 3274 as the highest VGPR number 3275 explicitly referenced plus 3276 one. 3277 3278 Used by CP to set up 3279 ``COMPUTE_PGM_RSRC1.VGPRS``. 3280 3281 The 3282 :ref:`amdgpu-assembler` 3283 calculates this 3284 automatically for the 3285 selected processor from 3286 values provided to the 3287 `.amdhsa_kernel` directive 3288 by the 3289 `.amdhsa_next_free_vgpr` 3290 nested directive (see 3291 :ref:`amdhsa-kernel-directives-table`). 3292 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3293 blocks used by a wavefront; 3294 granularity is device 3295 specific: 3296 3297 GFX6-GFX8 3298 - sgprs_used 0..112 3299 - max(0, ceil(sgprs_used / 8) - 1) 3300 GFX9 3301 - sgprs_used 0..112 3302 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3303 GFX10 3304 Reserved, must be 0. 3305 (128 SGPRs always 3306 allocated.) 3307 3308 Where sgprs_used is 3309 defined as the highest 3310 SGPR number explicitly 3311 referenced plus one, plus 3312 a target specific number 3313 of additional special 3314 SGPRs for VCC, 3315 FLAT_SCRATCH (GFX7+) and 3316 XNACK_MASK (GFX8+), and 3317 any additional 3318 target specific 3319 limitations. It does not 3320 include the 16 SGPRs added 3321 if a trap handler is 3322 enabled. 3323 3324 The target specific 3325 limitations and special 3326 SGPR layout are defined in 3327 the hardware 3328 documentation, which can 3329 be found in the 3330 :ref:`amdgpu-processors` 3331 table. 3332 3333 Used by CP to set up 3334 ``COMPUTE_PGM_RSRC1.SGPRS``. 3335 3336 The 3337 :ref:`amdgpu-assembler` 3338 calculates this 3339 automatically for the 3340 selected processor from 3341 values provided to the 3342 `.amdhsa_kernel` directive 3343 by the 3344 `.amdhsa_next_free_sgpr` 3345 and `.amdhsa_reserve_*` 3346 nested directives (see 3347 :ref:`amdhsa-kernel-directives-table`). 3348 11:10 2 bits PRIORITY Must be 0. 3349 3350 Start executing wavefront 3351 at the specified priority. 3352 3353 CP is responsible for 3354 filling in 3355 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3356 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3357 with specified rounding 3358 mode for single (32 3359 bit) floating point 3360 precision floating point 3361 operations. 3362 3363 Floating point rounding 3364 mode values are defined in 3365 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3366 3367 Used by CP to set up 3368 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3369 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3370 with specified rounding 3371 denorm mode for half/double (16 3372 and 64-bit) floating point 3373 precision floating point 3374 operations. 3375 3376 Floating point rounding 3377 mode values are defined in 3378 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3379 3380 Used by CP to set up 3381 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3382 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3383 with specified denorm mode 3384 for single (32 3385 bit) floating point 3386 precision floating point 3387 operations. 3388 3389 Floating point denorm mode 3390 values are defined in 3391 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3392 3393 Used by CP to set up 3394 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3395 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3396 with specified denorm mode 3397 for half/double (16 3398 and 64-bit) floating point 3399 precision floating point 3400 operations. 3401 3402 Floating point denorm mode 3403 values are defined in 3404 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3405 3406 Used by CP to set up 3407 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3408 20 1 bit PRIV Must be 0. 3409 3410 Start executing wavefront 3411 in privilege trap handler 3412 mode. 3413 3414 CP is responsible for 3415 filling in 3416 ``COMPUTE_PGM_RSRC1.PRIV``. 3417 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3418 with DX10 clamp mode 3419 enabled. Used by the vector 3420 ALU to force DX10 style 3421 treatment of NaN's (when 3422 set, clamp NaN to zero, 3423 otherwise pass NaN 3424 through). 3425 3426 Used by CP to set up 3427 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3428 22 1 bit DEBUG_MODE Must be 0. 3429 3430 Start executing wavefront 3431 in single step mode. 3432 3433 CP is responsible for 3434 filling in 3435 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3436 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3437 with IEEE mode 3438 enabled. Floating point 3439 opcodes that support 3440 exception flag gathering 3441 will quiet and propagate 3442 signaling-NaN inputs per 3443 IEEE 754-2008. Min_dx10 and 3444 max_dx10 become IEEE 3445 754-2008 compliant due to 3446 signaling-NaN propagation 3447 and quieting. 3448 3449 Used by CP to set up 3450 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3451 24 1 bit BULKY Must be 0. 3452 3453 Only one work-group allowed 3454 to execute on a compute 3455 unit. 3456 3457 CP is responsible for 3458 filling in 3459 ``COMPUTE_PGM_RSRC1.BULKY``. 3460 25 1 bit CDBG_USER Must be 0. 3461 3462 Flag that can be used to 3463 control debugging code. 3464 3465 CP is responsible for 3466 filling in 3467 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3468 26 1 bit FP16_OVFL GFX6-GFX8 3469 Reserved, must be 0. 3470 GFX9-GFX10 3471 Wavefront starts execution 3472 with specified fp16 overflow 3473 mode. 3474 3475 - If 0, fp16 overflow generates 3476 +/-INF values. 3477 - If 1, fp16 overflow that is the 3478 result of an +/-INF input value 3479 or divide by 0 produces a +/-INF, 3480 otherwise clamps computed 3481 overflow to +/-MAX_FP16 as 3482 appropriate. 3483 3484 Used by CP to set up 3485 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3486 28:27 2 bits Reserved, must be 0. 3487 29 1 bit WGP_MODE GFX6-GFX9 3488 Reserved, must be 0. 3489 GFX10 3490 - If 0 execute work-groups in 3491 CU wavefront execution mode. 3492 - If 1 execute work-groups on 3493 in WGP wavefront execution mode. 3494 3495 See :ref:`amdgpu-amdhsa-memory-model`. 3496 3497 Used by CP to set up 3498 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3499 30 1 bit MEM_ORDERED GFX6-9 3500 Reserved, must be 0. 3501 GFX10 3502 Controls the behavior of the 3503 s_waitcnt's vmcnt and vscnt 3504 counters. 3505 3506 - If 0 vmcnt reports completion 3507 of load and atomic with return 3508 out of order with sample 3509 instructions, and the vscnt 3510 reports the completion of 3511 store and atomic without 3512 return in order. 3513 - If 1 vmcnt reports completion 3514 of load, atomic with return 3515 and sample instructions in 3516 order, and the vscnt reports 3517 the completion of store and 3518 atomic without return in order. 3519 3520 Used by CP to set up 3521 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3522 31 1 bit FWD_PROGRESS GFX6-9 3523 Reserved, must be 0. 3524 GFX10 3525 - If 0 execute SIMD wavefronts 3526 using oldest first policy. 3527 - If 1 execute SIMD wavefronts to 3528 ensure wavefronts will make some 3529 forward progress. 3530 3531 Used by CP to set up 3532 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3533 32 **Total size 4 bytes** 3534 ======= =================================================================================================================== 3535 3536.. 3537 3538 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3539 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3540 3541 ======= ======= =============================== =========================================================================== 3542 Bits Size Field Name Description 3543 ======= ======= =============================== =========================================================================== 3544 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3545 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3546 system register (see 3547 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3548 3549 Used by CP to set up 3550 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3551 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3552 user data registers 3553 requested. This number must 3554 match the number of user 3555 data registers enabled. 3556 3557 Used by CP to set up 3558 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3559 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3560 3561 This bit represents 3562 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3563 which is set by the CP if 3564 the runtime has installed a 3565 trap handler. 3566 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3567 system SGPR register for 3568 the work-group id in the X 3569 dimension (see 3570 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3571 3572 Used by CP to set up 3573 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3574 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3575 system SGPR register for 3576 the work-group id in the Y 3577 dimension (see 3578 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3579 3580 Used by CP to set up 3581 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3582 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3583 system SGPR register for 3584 the work-group id in the Z 3585 dimension (see 3586 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3587 3588 Used by CP to set up 3589 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3590 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3591 system SGPR register for 3592 work-group information (see 3593 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3594 3595 Used by CP to set up 3596 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3597 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3598 VGPR system registers used 3599 for the work-item ID. 3600 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3601 defines the values. 3602 3603 Used by CP to set up 3604 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3605 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3606 3607 Wavefront starts execution 3608 with address watch 3609 exceptions enabled which 3610 are generated when L1 has 3611 witnessed a thread access 3612 an *address of 3613 interest*. 3614 3615 CP is responsible for 3616 filling in the address 3617 watch bit in 3618 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3619 according to what the 3620 runtime requests. 3621 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3622 3623 Wavefront starts execution 3624 with memory violation 3625 exceptions exceptions 3626 enabled which are generated 3627 when a memory violation has 3628 occurred for this wavefront from 3629 L1 or LDS 3630 (write-to-read-only-memory, 3631 mis-aligned atomic, LDS 3632 address out of range, 3633 illegal address, etc.). 3634 3635 CP sets the memory 3636 violation bit in 3637 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3638 according to what the 3639 runtime requests. 3640 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3641 3642 CP uses the rounded value 3643 from the dispatch packet, 3644 not this value, as the 3645 dispatch may contain 3646 dynamically allocated group 3647 segment memory. CP writes 3648 directly to 3649 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3650 3651 Amount of group segment 3652 (LDS) to allocate for each 3653 work-group. Granularity is 3654 device specific: 3655 3656 GFX6: 3657 roundup(lds-size / (64 * 4)) 3658 GFX7-GFX10: 3659 roundup(lds-size / (128 * 4)) 3660 3661 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3662 _INVALID_OPERATION with specified exceptions 3663 enabled. 3664 3665 Used by CP to set up 3666 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3667 (set from bits 0..6). 3668 3669 IEEE 754 FP Invalid 3670 Operation 3671 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3672 _SOURCE input operands is a 3673 denormal number 3674 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3675 _DIVISION_BY_ZERO Zero 3676 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3677 _OVERFLOW 3678 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3679 _UNDERFLOW 3680 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3681 _INEXACT 3682 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3683 _ZERO (rcp_iflag_f32 instruction 3684 only) 3685 31 1 bit Reserved, must be 0. 3686 32 **Total size 4 bytes.** 3687 ======= =================================================================================================================== 3688 3689.. 3690 3691 .. table:: compute_pgm_rsrc3 for GFX10 3692 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3693 3694 ======= ======= =============================== =========================================================================== 3695 Bits Size Field Name Description 3696 ======= ======= =============================== =========================================================================== 3697 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3698 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3699 31:4 28 Reserved, must be 0. 3700 bits 3701 32 **Total size 4 bytes.** 3702 ======= =================================================================================================================== 3703 3704.. 3705 3706 .. table:: Floating Point Rounding Mode Enumeration Values 3707 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3708 3709 ====================================== ===== ============================== 3710 Enumeration Name Value Description 3711 ====================================== ===== ============================== 3712 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3713 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3714 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3715 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3716 ====================================== ===== ============================== 3717 3718.. 3719 3720 .. table:: Floating Point Denorm Mode Enumeration Values 3721 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3722 3723 ====================================== ===== ============================== 3724 Enumeration Name Value Description 3725 ====================================== ===== ============================== 3726 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3727 Denorms 3728 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3729 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3730 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3731 ====================================== ===== ============================== 3732 3733.. 3734 3735 .. table:: System VGPR Work-Item ID Enumeration Values 3736 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3737 3738 ======================================== ===== ============================ 3739 Enumeration Name Value Description 3740 ======================================== ===== ============================ 3741 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3742 ID. 3743 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3744 dimensions ID. 3745 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3746 dimensions ID. 3747 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3748 ======================================== ===== ============================ 3749 3750.. _amdgpu-amdhsa-initial-kernel-execution-state: 3751 3752Initial Kernel Execution State 3753~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3754 3755This section defines the register state that will be set up by the packet 3756processor prior to the start of execution of every wavefront. This is limited by 3757the constraints of the hardware controllers of CP/ADC/SPI. 3758 3759The order of the SGPR registers is defined, but the compiler can specify which 3760ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3761fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3762for enabled registers are dense starting at SGPR0: the first enabled register is 3763SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3764an SGPR number. 3765 3766The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3767all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3768using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3769actually initialized. These are then immediately followed by the System SGPRs 3770that are set up by ADC/SPI and can have different values for each wavefront of 3771the grid dispatch. 3772 3773SGPR register initial state is defined in 3774:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3775 3776 .. table:: SGPR Register Set Up Order 3777 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3778 3779 ========== ========================== ====== ============================== 3780 SGPR Order Name Number Description 3781 (kernel descriptor enable of 3782 field) SGPRs 3783 ========== ========================== ====== ============================== 3784 First Private Segment Buffer 4 V# that can be used, together 3785 (enable_sgpr_private with Scratch Wavefront Offset 3786 _segment_buffer) as an offset, to access the 3787 private memory space using a 3788 segment address. 3789 3790 CP uses the value provided by 3791 the runtime. 3792 then Dispatch Ptr 2 64-bit address of AQL dispatch 3793 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3794 actually executing. 3795 then Queue Ptr 2 64-bit address of amd_queue_t 3796 (enable_sgpr_queue_ptr) object for AQL queue on which 3797 the dispatch packet was 3798 queued. 3799 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3800 (enable_sgpr_kernarg segment. This is directly 3801 _segment_ptr) copied from the 3802 kernarg_address in the kernel 3803 dispatch packet. 3804 3805 Having CP load it once avoids 3806 loading it at the beginning of 3807 every wavefront. 3808 then Dispatch Id 2 64-bit Dispatch ID of the 3809 (enable_sgpr_dispatch_id) dispatch packet being 3810 executed. 3811 then Flat Scratch Init 2 This is 2 SGPRs: 3812 (enable_sgpr_flat_scratch 3813 _init) GFX6 3814 Not supported. 3815 GFX7-GFX8 3816 The first SGPR is a 32-bit 3817 byte offset from 3818 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3819 to per SPI base of memory 3820 for scratch for the queue 3821 executing the kernel 3822 dispatch. CP obtains this 3823 from the runtime. (The 3824 Scratch Segment Buffer base 3825 address is 3826 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3827 plus this offset.) The value 3828 of Scratch Wavefront Offset must 3829 be added to this offset by 3830 the kernel machine code, 3831 right shifted by 8, and 3832 moved to the FLAT_SCRATCH_HI 3833 SGPR register. 3834 FLAT_SCRATCH_HI corresponds 3835 to SGPRn-4 on GFX7, and 3836 SGPRn-6 on GFX8 (where SGPRn 3837 is the highest numbered SGPR 3838 allocated to the wavefront). 3839 FLAT_SCRATCH_HI is 3840 multiplied by 256 (as it is 3841 in units of 256 bytes) and 3842 added to 3843 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3844 to calculate the per wavefront 3845 FLAT SCRATCH BASE in flat 3846 memory instructions that 3847 access the scratch 3848 aperture. 3849 3850 The second SGPR is 32-bit 3851 byte size of a single 3852 work-item's scratch memory 3853 usage. CP obtains this from 3854 the runtime, and it is 3855 always a multiple of DWORD. 3856 CP checks that the value in 3857 the kernel dispatch packet 3858 Private Segment Byte Size is 3859 not larger and requests the 3860 runtime to increase the 3861 queue's scratch size if 3862 necessary. The kernel code 3863 must move it to 3864 FLAT_SCRATCH_LO which is 3865 SGPRn-3 on GFX7 and SGPRn-5 3866 on GFX8. FLAT_SCRATCH_LO is 3867 used as the FLAT SCRATCH 3868 SIZE in flat memory 3869 instructions. Having CP load 3870 it once avoids loading it at 3871 the beginning of every 3872 wavefront. 3873 GFX9-GFX10 3874 This is the 3875 64-bit base address of the 3876 per SPI scratch backing 3877 memory managed by SPI for 3878 the queue executing the 3879 kernel dispatch. CP obtains 3880 this from the runtime (and 3881 divides it if there are 3882 multiple Shader Arrays each 3883 with its own SPI). The value 3884 of Scratch Wavefront Offset must 3885 be added by the kernel 3886 machine code and the result 3887 moved to the FLAT_SCRATCH 3888 SGPR which is SGPRn-6 and 3889 SGPRn-5. It is used as the 3890 FLAT SCRATCH BASE in flat 3891 memory instructions. 3892 then Private Segment Size 1 The 32-bit byte size of a 3893 (enable_sgpr_private single 3894 work-item's 3895 scratch_segment_size) memory 3896 allocation. This is the 3897 value from the kernel 3898 dispatch packet Private 3899 Segment Byte Size rounded up 3900 by CP to a multiple of 3901 DWORD. 3902 3903 Having CP load it once avoids 3904 loading it at the beginning of 3905 every wavefront. 3906 3907 This is not used for 3908 GFX7-GFX8 since it is the same 3909 value as the second SGPR of 3910 Flat Scratch Init. However, it 3911 may be needed for GFX9-GFX10 which 3912 changes the meaning of the 3913 Flat Scratch Init value. 3914 then Grid Work-Group Count X 1 32-bit count of the number of 3915 (enable_sgpr_grid work-groups in the X dimension 3916 _workgroup_count_X) for the grid being 3917 executed. Computed from the 3918 fields in the kernel dispatch 3919 packet as ((grid_size.x + 3920 workgroup_size.x - 1) / 3921 workgroup_size.x). 3922 then Grid Work-Group Count Y 1 32-bit count of the number of 3923 (enable_sgpr_grid work-groups in the Y dimension 3924 _workgroup_count_Y && for the grid being 3925 less than 16 previous executed. Computed from the 3926 SGPRs) fields in the kernel dispatch 3927 packet as ((grid_size.y + 3928 workgroup_size.y - 1) / 3929 workgroupSize.y). 3930 3931 Only initialized if <16 3932 previous SGPRs initialized. 3933 then Grid Work-Group Count Z 1 32-bit count of the number of 3934 (enable_sgpr_grid work-groups in the Z dimension 3935 _workgroup_count_Z && for the grid being 3936 less than 16 previous executed. Computed from the 3937 SGPRs) fields in the kernel dispatch 3938 packet as ((grid_size.z + 3939 workgroup_size.z - 1) / 3940 workgroupSize.z). 3941 3942 Only initialized if <16 3943 previous SGPRs initialized. 3944 then Work-Group Id X 1 32-bit work-group id in X 3945 (enable_sgpr_workgroup_id dimension of grid for 3946 _X) wavefront. 3947 then Work-Group Id Y 1 32-bit work-group id in Y 3948 (enable_sgpr_workgroup_id dimension of grid for 3949 _Y) wavefront. 3950 then Work-Group Id Z 1 32-bit work-group id in Z 3951 (enable_sgpr_workgroup_id dimension of grid for 3952 _Z) wavefront. 3953 then Work-Group Info 1 {first_wavefront, 14'b0000, 3954 (enable_sgpr_workgroup ordered_append_term[10:0], 3955 _info) threadgroup_size_in_wavefronts[5:0]} 3956 then Scratch Wavefront Offset 1 32-bit byte offset from base 3957 (enable_sgpr_private of scratch base of queue 3958 _segment_wavefront_offset) executing the kernel 3959 dispatch. Must be used as an 3960 offset with Private 3961 segment address when using 3962 Scratch Segment Buffer. It 3963 must be used to set up FLAT 3964 SCRATCH for flat addressing 3965 (see 3966 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3967 ========== ========================== ====== ============================== 3968 3969The order of the VGPR registers is defined, but the compiler can specify which 3970ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3971fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3972for enabled registers are dense starting at VGPR0: the first enabled register is 3973VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3974VGPR number. 3975 3976VGPR register initial state is defined in 3977:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3978 3979 .. table:: VGPR Register Set Up Order 3980 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3981 3982 ========== ========================== ====== ============================== 3983 VGPR Order Name Number Description 3984 (kernel descriptor enable of 3985 field) VGPRs 3986 ========== ========================== ====== ============================== 3987 First Work-Item Id X 1 32-bit work-item id in X 3988 (Always initialized) dimension of work-group for 3989 wavefront lane. 3990 then Work-Item Id Y 1 32-bit work-item id in Y 3991 (enable_vgpr_workitem_id dimension of work-group for 3992 > 0) wavefront lane. 3993 then Work-Item Id Z 1 32-bit work-item id in Z 3994 (enable_vgpr_workitem_id dimension of work-group for 3995 > 1) wavefront lane. 3996 ========== ========================== ====== ============================== 3997 3998The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3999 40001. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4001 registers. 40022. Work-group Id registers X, Y, Z are set by ADC which supports any 4003 combination including none. 40043. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4005 its value cannot be included with the flat scratch init value which is per 4006 queue. 40074. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4008 or (X, Y, Z). 4009 4010Flat Scratch register pair are adjacent SGPRs so they can be moved as a 64-bit 4011value to the hardware required SGPRn-3 and SGPRn-4 respectively. 4012 4013The global segment can be accessed either using buffer instructions (GFX6 which 4014has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4015instructions (GFX9-GFX10). 4016 4017If buffer operations are used, then the compiler can generate a V# with the 4018following properties: 4019 4020* base address of 0 4021* no swizzle 4022* ATC: 1 if IOMMU present (such as APU) 4023* ptr64: 1 4024* MTYPE set to support memory coherence that matches the runtime (such as CC for 4025 APU and NC for dGPU). 4026 4027.. _amdgpu-amdhsa-kernel-prolog: 4028 4029Kernel Prolog 4030~~~~~~~~~~~~~ 4031 4032The compiler performs initialization in the kernel prologue depending on the 4033target and information about things like stack usage in the kernel and called 4034functions. Some of this initialization requires the compiler to request certain 4035User and System SGPRs be present in the 4036:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4037:ref:`amdgpu-amdhsa-kernel-descriptor`. 4038 4039.. _amdgpu-amdhsa-kernel-prolog-cfi: 4040 4041CFI 4042+++ 4043 40441. The CFI return address is undefined. 4045 40462. The CFI CFA is defined using an expression which evaluates to a location 4047 description that comprises one memory location description for the 4048 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4049 4050.. _amdgpu-amdhsa-kernel-prolog-m0: 4051 4052M0 4053++ 4054 4055GFX6-GFX8 4056 The M0 register must be initialized with a value at least the total LDS size 4057 if the kernel may access LDS via DS or flat operations. Total LDS size is 4058 available in dispatch packet. For M0, it is also possible to use maximum 4059 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4060 GFX7-GFX8). 4061GFX9-GFX10 4062 The M0 register is not used for range checking LDS accesses and so does not 4063 need to be initialized in the prolog. 4064 4065.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4066 4067Stack Pointer 4068+++++++++++++ 4069 4070If the kernel has function calls it must set up the ABI stack pointer described 4071in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4072SGPR32 to the unswizzled scratch offset of the address past the last local 4073allocation. 4074 4075.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4076 4077Frame Pointer 4078+++++++++++++ 4079 4080If the kernel needs a frame pointer for the reasons defined in 4081``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4082kernel prolog. If a frame pointer is not required then all uses of the frame 4083pointer are replaced with immediate ``0`` offsets. 4084 4085.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4086 4087Flat Scratch 4088++++++++++++ 4089 4090If the kernel or any function it calls may use flat operations to access 4091scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4092(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4093uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4094:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4095 4096GFX6 4097 Flat scratch is not supported. 4098 4099GFX7-GFX8 4100 4101 1. The low word of Flat Scratch Init is 32-bit byte offset from 4102 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4103 being managed by SPI for the queue executing the kernel dispatch. This is 4104 the same value used in the Scratch Segment Buffer V# base address. The 4105 prolog must add the value of Scratch Wavefront Offset to get the 4106 wavefront's byte scratch backing memory offset from 4107 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 4108 bytes, the offset must be right shifted by 8 before moving into 4109 FLAT_SCRATCH_LO. 4110 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4111 work-items scratch memory usage. This is directly loaded from the kernel 4112 dispatch packet Private Segment Byte Size and rounded up to a multiple of 4113 DWORD. Having CP load it once avoids loading it at the beginning of every 4114 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 4115 SCRATCH SIZE. 4116 4117GFX9-GFX10 4118 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4119 memory being managed by SPI for the queue executing the kernel dispatch. The 4120 prolog must add the value of Scratch Wavefront Offset and moved to the 4121 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 4122 instructions. 4123 4124.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4125 4126Private Segment Buffer 4127++++++++++++++++++++++ 4128 4129A set of four SGPRs beginning at a four-aligned SGPR index are always selected 4130to serve as the scratch V# for the kernel as follows: 4131 4132 - If it is known during instruction selection that there is stack usage, 4133 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4134 optimizations are disabled (``-O0``), if stack objects already exist (for 4135 locals, etc.), or if there are any function calls. 4136 4137 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4138 are reserved for the tentative scratch V#. These will be used if it is 4139 determined that spilling is needed. 4140 4141 - If no use is made of the tentative scratch V#, then it is unreserved, 4142 and the register count is determined ignoring it. 4143 - If use is made of the tentative scratch V#, then its register numbers 4144 are shifted to the first four-aligned SGPR index after the highest one 4145 allocated by the register allocator, and all uses are updated. The 4146 register count includes them in the shifted location. 4147 - In either case, if the processor has the SGPR allocation bug, the 4148 tentative allocation is not shifted or unreserved in order to ensure 4149 the register count is higher to workaround the bug. 4150 4151 .. note:: 4152 4153 This approach of using a tentative scratch V# and shifting the register 4154 numbers if used avoids having to perform register allocation a second 4155 time if the tentative V# is eliminated. This is more efficient and 4156 avoids the problem that the second register allocation may perform 4157 spilling which will fail as there is no longer a scratch V#. 4158 4159When the kernel prolog code is being emitted it is known whether the scratch V# 4160described above is actually used. If it is, the prolog code must set it up by 4161copying the Private Segment Buffer to the scratch V# registers and then adding 4162the Private Segment Wavefront Offset to the queue base address in the V#. The 4163result is a V# with a base address pointing to the beginning of the wavefront 4164scratch backing memory. 4165 4166The Private Segment Buffer is always requested, but the Private Segment 4167Wavefront Offset is only requested if it is used (see 4168:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4169 4170.. _amdgpu-amdhsa-memory-model: 4171 4172Memory Model 4173~~~~~~~~~~~~ 4174 4175This section describes the mapping of the LLVM memory model onto AMDGPU machine 4176code (see :ref:`memmodel`). 4177 4178The AMDGPU backend supports the memory synchronization scopes specified in 4179:ref:`amdgpu-memory-scopes`. 4180 4181The code sequences used to implement the memory model specify the order of 4182instructions that a single thread must execute. The ``s_waitcnt`` and cache 4183management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4184to other memory instructions executed by the same thread. This allows them to be 4185moved earlier or later which can allow them to be combined with other instances 4186of the same instruction, or hoisted/sunk out of loops to improve performance. 4187Only the instructions related to the memory model are given; additional 4188``s_waitcnt`` instructions are required to ensure registers are defined before 4189being used. These may be able to be combined with the memory model ``s_waitcnt`` 4190instructions as described above. 4191 4192The AMDGPU backend supports the following memory models: 4193 4194 HSA Memory Model [HSA]_ 4195 The HSA memory model uses a single happens-before relation for all address 4196 spaces (see :ref:`amdgpu-address-spaces`). 4197 OpenCL Memory Model [OpenCL]_ 4198 The OpenCL memory model which has separate happens-before relations for the 4199 global and local address spaces. Only a fence specifying both global and 4200 local address space, and seq_cst instructions join the relationships. Since 4201 the LLVM ``memfence`` instruction does not allow an address space to be 4202 specified the OpenCL fence has to conservatively assume both local and 4203 global address space was specified. However, optimizations can often be 4204 done to eliminate the additional ``s_waitcnt`` instructions when there are 4205 no intervening memory instructions which access the corresponding address 4206 space. The code sequences in the table indicate what can be omitted for the 4207 OpenCL memory. The target triple environment is used to determine if the 4208 source language is OpenCL (see :ref:`amdgpu-opencl`). 4209 4210``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4211operations. 4212 4213``buffer/global/flat_load/store/atomic`` instructions to global memory are 4214termed vector memory operations. 4215 4216Private address space uses ``buffer_load/store`` using the scratch V# 4217(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4218is accessing the memory, atomic memory orderings are not meaningful, and all 4219accesses are treated as non-atomic. 4220 4221Constant address space uses ``buffer/global_load`` instructions (or equivalent 4222scalar memory instructions). Since the constant address space contents do not 4223change during the execution of a kernel dispatch it is not legal to perform 4224stores, and atomic memory orderings are not meaningful, and all accesses are 4225treated as non-atomic. 4226 4227A memory synchronization scope wider than work-group is not meaningful for the 4228group (LDS) address space and is treated as work-group. 4229 4230The memory model does not support the region address space which is treated as 4231non-atomic. 4232 4233Acquire memory ordering is not meaningful on store atomic instructions and is 4234treated as non-atomic. 4235 4236Release memory ordering is not meaningful on load atomic instructions and is 4237treated a non-atomic. 4238 4239Acquire-release memory ordering is not meaningful on load or store atomic 4240instructions and is treated as acquire and release respectively. 4241 4242The memory order also adds the single thread optimization constraints defined in 4243table 4244:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4245 4246 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4247 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4248 4249 ============ ============================================================== 4250 LLVM Memory Optimization Constraints 4251 Ordering 4252 ============ ============================================================== 4253 unordered *none* 4254 monotonic *none* 4255 acquire - If a load atomic/atomicrmw then no following load/load 4256 atomic/store/store atomic/atomicrmw/fence instruction can be 4257 moved before the acquire. 4258 - If a fence then same as load atomic, plus no preceding 4259 associated fence-paired-atomic can be moved after the fence. 4260 release - If a store atomic/atomicrmw then no preceding load/load 4261 atomic/store/store atomic/atomicrmw/fence instruction can be 4262 moved after the release. 4263 - If a fence then same as store atomic, plus no following 4264 associated fence-paired-atomic can be moved before the 4265 fence. 4266 acq_rel Same constraints as both acquire and release. 4267 seq_cst - If a load atomic then same constraints as acquire, plus no 4268 preceding sequentially consistent load atomic/store 4269 atomic/atomicrmw/fence instruction can be moved after the 4270 seq_cst. 4271 - If a store atomic then the same constraints as release, plus 4272 no following sequentially consistent load atomic/store 4273 atomic/atomicrmw/fence instruction can be moved before the 4274 seq_cst. 4275 - If an atomicrmw/fence then same constraints as acq_rel. 4276 ============ ============================================================== 4277 4278The code sequences used to implement the memory model are defined in the 4279following sections: 4280 4281* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4282* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4283 4284.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4285 4286Memory Model GFX6-GFX9 4287++++++++++++++++++++++ 4288 4289For GFX6-GFX9: 4290 4291* Each agent has multiple shader arrays (SA). 4292* Each SA has multiple compute units (CU). 4293* Each CU has multiple SIMDs that execute wavefronts. 4294* The wavefronts for a single work-group are executed in the same CU but may be 4295 executed by different SIMDs. 4296* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4297 executing on it. 4298* All LDS operations of a CU are performed as wavefront wide operations in a 4299 global order and involve no caching. Completion is reported to a wavefront in 4300 execution order. 4301* The LDS memory has multiple request queues shared by the SIMDs of a 4302 CU. Therefore, the LDS operations performed by different wavefronts of a 4303 work-group can be reordered relative to each other, which can result in 4304 reordering the visibility of vector memory operations with respect to LDS 4305 operations of other wavefronts in the same work-group. A ``s_waitcnt 4306 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4307 vector memory operations between wavefronts of a work-group, but not between 4308 operations performed by the same wavefront. 4309* The vector memory operations are performed as wavefront wide operations and 4310 completion is reported to a wavefront in execution order. The exception is 4311 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4312 vector memory order if they access LDS memory, and out of LDS operation order 4313 if they access global memory. 4314* The vector memory operations access a single vector L1 cache shared by all 4315 SIMDs a CU. Therefore, no special action is required for coherence between the 4316 lanes of a single wavefront, or for coherence between wavefronts in the same 4317 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4318 wavefronts executing in different work-groups as they may be executing on 4319 different CUs. 4320* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4321 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4322 scalar operations are used in a restricted way so do not impact the memory 4323 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4324* The vector and scalar memory operations use an L2 cache shared by all CUs on 4325 the same agent. 4326* The L2 cache has independent channels to service disjoint ranges of virtual 4327 addresses. 4328* Each CU has a separate request queue per channel. Therefore, the vector and 4329 scalar memory operations performed by wavefronts executing in different 4330 work-groups (which may be executing on different CUs) of an agent can be 4331 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4332 ensure synchronization between vector memory operations of different CUs. It 4333 ensures a previous vector memory operation has completed before executing a 4334 subsequent vector memory or LDS operation and so can be used to meet the 4335 requirements of acquire and release. 4336* The L2 cache can be kept coherent with other agents on some targets, or ranges 4337 of virtual addresses can be set up to bypass it to ensure system coherence. 4338 4339Scalar memory operations are only used to access memory that is proven to not 4340change during the execution of the kernel dispatch. This includes constant 4341address space and global address space for program scope ``const`` variables. 4342Therefore, the kernel machine code does not have to maintain the scalar cache to 4343ensure it is coherent with the vector caches. The scalar and vector caches are 4344invalidated between kernel dispatches by CP since constant address space data 4345may change between kernel dispatch executions. See 4346:ref:`amdgpu-amdhsa-memory-spaces`. 4347 4348The one exception is if scalar writes are used to spill SGPR registers. In this 4349case the AMDGPU backend ensures the memory location used to spill is never 4350accessed by vector memory operations at the same time. If scalar writes are used 4351then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4352return since the locations may be used for vector memory instructions by a 4353future wavefront that uses the same scratch area, or a function call that 4354creates a frame at the same address, respectively. There is no need for a 4355``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4356 4357For kernarg backing memory: 4358 4359* CP invalidates the L1 cache at the start of each kernel dispatch. 4360* On dGPU the kernarg backing memory is allocated in host memory accessed as 4361 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4362 causes it to be treated as non-volatile and so is not invalidated by 4363 ``*_vol``. 4364* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4365 and so the L2 cache will be coherent with the CPU and other agents. 4366 4367Scratch backing memory (which is used for the private address space) is accessed 4368with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4369only accessed by a single thread, and is always write-before-read, there is 4370never a need to invalidate these entries from the L1 cache. Hence all cache 4371invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4372 4373The code sequences used to implement the memory model for GFX6-GFX9 are defined 4374in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4375 4376 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4377 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4378 4379 ============ ============ ============== ========== ================================ 4380 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4381 Ordering Sync Scope Address GFX6-9 4382 Space 4383 ============ ============ ============== ========== ================================ 4384 **Non-Atomic** 4385 ------------------------------------------------------------------------------------ 4386 load *none* *none* - global - !volatile & !nontemporal 4387 - generic 4388 - private 1. buffer/global/flat_load 4389 - constant 4390 - volatile & !nontemporal 4391 4392 1. buffer/global/flat_load 4393 glc=1 4394 4395 - nontemporal 4396 4397 1. buffer/global/flat_load 4398 glc=1 slc=1 4399 4400 load *none* *none* - local 1. ds_load 4401 store *none* *none* - global - !nontemporal 4402 - generic 4403 - private 1. buffer/global/flat_store 4404 - constant 4405 - nontemporal 4406 4407 1. buffer/global/flat_store 4408 glc=1 slc=1 4409 4410 store *none* *none* - local 1. ds_store 4411 **Unordered Atomic** 4412 ------------------------------------------------------------------------------------ 4413 load atomic unordered *any* *any* *Same as non-atomic*. 4414 store atomic unordered *any* *any* *Same as non-atomic*. 4415 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4416 **Monotonic Atomic** 4417 ------------------------------------------------------------------------------------ 4418 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4419 - wavefront - local 4420 - workgroup - generic 4421 load atomic monotonic - agent - global 1. buffer/global/flat_load 4422 - system - generic glc=1 4423 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4424 - wavefront - generic 4425 - workgroup 4426 - agent 4427 - system 4428 store atomic monotonic - singlethread - local 1. ds_store 4429 - wavefront 4430 - workgroup 4431 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4432 - wavefront - generic 4433 - workgroup 4434 - agent 4435 - system 4436 atomicrmw monotonic - singlethread - local 1. ds_atomic 4437 - wavefront 4438 - workgroup 4439 **Acquire Atomic** 4440 ------------------------------------------------------------------------------------ 4441 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4442 - wavefront - local 4443 - generic 4444 load atomic acquire - workgroup - global 1. buffer/global_load 4445 load atomic acquire - workgroup - local 1. ds/flat_load 4446 - generic 2. s_waitcnt lgkmcnt(0) 4447 4448 - If OpenCL, omit. 4449 - Must happen before 4450 any following 4451 global/generic 4452 load/load 4453 atomic/store/store 4454 atomic/atomicrmw. 4455 - Ensures any 4456 following global 4457 data read is no 4458 older than a local load 4459 atomic value being 4460 acquired. 4461 4462 load atomic acquire - agent - global 1. buffer/global_load 4463 - system glc=1 4464 2. s_waitcnt vmcnt(0) 4465 4466 - Must happen before 4467 following 4468 buffer_wbinvl1_vol. 4469 - Ensures the load 4470 has completed 4471 before invalidating 4472 the cache. 4473 4474 3. buffer_wbinvl1_vol 4475 4476 - Must happen before 4477 any following 4478 global/generic 4479 load/load 4480 atomic/atomicrmw. 4481 - Ensures that 4482 following 4483 loads will not see 4484 stale global data. 4485 4486 load atomic acquire - agent - generic 1. flat_load glc=1 4487 - system 2. s_waitcnt vmcnt(0) & 4488 lgkmcnt(0) 4489 4490 - If OpenCL omit 4491 lgkmcnt(0). 4492 - Must happen before 4493 following 4494 buffer_wbinvl1_vol. 4495 - Ensures the flat_load 4496 has completed 4497 before invalidating 4498 the cache. 4499 4500 3. buffer_wbinvl1_vol 4501 4502 - Must happen before 4503 any following 4504 global/generic 4505 load/load 4506 atomic/atomicrmw. 4507 - Ensures that 4508 following loads 4509 will not see stale 4510 global data. 4511 4512 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 4513 - wavefront - local 4514 - generic 4515 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 4516 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 4517 - generic 2. s_waitcnt lgkmcnt(0) 4518 4519 - If OpenCL, omit. 4520 - Must happen before 4521 any following 4522 global/generic 4523 load/load 4524 atomic/store/store 4525 atomic/atomicrmw. 4526 - Ensures any 4527 following global 4528 data read is no 4529 older than a local 4530 atomicrmw value 4531 being acquired. 4532 4533 atomicrmw acquire - agent - global 1. buffer/global_atomic 4534 - system 2. s_waitcnt vmcnt(0) 4535 4536 - Must happen before 4537 following 4538 buffer_wbinvl1_vol. 4539 - Ensures the 4540 atomicrmw has 4541 completed before 4542 invalidating the 4543 cache. 4544 4545 3. buffer_wbinvl1_vol 4546 4547 - Must happen before 4548 any following 4549 global/generic 4550 load/load 4551 atomic/atomicrmw. 4552 - Ensures that 4553 following loads 4554 will not see stale 4555 global data. 4556 4557 atomicrmw acquire - agent - generic 1. flat_atomic 4558 - system 2. s_waitcnt vmcnt(0) & 4559 lgkmcnt(0) 4560 4561 - If OpenCL, omit 4562 lgkmcnt(0). 4563 - Must happen before 4564 following 4565 buffer_wbinvl1_vol. 4566 - Ensures the 4567 atomicrmw has 4568 completed before 4569 invalidating the 4570 cache. 4571 4572 3. buffer_wbinvl1_vol 4573 4574 - Must happen before 4575 any following 4576 global/generic 4577 load/load 4578 atomic/atomicrmw. 4579 - Ensures that 4580 following loads 4581 will not see stale 4582 global data. 4583 4584 fence acquire - singlethread *none* *none* 4585 - wavefront 4586 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 4587 4588 - If OpenCL and 4589 address space is 4590 not generic, omit. 4591 - However, since LLVM 4592 currently has no 4593 address space on 4594 the fence need to 4595 conservatively 4596 always generate. If 4597 fence had an 4598 address space then 4599 set to address 4600 space of OpenCL 4601 fence flag, or to 4602 generic if both 4603 local and global 4604 flags are 4605 specified. 4606 - Must happen after 4607 any preceding 4608 local/generic load 4609 atomic/atomicrmw 4610 with an equal or 4611 wider sync scope 4612 and memory ordering 4613 stronger than 4614 unordered (this is 4615 termed the 4616 fence-paired-atomic). 4617 - Must happen before 4618 any following 4619 global/generic 4620 load/load 4621 atomic/store/store 4622 atomic/atomicrmw. 4623 - Ensures any 4624 following global 4625 data read is no 4626 older than the 4627 value read by the 4628 fence-paired-atomic. 4629 4630 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 4631 - system vmcnt(0) 4632 4633 - If OpenCL and 4634 address space is 4635 not generic, omit 4636 lgkmcnt(0). 4637 - However, since LLVM 4638 currently has no 4639 address space on 4640 the fence need to 4641 conservatively 4642 always generate 4643 (see comment for 4644 previous fence). 4645 - Could be split into 4646 separate s_waitcnt 4647 vmcnt(0) and 4648 s_waitcnt 4649 lgkmcnt(0) to allow 4650 them to be 4651 independently moved 4652 according to the 4653 following rules. 4654 - s_waitcnt vmcnt(0) 4655 must happen after 4656 any preceding 4657 global/generic load 4658 atomic/atomicrmw 4659 with an equal or 4660 wider sync scope 4661 and memory ordering 4662 stronger than 4663 unordered (this is 4664 termed the 4665 fence-paired-atomic). 4666 - s_waitcnt lgkmcnt(0) 4667 must happen after 4668 any preceding 4669 local/generic load 4670 atomic/atomicrmw 4671 with an equal or 4672 wider sync scope 4673 and memory ordering 4674 stronger than 4675 unordered (this is 4676 termed the 4677 fence-paired-atomic). 4678 - Must happen before 4679 the following 4680 buffer_wbinvl1_vol. 4681 - Ensures that the 4682 fence-paired atomic 4683 has completed 4684 before invalidating 4685 the 4686 cache. Therefore 4687 any following 4688 locations read must 4689 be no older than 4690 the value read by 4691 the 4692 fence-paired-atomic. 4693 4694 2. buffer_wbinvl1_vol 4695 4696 - Must happen before any 4697 following global/generic 4698 load/load 4699 atomic/store/store 4700 atomic/atomicrmw. 4701 - Ensures that 4702 following loads 4703 will not see stale 4704 global data. 4705 4706 **Release Atomic** 4707 ------------------------------------------------------------------------------------ 4708 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 4709 - wavefront - local 4710 - generic 4711 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 4712 - generic 4713 - If OpenCL, omit. 4714 - Must happen after 4715 any preceding 4716 local/generic 4717 load/store/load 4718 atomic/store 4719 atomic/atomicrmw. 4720 - Must happen before 4721 the following 4722 store. 4723 - Ensures that all 4724 memory operations 4725 to local have 4726 completed before 4727 performing the 4728 store that is being 4729 released. 4730 4731 2. buffer/global/flat_store 4732 store atomic release - workgroup - local 1. ds_store 4733 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 4734 - system - generic vmcnt(0) 4735 4736 - If OpenCL and 4737 address space is 4738 not generic, omit 4739 lgkmcnt(0). 4740 - Could be split into 4741 separate s_waitcnt 4742 vmcnt(0) and 4743 s_waitcnt 4744 lgkmcnt(0) to allow 4745 them to be 4746 independently moved 4747 according to the 4748 following rules. 4749 - s_waitcnt vmcnt(0) 4750 must happen after 4751 any preceding 4752 global/generic 4753 load/store/load 4754 atomic/store 4755 atomic/atomicrmw. 4756 - s_waitcnt lgkmcnt(0) 4757 must happen after 4758 any preceding 4759 local/generic 4760 load/store/load 4761 atomic/store 4762 atomic/atomicrmw. 4763 - Must happen before 4764 the following 4765 store. 4766 - Ensures that all 4767 memory operations 4768 to memory have 4769 completed before 4770 performing the 4771 store that is being 4772 released. 4773 4774 2. buffer/global/flat_store 4775 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 4776 - wavefront - local 4777 - generic 4778 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 4779 - generic 4780 - If OpenCL, omit. 4781 - Must happen after 4782 any preceding 4783 local/generic 4784 load/store/load 4785 atomic/store 4786 atomic/atomicrmw. 4787 - Must happen before 4788 the following 4789 atomicrmw. 4790 - Ensures that all 4791 memory operations 4792 to local have 4793 completed before 4794 performing the 4795 atomicrmw that is 4796 being released. 4797 4798 2. buffer/global/flat_atomic 4799 atomicrmw release - workgroup - local 1. ds_atomic 4800 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 4801 - system - generic vmcnt(0) 4802 4803 - If OpenCL, omit 4804 lgkmcnt(0). 4805 - Could be split into 4806 separate s_waitcnt 4807 vmcnt(0) and 4808 s_waitcnt 4809 lgkmcnt(0) to allow 4810 them to be 4811 independently moved 4812 according to the 4813 following rules. 4814 - s_waitcnt vmcnt(0) 4815 must happen after 4816 any preceding 4817 global/generic 4818 load/store/load 4819 atomic/store 4820 atomic/atomicrmw. 4821 - s_waitcnt lgkmcnt(0) 4822 must happen after 4823 any preceding 4824 local/generic 4825 load/store/load 4826 atomic/store 4827 atomic/atomicrmw. 4828 - Must happen before 4829 the following 4830 atomicrmw. 4831 - Ensures that all 4832 memory operations 4833 to global and local 4834 have completed 4835 before performing 4836 the atomicrmw that 4837 is being released. 4838 4839 2. buffer/global/flat_atomic 4840 fence release - singlethread *none* *none* 4841 - wavefront 4842 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 4843 4844 - If OpenCL and 4845 address space is 4846 not generic, omit. 4847 - However, since LLVM 4848 currently has no 4849 address space on 4850 the fence need to 4851 conservatively 4852 always generate. If 4853 fence had an 4854 address space then 4855 set to address 4856 space of OpenCL 4857 fence flag, or to 4858 generic if both 4859 local and global 4860 flags are 4861 specified. 4862 - Must happen after 4863 any preceding 4864 local/generic 4865 load/load 4866 atomic/store/store 4867 atomic/atomicrmw. 4868 - Must happen before 4869 any following store 4870 atomic/atomicrmw 4871 with an equal or 4872 wider sync scope 4873 and memory ordering 4874 stronger than 4875 unordered (this is 4876 termed the 4877 fence-paired-atomic). 4878 - Ensures that all 4879 memory operations 4880 to local have 4881 completed before 4882 performing the 4883 following 4884 fence-paired-atomic. 4885 4886 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 4887 - system vmcnt(0) 4888 4889 - If OpenCL and 4890 address space is 4891 not generic, omit 4892 lgkmcnt(0). 4893 - If OpenCL and 4894 address space is 4895 local, omit 4896 vmcnt(0). 4897 - However, since LLVM 4898 currently has no 4899 address space on 4900 the fence need to 4901 conservatively 4902 always generate. If 4903 fence had an 4904 address space then 4905 set to address 4906 space of OpenCL 4907 fence flag, or to 4908 generic if both 4909 local and global 4910 flags are 4911 specified. 4912 - Could be split into 4913 separate s_waitcnt 4914 vmcnt(0) and 4915 s_waitcnt 4916 lgkmcnt(0) to allow 4917 them to be 4918 independently moved 4919 according to the 4920 following rules. 4921 - s_waitcnt vmcnt(0) 4922 must happen after 4923 any preceding 4924 global/generic 4925 load/store/load 4926 atomic/store 4927 atomic/atomicrmw. 4928 - s_waitcnt lgkmcnt(0) 4929 must happen after 4930 any preceding 4931 local/generic 4932 load/store/load 4933 atomic/store 4934 atomic/atomicrmw. 4935 - Must happen before 4936 any following store 4937 atomic/atomicrmw 4938 with an equal or 4939 wider sync scope 4940 and memory ordering 4941 stronger than 4942 unordered (this is 4943 termed the 4944 fence-paired-atomic). 4945 - Ensures that all 4946 memory operations 4947 have 4948 completed before 4949 performing the 4950 following 4951 fence-paired-atomic. 4952 4953 **Acquire-Release Atomic** 4954 ------------------------------------------------------------------------------------ 4955 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 4956 - wavefront - local 4957 - generic 4958 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 4959 4960 - If OpenCL, omit. 4961 - Must happen after 4962 any preceding 4963 local/generic 4964 load/store/load 4965 atomic/store 4966 atomic/atomicrmw. 4967 - Must happen before 4968 the following 4969 atomicrmw. 4970 - Ensures that all 4971 memory operations 4972 to local have 4973 completed before 4974 performing the 4975 atomicrmw that is 4976 being released. 4977 4978 2. buffer/global_atomic 4979 4980 atomicrmw acq_rel - workgroup - local 1. ds_atomic 4981 2. s_waitcnt lgkmcnt(0) 4982 4983 - If OpenCL, omit. 4984 - Must happen before 4985 any following 4986 global/generic 4987 load/load 4988 atomic/store/store 4989 atomic/atomicrmw. 4990 - Ensures any 4991 following global 4992 data read is no 4993 older than the local load 4994 atomic value being 4995 acquired. 4996 4997 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 4998 4999 - If OpenCL, omit. 5000 - Must happen after 5001 any preceding 5002 local/generic 5003 load/store/load 5004 atomic/store 5005 atomic/atomicrmw. 5006 - Must happen before 5007 the following 5008 atomicrmw. 5009 - Ensures that all 5010 memory operations 5011 to local have 5012 completed before 5013 performing the 5014 atomicrmw that is 5015 being released. 5016 5017 2. flat_atomic 5018 3. s_waitcnt lgkmcnt(0) 5019 5020 - If OpenCL, omit. 5021 - Must happen before 5022 any following 5023 global/generic 5024 load/load 5025 atomic/store/store 5026 atomic/atomicrmw. 5027 - Ensures any 5028 following global 5029 data read is no 5030 older than a local load 5031 atomic value being 5032 acquired. 5033 5034 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5035 - system vmcnt(0) 5036 5037 - If OpenCL, omit 5038 lgkmcnt(0). 5039 - Could be split into 5040 separate s_waitcnt 5041 vmcnt(0) and 5042 s_waitcnt 5043 lgkmcnt(0) to allow 5044 them to be 5045 independently moved 5046 according to the 5047 following rules. 5048 - s_waitcnt vmcnt(0) 5049 must happen after 5050 any preceding 5051 global/generic 5052 load/store/load 5053 atomic/store 5054 atomic/atomicrmw. 5055 - s_waitcnt lgkmcnt(0) 5056 must happen after 5057 any preceding 5058 local/generic 5059 load/store/load 5060 atomic/store 5061 atomic/atomicrmw. 5062 - Must happen before 5063 the following 5064 atomicrmw. 5065 - Ensures that all 5066 memory operations 5067 to global have 5068 completed before 5069 performing the 5070 atomicrmw that is 5071 being released. 5072 5073 2. buffer/global_atomic 5074 3. s_waitcnt vmcnt(0) 5075 5076 - Must happen before 5077 following 5078 buffer_wbinvl1_vol. 5079 - Ensures the 5080 atomicrmw has 5081 completed before 5082 invalidating the 5083 cache. 5084 5085 4. buffer_wbinvl1_vol 5086 5087 - Must happen before 5088 any following 5089 global/generic 5090 load/load 5091 atomic/atomicrmw. 5092 - Ensures that 5093 following loads 5094 will not see stale 5095 global data. 5096 5097 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5098 - system vmcnt(0) 5099 5100 - If OpenCL, omit 5101 lgkmcnt(0). 5102 - Could be split into 5103 separate s_waitcnt 5104 vmcnt(0) and 5105 s_waitcnt 5106 lgkmcnt(0) to allow 5107 them to be 5108 independently moved 5109 according to the 5110 following rules. 5111 - s_waitcnt vmcnt(0) 5112 must happen after 5113 any preceding 5114 global/generic 5115 load/store/load 5116 atomic/store 5117 atomic/atomicrmw. 5118 - s_waitcnt lgkmcnt(0) 5119 must happen after 5120 any preceding 5121 local/generic 5122 load/store/load 5123 atomic/store 5124 atomic/atomicrmw. 5125 - Must happen before 5126 the following 5127 atomicrmw. 5128 - Ensures that all 5129 memory operations 5130 to global have 5131 completed before 5132 performing the 5133 atomicrmw that is 5134 being released. 5135 5136 2. flat_atomic 5137 3. s_waitcnt vmcnt(0) & 5138 lgkmcnt(0) 5139 5140 - If OpenCL, omit 5141 lgkmcnt(0). 5142 - Must happen before 5143 following 5144 buffer_wbinvl1_vol. 5145 - Ensures the 5146 atomicrmw has 5147 completed before 5148 invalidating the 5149 cache. 5150 5151 4. buffer_wbinvl1_vol 5152 5153 - Must happen before 5154 any following 5155 global/generic 5156 load/load 5157 atomic/atomicrmw. 5158 - Ensures that 5159 following loads 5160 will not see stale 5161 global data. 5162 5163 fence acq_rel - singlethread *none* *none* 5164 - wavefront 5165 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5166 5167 - If OpenCL and 5168 address space is 5169 not generic, omit. 5170 - However, 5171 since LLVM 5172 currently has no 5173 address space on 5174 the fence need to 5175 conservatively 5176 always generate 5177 (see comment for 5178 previous fence). 5179 - Must happen after 5180 any preceding 5181 local/generic 5182 load/load 5183 atomic/store/store 5184 atomic/atomicrmw. 5185 - Must happen before 5186 any following 5187 global/generic 5188 load/load 5189 atomic/store/store 5190 atomic/atomicrmw. 5191 - Ensures that all 5192 memory operations 5193 to local have 5194 completed before 5195 performing any 5196 following global 5197 memory operations. 5198 - Ensures that the 5199 preceding 5200 local/generic load 5201 atomic/atomicrmw 5202 with an equal or 5203 wider sync scope 5204 and memory ordering 5205 stronger than 5206 unordered (this is 5207 termed the 5208 acquire-fence-paired-atomic) 5209 has completed 5210 before following 5211 global memory 5212 operations. This 5213 satisfies the 5214 requirements of 5215 acquire. 5216 - Ensures that all 5217 previous memory 5218 operations have 5219 completed before a 5220 following 5221 local/generic store 5222 atomic/atomicrmw 5223 with an equal or 5224 wider sync scope 5225 and memory ordering 5226 stronger than 5227 unordered (this is 5228 termed the 5229 release-fence-paired-atomic). 5230 This satisfies the 5231 requirements of 5232 release. 5233 5234 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5235 - system vmcnt(0) 5236 5237 - If OpenCL and 5238 address space is 5239 not generic, omit 5240 lgkmcnt(0). 5241 - However, since LLVM 5242 currently has no 5243 address space on 5244 the fence need to 5245 conservatively 5246 always generate 5247 (see comment for 5248 previous fence). 5249 - Could be split into 5250 separate s_waitcnt 5251 vmcnt(0) and 5252 s_waitcnt 5253 lgkmcnt(0) to allow 5254 them to be 5255 independently moved 5256 according to the 5257 following rules. 5258 - s_waitcnt vmcnt(0) 5259 must happen after 5260 any preceding 5261 global/generic 5262 load/store/load 5263 atomic/store 5264 atomic/atomicrmw. 5265 - s_waitcnt lgkmcnt(0) 5266 must happen after 5267 any preceding 5268 local/generic 5269 load/store/load 5270 atomic/store 5271 atomic/atomicrmw. 5272 - Must happen before 5273 the following 5274 buffer_wbinvl1_vol. 5275 - Ensures that the 5276 preceding 5277 global/local/generic 5278 load 5279 atomic/atomicrmw 5280 with an equal or 5281 wider sync scope 5282 and memory ordering 5283 stronger than 5284 unordered (this is 5285 termed the 5286 acquire-fence-paired-atomic) 5287 has completed 5288 before invalidating 5289 the cache. This 5290 satisfies the 5291 requirements of 5292 acquire. 5293 - Ensures that all 5294 previous memory 5295 operations have 5296 completed before a 5297 following 5298 global/local/generic 5299 store 5300 atomic/atomicrmw 5301 with an equal or 5302 wider sync scope 5303 and memory ordering 5304 stronger than 5305 unordered (this is 5306 termed the 5307 release-fence-paired-atomic). 5308 This satisfies the 5309 requirements of 5310 release. 5311 5312 2. buffer_wbinvl1_vol 5313 5314 - Must happen before 5315 any following 5316 global/generic 5317 load/load 5318 atomic/store/store 5319 atomic/atomicrmw. 5320 - Ensures that 5321 following loads 5322 will not see stale 5323 global data. This 5324 satisfies the 5325 requirements of 5326 acquire. 5327 5328 **Sequential Consistent Atomic** 5329 ------------------------------------------------------------------------------------ 5330 load atomic seq_cst - singlethread - global *Same as corresponding 5331 - wavefront - local load atomic acquire, 5332 - generic except must generated 5333 all instructions even 5334 for OpenCL.* 5335 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5336 - generic 5337 5338 - Must 5339 happen after 5340 preceding 5341 local/generic load 5342 atomic/store 5343 atomic/atomicrmw 5344 with memory 5345 ordering of seq_cst 5346 and with equal or 5347 wider sync scope. 5348 (Note that seq_cst 5349 fences have their 5350 own s_waitcnt 5351 lgkmcnt(0) and so do 5352 not need to be 5353 considered.) 5354 - Ensures any 5355 preceding 5356 sequential 5357 consistent local 5358 memory instructions 5359 have completed 5360 before executing 5361 this sequentially 5362 consistent 5363 instruction. This 5364 prevents reordering 5365 a seq_cst store 5366 followed by a 5367 seq_cst load. (Note 5368 that seq_cst is 5369 stronger than 5370 acquire/release as 5371 the reordering of 5372 load acquire 5373 followed by a store 5374 release is 5375 prevented by the 5376 s_waitcnt of 5377 the release, but 5378 there is nothing 5379 preventing a store 5380 release followed by 5381 load acquire from 5382 completing out of 5383 order. The s_waitcnt 5384 could be placed after 5385 seq_store or before 5386 the seq_load. We 5387 choose the load to 5388 make the s_waitcnt be 5389 as late as possible 5390 so that the store 5391 may have already 5392 completed.) 5393 5394 2. *Following 5395 instructions same as 5396 corresponding load 5397 atomic acquire, 5398 except must generated 5399 all instructions even 5400 for OpenCL.* 5401 load atomic seq_cst - workgroup - local *Same as corresponding 5402 load atomic acquire, 5403 except must generated 5404 all instructions even 5405 for OpenCL.* 5406 5407 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5408 - system - generic vmcnt(0) 5409 5410 - Could be split into 5411 separate s_waitcnt 5412 vmcnt(0) 5413 and s_waitcnt 5414 lgkmcnt(0) to allow 5415 them to be 5416 independently moved 5417 according to the 5418 following rules. 5419 - s_waitcnt lgkmcnt(0) 5420 must happen after 5421 preceding 5422 global/generic load 5423 atomic/store 5424 atomic/atomicrmw 5425 with memory 5426 ordering of seq_cst 5427 and with equal or 5428 wider sync scope. 5429 (Note that seq_cst 5430 fences have their 5431 own s_waitcnt 5432 lgkmcnt(0) and so do 5433 not need to be 5434 considered.) 5435 - s_waitcnt vmcnt(0) 5436 must happen after 5437 preceding 5438 global/generic load 5439 atomic/store 5440 atomic/atomicrmw 5441 with memory 5442 ordering of seq_cst 5443 and with equal or 5444 wider sync scope. 5445 (Note that seq_cst 5446 fences have their 5447 own s_waitcnt 5448 vmcnt(0) and so do 5449 not need to be 5450 considered.) 5451 - Ensures any 5452 preceding 5453 sequential 5454 consistent global 5455 memory instructions 5456 have completed 5457 before executing 5458 this sequentially 5459 consistent 5460 instruction. This 5461 prevents reordering 5462 a seq_cst store 5463 followed by a 5464 seq_cst load. (Note 5465 that seq_cst is 5466 stronger than 5467 acquire/release as 5468 the reordering of 5469 load acquire 5470 followed by a store 5471 release is 5472 prevented by the 5473 s_waitcnt of 5474 the release, but 5475 there is nothing 5476 preventing a store 5477 release followed by 5478 load acquire from 5479 completing out of 5480 order. The s_waitcnt 5481 could be placed after 5482 seq_store or before 5483 the seq_load. We 5484 choose the load to 5485 make the s_waitcnt be 5486 as late as possible 5487 so that the store 5488 may have already 5489 completed.) 5490 5491 2. *Following 5492 instructions same as 5493 corresponding load 5494 atomic acquire, 5495 except must generated 5496 all instructions even 5497 for OpenCL.* 5498 store atomic seq_cst - singlethread - global *Same as corresponding 5499 - wavefront - local store atomic release, 5500 - workgroup - generic except must generated 5501 - agent all instructions even 5502 - system for OpenCL.* 5503 atomicrmw seq_cst - singlethread - global *Same as corresponding 5504 - wavefront - local atomicrmw acq_rel, 5505 - workgroup - generic except must generated 5506 - agent all instructions even 5507 - system for OpenCL.* 5508 fence seq_cst - singlethread *none* *Same as corresponding 5509 - wavefront fence acq_rel, 5510 - workgroup except must generated 5511 - agent all instructions even 5512 - system for OpenCL.* 5513 ============ ============ ============== ========== ================================ 5514 5515.. _amdgpu-amdhsa-memory-model-gfx10: 5516 5517Memory Model GFX10 5518++++++++++++++++++ 5519 5520For GFX10: 5521 5522* Each agent has multiple shader arrays (SA). 5523* Each SA has multiple work-group processors (WGP). 5524* Each WGP has multiple compute units (CU). 5525* Each CU has multiple SIMDs that execute wavefronts. 5526* The wavefronts for a single work-group are executed in the same 5527 WGP. In CU wavefront execution mode the wavefronts may be executed by 5528 different SIMDs in the same CU. In WGP wavefront execution mode the 5529 wavefronts may be executed by different SIMDs in different CUs in the same 5530 WGP. 5531* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 5532 executing on it. 5533* All LDS operations of a WGP are performed as wavefront wide operations in a 5534 global order and involve no caching. Completion is reported to a wavefront in 5535 execution order. 5536* The LDS memory has multiple request queues shared by the SIMDs of a 5537 WGP. Therefore, the LDS operations performed by different wavefronts of a 5538 work-group can be reordered relative to each other, which can result in 5539 reordering the visibility of vector memory operations with respect to LDS 5540 operations of other wavefronts in the same work-group. A ``s_waitcnt 5541 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5542 vector memory operations between wavefronts of a work-group, but not between 5543 operations performed by the same wavefront. 5544* The vector memory operations are performed as wavefront wide operations. 5545 Completion of load/store/sample operations are reported to a wavefront in 5546 execution order of other load/store/sample operations performed by that 5547 wavefront. 5548* The vector memory operations access a vector L0 cache. There is a single L0 5549 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 5550 special action is required for coherence between the lanes of a single 5551 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 5552 wavefronts executing in the same work-group as they may be executing on SIMDs 5553 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 5554 required for coherence between wavefronts executing in different work-groups 5555 as they may be executing on different WGPs. 5556* The scalar memory operations access a scalar L0 cache shared by all wavefronts 5557 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 5558 operations are used in a restricted way so do not impact the memory model. See 5559 :ref:`amdgpu-amdhsa-memory-spaces`. 5560* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 5561 the same SA. Therefore, no special action is required for coherence between 5562 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 5563 required for coherence between wavefronts executing in different work-groups 5564 as they may be executing on different SAs that access different L1s. 5565* The L1 caches have independent quadrants to service disjoint ranges of virtual 5566 addresses. 5567* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 5568 vector and scalar memory operations performed by different wavefronts, whether 5569 executing in the same or different work-groups (which may be executing on 5570 different CUs accessing different L0s), can be reordered relative to each 5571 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 5572 synchronization between vector memory operations of different wavefronts. It 5573 ensures a previous vector memory operation has completed before executing a 5574 subsequent vector memory or LDS operation and so can be used to meet the 5575 requirements of acquire, release and sequential consistency. 5576* The L1 caches use an L2 cache shared by all SAs on the same agent. 5577* The L2 cache has independent channels to service disjoint ranges of virtual 5578 addresses. 5579* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 5580 quadrant has a separate request queue per L2 channel. Therefore, the vector 5581 and scalar memory operations performed by wavefronts executing in different 5582 work-groups (which may be executing on different SAs) of an agent can be 5583 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 5584 required to ensure synchronization between vector memory operations of 5585 different SAs. It ensures a previous vector memory operation has completed 5586 before executing a subsequent vector memory and so can be used to meet the 5587 requirements of acquire, release and sequential consistency. 5588* The L2 cache can be kept coherent with other agents on some targets, or ranges 5589 of virtual addresses can be set up to bypass it to ensure system coherence. 5590 5591Scalar memory operations are only used to access memory that is proven to not 5592change during the execution of the kernel dispatch. This includes constant 5593address space and global address space for program scope ``const`` variables. 5594Therefore, the kernel machine code does not have to maintain the scalar cache to 5595ensure it is coherent with the vector caches. The scalar and vector caches are 5596invalidated between kernel dispatches by CP since constant address space data 5597may change between kernel dispatch executions. See 5598:ref:`amdgpu-amdhsa-memory-spaces`. 5599 5600The one exception is if scalar writes are used to spill SGPR registers. In this 5601case the AMDGPU backend ensures the memory location used to spill is never 5602accessed by vector memory operations at the same time. If scalar writes are used 5603then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5604return since the locations may be used for vector memory instructions by a 5605future wavefront that uses the same scratch area, or a function call that 5606creates a frame at the same address, respectively. There is no need for a 5607``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5608 5609For kernarg backing memory: 5610 5611* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 5612* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 5613 needing to invalidate the L2 cache. 5614* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 5615 so the L2 cache will be coherent with the CPU and other agents. 5616 5617Scratch backing memory (which is used for the private address space) is accessed 5618with MTYPE NC (non-coherent). Since the private address space is only accessed 5619by a single thread, and is always write-before-read, there is never a need to 5620invalidate these entries from the L0 or L1 caches. 5621 5622Wavefronts are executed in native mode with in-order reporting of loads and 5623sample instructions. In this mode vmcnt reports completion of load, atomic with 5624return and sample instructions in order, and the vscnt reports the completion of 5625store and atomic without return in order. See ``MEM_ORDERED`` field in 5626:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 5627 5628Wavefronts can be executed in WGP or CU wavefront execution mode: 5629 5630* In WGP wavefront execution mode the wavefronts of a work-group are executed 5631 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 5632 CU L0 caches is required for work-group synchronization. Also accesses to L1 5633 at work-group scope need to be explicitly ordered as the accesses from 5634 different CUs are not ordered. 5635* In CU wavefront execution mode the wavefronts of a work-group are executed on 5636 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 5637 the work-group access the same L0 which in turn ensures L1 accesses are 5638 ordered and so do not require explicit management of the caches for 5639 work-group synchronization. 5640 5641See ``WGP_MODE`` field in 5642:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 5643:ref:`amdgpu-target-features`. 5644 5645The code sequences used to implement the memory model for GFX10 are defined in 5646table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 5647 5648 .. table:: AMDHSA Memory Model Code Sequences GFX10 5649 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 5650 5651 ============ ============ ============== ========== ================================ 5652 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5653 Ordering Sync Scope Address GFX10 5654 Space 5655 ============ ============ ============== ========== ================================ 5656 **Non-Atomic** 5657 ------------------------------------------------------------------------------------ 5658 load *none* *none* - global - !volatile & !nontemporal 5659 - generic 5660 - private 1. buffer/global/flat_load 5661 - constant 5662 - volatile & !nontemporal 5663 5664 1. buffer/global/flat_load 5665 glc=1 dlc=1 5666 5667 - nontemporal 5668 5669 1. buffer/global/flat_load 5670 slc=1 5671 5672 load *none* *none* - local 1. ds_load 5673 store *none* *none* - global - !nontemporal 5674 - generic 5675 - private 1. buffer/global/flat_store 5676 - constant 5677 - nontemporal 5678 5679 1. buffer/global/flat_store 5680 slc=1 5681 5682 store *none* *none* - local 1. ds_store 5683 **Unordered Atomic** 5684 ------------------------------------------------------------------------------------ 5685 load atomic unordered *any* *any* *Same as non-atomic*. 5686 store atomic unordered *any* *any* *Same as non-atomic*. 5687 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5688 **Monotonic Atomic** 5689 ------------------------------------------------------------------------------------ 5690 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 5691 - wavefront - generic 5692 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 5693 - generic glc=1 5694 5695 - If CU wavefront execution 5696 mode, omit glc=1. 5697 5698 load atomic monotonic - singlethread - local 1. ds_load 5699 - wavefront 5700 - workgroup 5701 load atomic monotonic - agent - global 1. buffer/global/flat_load 5702 - system - generic glc=1 dlc=1 5703 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5704 - wavefront - generic 5705 - workgroup 5706 - agent 5707 - system 5708 store atomic monotonic - singlethread - local 1. ds_store 5709 - wavefront 5710 - workgroup 5711 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5712 - wavefront - generic 5713 - workgroup 5714 - agent 5715 - system 5716 atomicrmw monotonic - singlethread - local 1. ds_atomic 5717 - wavefront 5718 - workgroup 5719 **Acquire Atomic** 5720 ------------------------------------------------------------------------------------ 5721 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5722 - wavefront - local 5723 - generic 5724 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 5725 5726 - If CU wavefront execution 5727 mode, omit glc=1. 5728 5729 2. s_waitcnt vmcnt(0) 5730 5731 - If CU wavefront execution 5732 mode, omit. 5733 - Must happen before 5734 the following buffer_gl0_inv 5735 and before any following 5736 global/generic 5737 load/load 5738 atomic/store/store 5739 atomic/atomicrmw. 5740 5741 3. buffer_gl0_inv 5742 5743 - If CU wavefront execution 5744 mode, omit. 5745 - Ensures that 5746 following 5747 loads will not see 5748 stale data. 5749 5750 load atomic acquire - workgroup - local 1. ds_load 5751 2. s_waitcnt lgkmcnt(0) 5752 5753 - If OpenCL, omit. 5754 - Must happen before 5755 the following buffer_gl0_inv 5756 and before any following 5757 global/generic load/load 5758 atomic/store/store 5759 atomic/atomicrmw. 5760 - Ensures any 5761 following global 5762 data read is no 5763 older than the local load 5764 atomic value being 5765 acquired. 5766 5767 3. buffer_gl0_inv 5768 5769 - If CU wavefront execution 5770 mode, omit. 5771 - If OpenCL, omit. 5772 - Ensures that 5773 following 5774 loads will not see 5775 stale data. 5776 5777 load atomic acquire - workgroup - generic 1. flat_load glc=1 5778 5779 - If CU wavefront execution 5780 mode, omit glc=1. 5781 5782 2. s_waitcnt lgkmcnt(0) & 5783 vmcnt(0) 5784 5785 - If CU wavefront execution 5786 mode, omit vmcnt(0). 5787 - If OpenCL, omit 5788 lgkmcnt(0). 5789 - Must happen before 5790 the following 5791 buffer_gl0_inv and any 5792 following global/generic 5793 load/load 5794 atomic/store/store 5795 atomic/atomicrmw. 5796 - Ensures any 5797 following global 5798 data read is no 5799 older than a local load 5800 atomic value being 5801 acquired. 5802 5803 3. buffer_gl0_inv 5804 5805 - If CU wavefront execution 5806 mode, omit. 5807 - Ensures that 5808 following 5809 loads will not see 5810 stale data. 5811 5812 load atomic acquire - agent - global 1. buffer/global_load 5813 - system glc=1 dlc=1 5814 2. s_waitcnt vmcnt(0) 5815 5816 - Must happen before 5817 following 5818 buffer_gl*_inv. 5819 - Ensures the load 5820 has completed 5821 before invalidating 5822 the caches. 5823 5824 3. buffer_gl0_inv; 5825 buffer_gl1_inv 5826 5827 - Must happen before 5828 any following 5829 global/generic 5830 load/load 5831 atomic/atomicrmw. 5832 - Ensures that 5833 following 5834 loads will not see 5835 stale global data. 5836 5837 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 5838 - system 2. s_waitcnt vmcnt(0) & 5839 lgkmcnt(0) 5840 5841 - If OpenCL omit 5842 lgkmcnt(0). 5843 - Must happen before 5844 following 5845 buffer_gl*_invl. 5846 - Ensures the flat_load 5847 has completed 5848 before invalidating 5849 the caches. 5850 5851 3. buffer_gl0_inv; 5852 buffer_gl1_inv 5853 5854 - Must happen before 5855 any following 5856 global/generic 5857 load/load 5858 atomic/atomicrmw. 5859 - Ensures that 5860 following loads 5861 will not see stale 5862 global data. 5863 5864 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5865 - wavefront - local 5866 - generic 5867 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5868 2. s_waitcnt vm/vscnt(0) 5869 5870 - If CU wavefront execution 5871 mode, omit. 5872 - Use vmcnt(0) if atomic with 5873 return and vscnt(0) if 5874 atomic with no-return. 5875 - Must happen before 5876 the following buffer_gl0_inv 5877 and before any following 5878 global/generic 5879 load/load 5880 atomic/store/store 5881 atomic/atomicrmw. 5882 5883 3. buffer_gl0_inv 5884 5885 - If CU wavefront execution 5886 mode, omit. 5887 - Ensures that 5888 following 5889 loads will not see 5890 stale data. 5891 5892 atomicrmw acquire - workgroup - local 1. ds_atomic 5893 2. s_waitcnt lgkmcnt(0) 5894 5895 - If OpenCL, omit. 5896 - Must happen before 5897 the following 5898 buffer_gl0_inv. 5899 - Ensures any 5900 following global 5901 data read is no 5902 older than the local 5903 atomicrmw value 5904 being acquired. 5905 5906 3. buffer_gl0_inv 5907 5908 - If OpenCL omit. 5909 - Ensures that 5910 following 5911 loads will not see 5912 stale data. 5913 5914 atomicrmw acquire - workgroup - generic 1. flat_atomic 5915 2. s_waitcnt lgkmcnt(0) & 5916 vm/vscnt(0) 5917 5918 - If CU wavefront execution 5919 mode, omit vm/vscnt(0). 5920 - If OpenCL, omit lgkmcnt(0). 5921 - Use vmcnt(0) if atomic with 5922 return and vscnt(0) if 5923 atomic with no-return. 5924 - Must happen before 5925 the following 5926 buffer_gl0_inv. 5927 - Ensures any 5928 following global 5929 data read is no 5930 older than a local 5931 atomicrmw value 5932 being acquired. 5933 5934 3. buffer_gl0_inv 5935 5936 - If CU wavefront execution 5937 mode, omit. 5938 - Ensures that 5939 following 5940 loads will not see 5941 stale data. 5942 5943 atomicrmw acquire - agent - global 1. buffer/global_atomic 5944 - system 2. s_waitcnt vm/vscnt(0) 5945 5946 - Use vmcnt(0) if atomic with 5947 return and vscnt(0) if 5948 atomic with no-return. 5949 - Must happen before 5950 following 5951 buffer_gl*_inv. 5952 - Ensures the 5953 atomicrmw has 5954 completed before 5955 invalidating the 5956 caches. 5957 5958 3. buffer_gl0_inv; 5959 buffer_gl1_inv 5960 5961 - Must happen before 5962 any following 5963 global/generic 5964 load/load 5965 atomic/atomicrmw. 5966 - Ensures that 5967 following loads 5968 will not see stale 5969 global data. 5970 5971 atomicrmw acquire - agent - generic 1. flat_atomic 5972 - system 2. s_waitcnt vm/vscnt(0) & 5973 lgkmcnt(0) 5974 5975 - If OpenCL, omit 5976 lgkmcnt(0). 5977 - Use vmcnt(0) if atomic with 5978 return and vscnt(0) if 5979 atomic with no-return. 5980 - Must happen before 5981 following 5982 buffer_gl*_inv. 5983 - Ensures the 5984 atomicrmw has 5985 completed before 5986 invalidating the 5987 caches. 5988 5989 3. buffer_gl0_inv; 5990 buffer_gl1_inv 5991 5992 - Must happen before 5993 any following 5994 global/generic 5995 load/load 5996 atomic/atomicrmw. 5997 - Ensures that 5998 following loads 5999 will not see stale 6000 global data. 6001 6002 fence acquire - singlethread *none* *none* 6003 - wavefront 6004 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 6005 vmcnt(0) & vscnt(0) 6006 6007 - If CU wavefront execution 6008 mode, omit vmcnt(0) and 6009 vscnt(0). 6010 - If OpenCL and 6011 address space is 6012 not generic, omit 6013 lgkmcnt(0). 6014 - If OpenCL and 6015 address space is 6016 local, omit 6017 vmcnt(0) and vscnt(0). 6018 - However, since LLVM 6019 currently has no 6020 address space on 6021 the fence need to 6022 conservatively 6023 always generate. If 6024 fence had an 6025 address space then 6026 set to address 6027 space of OpenCL 6028 fence flag, or to 6029 generic if both 6030 local and global 6031 flags are 6032 specified. 6033 - Could be split into 6034 separate s_waitcnt 6035 vmcnt(0), s_waitcnt 6036 vscnt(0) and s_waitcnt 6037 lgkmcnt(0) to allow 6038 them to be 6039 independently moved 6040 according to the 6041 following rules. 6042 - s_waitcnt vmcnt(0) 6043 must happen after 6044 any preceding 6045 global/generic load 6046 atomic/ 6047 atomicrmw-with-return-value 6048 with an equal or 6049 wider sync scope 6050 and memory ordering 6051 stronger than 6052 unordered (this is 6053 termed the 6054 fence-paired-atomic). 6055 - s_waitcnt vscnt(0) 6056 must happen after 6057 any preceding 6058 global/generic 6059 atomicrmw-no-return-value 6060 with an equal or 6061 wider sync scope 6062 and memory ordering 6063 stronger than 6064 unordered (this is 6065 termed the 6066 fence-paired-atomic). 6067 - s_waitcnt lgkmcnt(0) 6068 must happen after 6069 any preceding 6070 local/generic load 6071 atomic/atomicrmw 6072 with an equal or 6073 wider sync scope 6074 and memory ordering 6075 stronger than 6076 unordered (this is 6077 termed the 6078 fence-paired-atomic). 6079 - Must happen before 6080 the following 6081 buffer_gl0_inv. 6082 - Ensures that the 6083 fence-paired atomic 6084 has completed 6085 before invalidating 6086 the 6087 cache. Therefore 6088 any following 6089 locations read must 6090 be no older than 6091 the value read by 6092 the 6093 fence-paired-atomic. 6094 6095 3. buffer_gl0_inv 6096 6097 - If CU wavefront execution 6098 mode, omit. 6099 - Ensures that 6100 following 6101 loads will not see 6102 stale data. 6103 6104 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6105 - system vmcnt(0) & vscnt(0) 6106 6107 - If OpenCL and 6108 address space is 6109 not generic, omit 6110 lgkmcnt(0). 6111 - If OpenCL and 6112 address space is 6113 local, omit 6114 vmcnt(0) and vscnt(0). 6115 - However, since LLVM 6116 currently has no 6117 address space on 6118 the fence need to 6119 conservatively 6120 always generate 6121 (see comment for 6122 previous fence). 6123 - Could be split into 6124 separate s_waitcnt 6125 vmcnt(0), s_waitcnt 6126 vscnt(0) and s_waitcnt 6127 lgkmcnt(0) to allow 6128 them to be 6129 independently moved 6130 according to the 6131 following rules. 6132 - s_waitcnt vmcnt(0) 6133 must happen after 6134 any preceding 6135 global/generic load 6136 atomic/ 6137 atomicrmw-with-return-value 6138 with an equal or 6139 wider sync scope 6140 and memory ordering 6141 stronger than 6142 unordered (this is 6143 termed the 6144 fence-paired-atomic). 6145 - s_waitcnt vscnt(0) 6146 must happen after 6147 any preceding 6148 global/generic 6149 atomicrmw-no-return-value 6150 with an equal or 6151 wider sync scope 6152 and memory ordering 6153 stronger than 6154 unordered (this is 6155 termed the 6156 fence-paired-atomic). 6157 - s_waitcnt lgkmcnt(0) 6158 must happen after 6159 any preceding 6160 local/generic load 6161 atomic/atomicrmw 6162 with an equal or 6163 wider sync scope 6164 and memory ordering 6165 stronger than 6166 unordered (this is 6167 termed the 6168 fence-paired-atomic). 6169 - Must happen before 6170 the following 6171 buffer_gl*_inv. 6172 - Ensures that the 6173 fence-paired atomic 6174 has completed 6175 before invalidating 6176 the 6177 caches. Therefore 6178 any following 6179 locations read must 6180 be no older than 6181 the value read by 6182 the 6183 fence-paired-atomic. 6184 6185 2. buffer_gl0_inv; 6186 buffer_gl1_inv 6187 6188 - Must happen before any 6189 following global/generic 6190 load/load 6191 atomic/store/store 6192 atomic/atomicrmw. 6193 - Ensures that 6194 following loads 6195 will not see stale 6196 global data. 6197 6198 **Release Atomic** 6199 ------------------------------------------------------------------------------------ 6200 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 6201 - wavefront - local 6202 - generic 6203 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 6204 - generic vmcnt(0) & vscnt(0) 6205 6206 - If CU wavefront execution 6207 mode, omit vmcnt(0) and 6208 vscnt(0). 6209 - If OpenCL, omit 6210 lgkmcnt(0). 6211 - Could be split into 6212 separate s_waitcnt 6213 vmcnt(0), s_waitcnt 6214 vscnt(0) and s_waitcnt 6215 lgkmcnt(0) to allow 6216 them to be 6217 independently moved 6218 according to the 6219 following rules. 6220 - s_waitcnt vmcnt(0) 6221 must happen after 6222 any preceding 6223 global/generic load/load 6224 atomic/ 6225 atomicrmw-with-return-value. 6226 - s_waitcnt vscnt(0) 6227 must happen after 6228 any preceding 6229 global/generic 6230 store/store 6231 atomic/ 6232 atomicrmw-no-return-value. 6233 - s_waitcnt lgkmcnt(0) 6234 must happen after 6235 any preceding 6236 local/generic 6237 load/store/load 6238 atomic/store 6239 atomic/atomicrmw. 6240 - Must happen before 6241 the following 6242 store. 6243 - Ensures that all 6244 memory operations 6245 have 6246 completed before 6247 performing the 6248 store that is being 6249 released. 6250 6251 2. buffer/global/flat_store 6252 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 6253 6254 - If CU wavefront execution 6255 mode, omit. 6256 - If OpenCL, omit. 6257 - Could be split into 6258 separate s_waitcnt 6259 vmcnt(0) and s_waitcnt 6260 vscnt(0) to allow 6261 them to be 6262 independently moved 6263 according to the 6264 following rules. 6265 - s_waitcnt vmcnt(0) 6266 must happen after 6267 any preceding 6268 global/generic load/load 6269 atomic/ 6270 atomicrmw-with-return-value. 6271 - s_waitcnt vscnt(0) 6272 must happen after 6273 any preceding 6274 global/generic 6275 store/store atomic/ 6276 atomicrmw-no-return-value. 6277 - Must happen before 6278 the following 6279 store. 6280 - Ensures that all 6281 global memory 6282 operations have 6283 completed before 6284 performing the 6285 store that is being 6286 released. 6287 6288 2. ds_store 6289 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6290 - system - generic vmcnt(0) & vscnt(0) 6291 6292 - If OpenCL and 6293 address space is 6294 not generic, omit 6295 lgkmcnt(0). 6296 - Could be split into 6297 separate s_waitcnt 6298 vmcnt(0), s_waitcnt vscnt(0) 6299 and s_waitcnt 6300 lgkmcnt(0) to allow 6301 them to be 6302 independently moved 6303 according to the 6304 following rules. 6305 - s_waitcnt vmcnt(0) 6306 must happen after 6307 any preceding 6308 global/generic 6309 load/load 6310 atomic/ 6311 atomicrmw-with-return-value. 6312 - s_waitcnt vscnt(0) 6313 must happen after 6314 any preceding 6315 global/generic 6316 store/store atomic/ 6317 atomicrmw-no-return-value. 6318 - s_waitcnt lgkmcnt(0) 6319 must happen after 6320 any preceding 6321 local/generic 6322 load/store/load 6323 atomic/store 6324 atomic/atomicrmw. 6325 - Must happen before 6326 the following 6327 store. 6328 - Ensures that all 6329 memory operations 6330 have 6331 completed before 6332 performing the 6333 store that is being 6334 released. 6335 6336 2. buffer/global/flat_store 6337 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 6338 - wavefront - local 6339 - generic 6340 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 6341 - generic vmcnt(0) & vscnt(0) 6342 6343 - If CU wavefront execution 6344 mode, omit vmcnt(0) and 6345 vscnt(0). 6346 - If OpenCL, omit lgkmcnt(0). 6347 - Could be split into 6348 separate s_waitcnt 6349 vmcnt(0), s_waitcnt 6350 vscnt(0) and s_waitcnt 6351 lgkmcnt(0) to allow 6352 them to be 6353 independently moved 6354 according to the 6355 following rules. 6356 - s_waitcnt vmcnt(0) 6357 must happen after 6358 any preceding 6359 global/generic load/load 6360 atomic/ 6361 atomicrmw-with-return-value. 6362 - s_waitcnt vscnt(0) 6363 must happen after 6364 any preceding 6365 global/generic 6366 store/store 6367 atomic/ 6368 atomicrmw-no-return-value. 6369 - s_waitcnt lgkmcnt(0) 6370 must happen after 6371 any preceding 6372 local/generic 6373 load/store/load 6374 atomic/store 6375 atomic/atomicrmw. 6376 - Must happen before 6377 the following 6378 atomicrmw. 6379 - Ensures that all 6380 memory operations 6381 have 6382 completed before 6383 performing the 6384 atomicrmw that is 6385 being released. 6386 6387 2. buffer/global/flat_atomic 6388 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 6389 6390 - If CU wavefront execution 6391 mode, omit. 6392 - If OpenCL, omit. 6393 - Could be split into 6394 separate s_waitcnt 6395 vmcnt(0) and s_waitcnt 6396 vscnt(0) to allow 6397 them to be 6398 independently moved 6399 according to the 6400 following rules. 6401 - s_waitcnt vmcnt(0) 6402 must happen after 6403 any preceding 6404 global/generic load/load 6405 atomic/ 6406 atomicrmw-with-return-value. 6407 - s_waitcnt vscnt(0) 6408 must happen after 6409 any preceding 6410 global/generic 6411 store/store atomic/ 6412 atomicrmw-no-return-value. 6413 - Must happen before 6414 the following 6415 store. 6416 - Ensures that all 6417 global memory 6418 operations have 6419 completed before 6420 performing the 6421 store that is being 6422 released. 6423 6424 2. ds_atomic 6425 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 6426 - system - generic vmcnt(0) & vscnt(0) 6427 6428 - If OpenCL, omit 6429 lgkmcnt(0). 6430 - Could be split into 6431 separate s_waitcnt 6432 vmcnt(0), s_waitcnt 6433 vscnt(0) and s_waitcnt 6434 lgkmcnt(0) to allow 6435 them to be 6436 independently moved 6437 according to the 6438 following rules. 6439 - s_waitcnt vmcnt(0) 6440 must happen after 6441 any preceding 6442 global/generic 6443 load/load atomic/ 6444 atomicrmw-with-return-value. 6445 - s_waitcnt vscnt(0) 6446 must happen after 6447 any preceding 6448 global/generic 6449 store/store atomic/ 6450 atomicrmw-no-return-value. 6451 - s_waitcnt lgkmcnt(0) 6452 must happen after 6453 any preceding 6454 local/generic 6455 load/store/load 6456 atomic/store 6457 atomic/atomicrmw. 6458 - Must happen before 6459 the following 6460 atomicrmw. 6461 - Ensures that all 6462 memory operations 6463 to global and local 6464 have completed 6465 before performing 6466 the atomicrmw that 6467 is being released. 6468 6469 2. buffer/global/flat_atomic 6470 fence release - singlethread *none* *none* 6471 - wavefront 6472 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 6473 vmcnt(0) & vscnt(0) 6474 6475 - If CU wavefront execution 6476 mode, omit vmcnt(0) and 6477 vscnt(0). 6478 - If OpenCL and 6479 address space is 6480 not generic, omit 6481 lgkmcnt(0). 6482 - If OpenCL and 6483 address space is 6484 local, omit 6485 vmcnt(0) and vscnt(0). 6486 - However, since LLVM 6487 currently has no 6488 address space on 6489 the fence need to 6490 conservatively 6491 always generate. If 6492 fence had an 6493 address space then 6494 set to address 6495 space of OpenCL 6496 fence flag, or to 6497 generic if both 6498 local and global 6499 flags are 6500 specified. 6501 - Could be split into 6502 separate s_waitcnt 6503 vmcnt(0), s_waitcnt 6504 vscnt(0) and s_waitcnt 6505 lgkmcnt(0) to allow 6506 them to be 6507 independently moved 6508 according to the 6509 following rules. 6510 - s_waitcnt vmcnt(0) 6511 must happen after 6512 any preceding 6513 global/generic 6514 load/load 6515 atomic/ 6516 atomicrmw-with-return-value. 6517 - s_waitcnt vscnt(0) 6518 must happen after 6519 any preceding 6520 global/generic 6521 store/store atomic/ 6522 atomicrmw-no-return-value. 6523 - s_waitcnt lgkmcnt(0) 6524 must happen after 6525 any preceding 6526 local/generic 6527 load/store/load 6528 atomic/store atomic/ 6529 atomicrmw. 6530 - Must happen before 6531 any following store 6532 atomic/atomicrmw 6533 with an equal or 6534 wider sync scope 6535 and memory ordering 6536 stronger than 6537 unordered (this is 6538 termed the 6539 fence-paired-atomic). 6540 - Ensures that all 6541 memory operations 6542 have 6543 completed before 6544 performing the 6545 following 6546 fence-paired-atomic. 6547 6548 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 6549 - system vmcnt(0) & vscnt(0) 6550 6551 - If OpenCL and 6552 address space is 6553 not generic, omit 6554 lgkmcnt(0). 6555 - If OpenCL and 6556 address space is 6557 local, omit 6558 vmcnt(0) and vscnt(0). 6559 - However, since LLVM 6560 currently has no 6561 address space on 6562 the fence need to 6563 conservatively 6564 always generate. If 6565 fence had an 6566 address space then 6567 set to address 6568 space of OpenCL 6569 fence flag, or to 6570 generic if both 6571 local and global 6572 flags are 6573 specified. 6574 - Could be split into 6575 separate s_waitcnt 6576 vmcnt(0), s_waitcnt 6577 vscnt(0) and s_waitcnt 6578 lgkmcnt(0) to allow 6579 them to be 6580 independently moved 6581 according to the 6582 following rules. 6583 - s_waitcnt vmcnt(0) 6584 must happen after 6585 any preceding 6586 global/generic 6587 load/load atomic/ 6588 atomicrmw-with-return-value. 6589 - s_waitcnt vscnt(0) 6590 must happen after 6591 any preceding 6592 global/generic 6593 store/store atomic/ 6594 atomicrmw-no-return-value. 6595 - s_waitcnt lgkmcnt(0) 6596 must happen after 6597 any preceding 6598 local/generic 6599 load/store/load 6600 atomic/store 6601 atomic/atomicrmw. 6602 - Must happen before 6603 any following store 6604 atomic/atomicrmw 6605 with an equal or 6606 wider sync scope 6607 and memory ordering 6608 stronger than 6609 unordered (this is 6610 termed the 6611 fence-paired-atomic). 6612 - Ensures that all 6613 memory operations 6614 have 6615 completed before 6616 performing the 6617 following 6618 fence-paired-atomic. 6619 6620 **Acquire-Release Atomic** 6621 ------------------------------------------------------------------------------------ 6622 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 6623 - wavefront - local 6624 - generic 6625 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 6626 vmcnt(0) & vscnt(0) 6627 6628 - If CU wavefront execution 6629 mode, omit vmcnt(0) and 6630 vscnt(0). 6631 - If OpenCL, omit 6632 lgkmcnt(0). 6633 - Must happen after 6634 any preceding 6635 local/generic 6636 load/store/load 6637 atomic/store 6638 atomic/atomicrmw. 6639 - Could be split into 6640 separate s_waitcnt 6641 vmcnt(0), s_waitcnt 6642 vscnt(0), and s_waitcnt 6643 lgkmcnt(0) to allow 6644 them to be 6645 independently moved 6646 according to the 6647 following rules. 6648 - s_waitcnt vmcnt(0) 6649 must happen after 6650 any preceding 6651 global/generic load/load 6652 atomic/ 6653 atomicrmw-with-return-value. 6654 - s_waitcnt vscnt(0) 6655 must happen after 6656 any preceding 6657 global/generic 6658 store/store 6659 atomic/ 6660 atomicrmw-no-return-value. 6661 - s_waitcnt lgkmcnt(0) 6662 must happen after 6663 any preceding 6664 local/generic 6665 load/store/load 6666 atomic/store 6667 atomic/atomicrmw. 6668 - Must happen before 6669 the following 6670 atomicrmw. 6671 - Ensures that all 6672 memory operations 6673 have 6674 completed before 6675 performing the 6676 atomicrmw that is 6677 being released. 6678 6679 2. buffer/global_atomic 6680 3. s_waitcnt vm/vscnt(0) 6681 6682 - If CU wavefront execution 6683 mode, omit. 6684 - Use vmcnt(0) if atomic with 6685 return and vscnt(0) if 6686 atomic with no-return. 6687 - Must happen before 6688 the following 6689 buffer_gl0_inv. 6690 - Ensures any 6691 following global 6692 data read is no 6693 older than the 6694 atomicrmw value 6695 being acquired. 6696 6697 4. buffer_gl0_inv 6698 6699 - If CU wavefront execution 6700 mode, omit. 6701 - Ensures that 6702 following 6703 loads will not see 6704 stale data. 6705 6706 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 6707 6708 - If CU wavefront execution 6709 mode, omit. 6710 - If OpenCL, omit. 6711 - Could be split into 6712 separate s_waitcnt 6713 vmcnt(0) and s_waitcnt 6714 vscnt(0) to allow 6715 them to be 6716 independently moved 6717 according to the 6718 following rules. 6719 - s_waitcnt vmcnt(0) 6720 must happen after 6721 any preceding 6722 global/generic load/load 6723 atomic/ 6724 atomicrmw-with-return-value. 6725 - s_waitcnt vscnt(0) 6726 must happen after 6727 any preceding 6728 global/generic 6729 store/store atomic/ 6730 atomicrmw-no-return-value. 6731 - Must happen before 6732 the following 6733 store. 6734 - Ensures that all 6735 global memory 6736 operations have 6737 completed before 6738 performing the 6739 store that is being 6740 released. 6741 6742 2. ds_atomic 6743 3. s_waitcnt lgkmcnt(0) 6744 6745 - If OpenCL, omit. 6746 - Must happen before 6747 the following 6748 buffer_gl0_inv. 6749 - Ensures any 6750 following global 6751 data read is no 6752 older than the local load 6753 atomic value being 6754 acquired. 6755 6756 4. buffer_gl0_inv 6757 6758 - If CU wavefront execution 6759 mode, omit. 6760 - If OpenCL omit. 6761 - Ensures that 6762 following 6763 loads will not see 6764 stale data. 6765 6766 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 6767 vmcnt(0) & vscnt(0) 6768 6769 - If CU wavefront execution 6770 mode, omit vmcnt(0) and 6771 vscnt(0). 6772 - If OpenCL, omit lgkmcnt(0). 6773 - Could be split into 6774 separate s_waitcnt 6775 vmcnt(0), s_waitcnt 6776 vscnt(0) and s_waitcnt 6777 lgkmcnt(0) to allow 6778 them to be 6779 independently moved 6780 according to the 6781 following rules. 6782 - s_waitcnt vmcnt(0) 6783 must happen after 6784 any preceding 6785 global/generic load/load 6786 atomic/ 6787 atomicrmw-with-return-value. 6788 - s_waitcnt vscnt(0) 6789 must happen after 6790 any preceding 6791 global/generic 6792 store/store 6793 atomic/ 6794 atomicrmw-no-return-value. 6795 - s_waitcnt lgkmcnt(0) 6796 must happen after 6797 any preceding 6798 local/generic 6799 load/store/load 6800 atomic/store 6801 atomic/atomicrmw. 6802 - Must happen before 6803 the following 6804 atomicrmw. 6805 - Ensures that all 6806 memory operations 6807 have 6808 completed before 6809 performing the 6810 atomicrmw that is 6811 being released. 6812 6813 2. flat_atomic 6814 3. s_waitcnt lgkmcnt(0) & 6815 vmcnt(0) & vscnt(0) 6816 6817 - If CU wavefront execution 6818 mode, omit vmcnt(0) and 6819 vscnt(0). 6820 - If OpenCL, omit lgkmcnt(0). 6821 - Must happen before 6822 the following 6823 buffer_gl0_inv. 6824 - Ensures any 6825 following global 6826 data read is no 6827 older than the load 6828 atomic value being 6829 acquired. 6830 6831 3. buffer_gl0_inv 6832 6833 - If CU wavefront execution 6834 mode, omit. 6835 - Ensures that 6836 following 6837 loads will not see 6838 stale data. 6839 6840 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 6841 - system vmcnt(0) & vscnt(0) 6842 6843 - If OpenCL, omit 6844 lgkmcnt(0). 6845 - Could be split into 6846 separate s_waitcnt 6847 vmcnt(0), s_waitcnt 6848 vscnt(0) and s_waitcnt 6849 lgkmcnt(0) to allow 6850 them to be 6851 independently moved 6852 according to the 6853 following rules. 6854 - s_waitcnt vmcnt(0) 6855 must happen after 6856 any preceding 6857 global/generic 6858 load/load atomic/ 6859 atomicrmw-with-return-value. 6860 - s_waitcnt vscnt(0) 6861 must happen after 6862 any preceding 6863 global/generic 6864 store/store atomic/ 6865 atomicrmw-no-return-value. 6866 - s_waitcnt lgkmcnt(0) 6867 must happen after 6868 any preceding 6869 local/generic 6870 load/store/load 6871 atomic/store 6872 atomic/atomicrmw. 6873 - Must happen before 6874 the following 6875 atomicrmw. 6876 - Ensures that all 6877 memory operations 6878 to global have 6879 completed before 6880 performing the 6881 atomicrmw that is 6882 being released. 6883 6884 2. buffer/global_atomic 6885 3. s_waitcnt vm/vscnt(0) 6886 6887 - Use vmcnt(0) if atomic with 6888 return and vscnt(0) if 6889 atomic with no-return. 6890 - Must happen before 6891 following 6892 buffer_gl*_inv. 6893 - Ensures the 6894 atomicrmw has 6895 completed before 6896 invalidating the 6897 caches. 6898 6899 4. buffer_gl0_inv; 6900 buffer_gl1_inv 6901 6902 - Must happen before 6903 any following 6904 global/generic 6905 load/load 6906 atomic/atomicrmw. 6907 - Ensures that 6908 following loads 6909 will not see stale 6910 global data. 6911 6912 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 6913 - system vmcnt(0) & vscnt(0) 6914 6915 - If OpenCL, omit 6916 lgkmcnt(0). 6917 - Could be split into 6918 separate s_waitcnt 6919 vmcnt(0), s_waitcnt 6920 vscnt(0), and s_waitcnt 6921 lgkmcnt(0) to allow 6922 them to be 6923 independently moved 6924 according to the 6925 following rules. 6926 - s_waitcnt vmcnt(0) 6927 must happen after 6928 any preceding 6929 global/generic 6930 load/load atomic 6931 atomicrmw-with-return-value. 6932 - s_waitcnt vscnt(0) 6933 must happen after 6934 any preceding 6935 global/generic 6936 store/store atomic/ 6937 atomicrmw-no-return-value. 6938 - s_waitcnt lgkmcnt(0) 6939 must happen after 6940 any preceding 6941 local/generic 6942 load/store/load 6943 atomic/store 6944 atomic/atomicrmw. 6945 - Must happen before 6946 the following 6947 atomicrmw. 6948 - Ensures that all 6949 memory operations 6950 have 6951 completed before 6952 performing the 6953 atomicrmw that is 6954 being released. 6955 6956 2. flat_atomic 6957 3. s_waitcnt vm/vscnt(0) & 6958 lgkmcnt(0) 6959 6960 - If OpenCL, omit 6961 lgkmcnt(0). 6962 - Use vmcnt(0) if atomic with 6963 return and vscnt(0) if 6964 atomic with no-return. 6965 - Must happen before 6966 following 6967 buffer_gl*_inv. 6968 - Ensures the 6969 atomicrmw has 6970 completed before 6971 invalidating the 6972 caches. 6973 6974 4. buffer_gl0_inv; 6975 buffer_gl1_inv 6976 6977 - Must happen before 6978 any following 6979 global/generic 6980 load/load 6981 atomic/atomicrmw. 6982 - Ensures that 6983 following loads 6984 will not see stale 6985 global data. 6986 6987 fence acq_rel - singlethread *none* *none* 6988 - wavefront 6989 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 6990 vmcnt(0) & vscnt(0) 6991 6992 - If CU wavefront execution 6993 mode, omit vmcnt(0) and 6994 vscnt(0). 6995 - If OpenCL and 6996 address space is 6997 not generic, omit 6998 lgkmcnt(0). 6999 - If OpenCL and 7000 address space is 7001 local, omit 7002 vmcnt(0) and vscnt(0). 7003 - However, 7004 since LLVM 7005 currently has no 7006 address space on 7007 the fence need to 7008 conservatively 7009 always generate 7010 (see comment for 7011 previous fence). 7012 - Could be split into 7013 separate s_waitcnt 7014 vmcnt(0), s_waitcnt 7015 vscnt(0) and s_waitcnt 7016 lgkmcnt(0) to allow 7017 them to be 7018 independently moved 7019 according to the 7020 following rules. 7021 - s_waitcnt vmcnt(0) 7022 must happen after 7023 any preceding 7024 global/generic 7025 load/load 7026 atomic/ 7027 atomicrmw-with-return-value. 7028 - s_waitcnt vscnt(0) 7029 must happen after 7030 any preceding 7031 global/generic 7032 store/store atomic/ 7033 atomicrmw-no-return-value. 7034 - s_waitcnt lgkmcnt(0) 7035 must happen after 7036 any preceding 7037 local/generic 7038 load/store/load 7039 atomic/store atomic/ 7040 atomicrmw. 7041 - Must happen before 7042 any following 7043 global/generic 7044 load/load 7045 atomic/store/store 7046 atomic/atomicrmw. 7047 - Ensures that all 7048 memory operations 7049 have 7050 completed before 7051 performing any 7052 following global 7053 memory operations. 7054 - Ensures that the 7055 preceding 7056 local/generic load 7057 atomic/atomicrmw 7058 with an equal or 7059 wider sync scope 7060 and memory ordering 7061 stronger than 7062 unordered (this is 7063 termed the 7064 acquire-fence-paired-atomic) 7065 has completed 7066 before following 7067 global memory 7068 operations. This 7069 satisfies the 7070 requirements of 7071 acquire. 7072 - Ensures that all 7073 previous memory 7074 operations have 7075 completed before a 7076 following 7077 local/generic store 7078 atomic/atomicrmw 7079 with an equal or 7080 wider sync scope 7081 and memory ordering 7082 stronger than 7083 unordered (this is 7084 termed the 7085 release-fence-paired-atomic). 7086 This satisfies the 7087 requirements of 7088 release. 7089 - Must happen before 7090 the following 7091 buffer_gl0_inv. 7092 - Ensures that the 7093 acquire-fence-paired 7094 atomic has completed 7095 before invalidating 7096 the 7097 cache. Therefore 7098 any following 7099 locations read must 7100 be no older than 7101 the value read by 7102 the 7103 acquire-fence-paired-atomic. 7104 7105 3. buffer_gl0_inv 7106 7107 - If CU wavefront execution 7108 mode, omit. 7109 - Ensures that 7110 following 7111 loads will not see 7112 stale data. 7113 7114 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7115 - system vmcnt(0) & vscnt(0) 7116 7117 - If OpenCL and 7118 address space is 7119 not generic, omit 7120 lgkmcnt(0). 7121 - If OpenCL and 7122 address space is 7123 local, omit 7124 vmcnt(0) and vscnt(0). 7125 - However, since LLVM 7126 currently has no 7127 address space on 7128 the fence need to 7129 conservatively 7130 always generate 7131 (see comment for 7132 previous fence). 7133 - Could be split into 7134 separate s_waitcnt 7135 vmcnt(0), s_waitcnt 7136 vscnt(0) and s_waitcnt 7137 lgkmcnt(0) to allow 7138 them to be 7139 independently moved 7140 according to the 7141 following rules. 7142 - s_waitcnt vmcnt(0) 7143 must happen after 7144 any preceding 7145 global/generic 7146 load/load 7147 atomic/ 7148 atomicrmw-with-return-value. 7149 - s_waitcnt vscnt(0) 7150 must happen after 7151 any preceding 7152 global/generic 7153 store/store atomic/ 7154 atomicrmw-no-return-value. 7155 - s_waitcnt lgkmcnt(0) 7156 must happen after 7157 any preceding 7158 local/generic 7159 load/store/load 7160 atomic/store 7161 atomic/atomicrmw. 7162 - Must happen before 7163 the following 7164 buffer_gl*_inv. 7165 - Ensures that the 7166 preceding 7167 global/local/generic 7168 load 7169 atomic/atomicrmw 7170 with an equal or 7171 wider sync scope 7172 and memory ordering 7173 stronger than 7174 unordered (this is 7175 termed the 7176 acquire-fence-paired-atomic) 7177 has completed 7178 before invalidating 7179 the caches. This 7180 satisfies the 7181 requirements of 7182 acquire. 7183 - Ensures that all 7184 previous memory 7185 operations have 7186 completed before a 7187 following 7188 global/local/generic 7189 store 7190 atomic/atomicrmw 7191 with an equal or 7192 wider sync scope 7193 and memory ordering 7194 stronger than 7195 unordered (this is 7196 termed the 7197 release-fence-paired-atomic). 7198 This satisfies the 7199 requirements of 7200 release. 7201 7202 2. buffer_gl0_inv; 7203 buffer_gl1_inv 7204 7205 - Must happen before 7206 any following 7207 global/generic 7208 load/load 7209 atomic/store/store 7210 atomic/atomicrmw. 7211 - Ensures that 7212 following loads 7213 will not see stale 7214 global data. This 7215 satisfies the 7216 requirements of 7217 acquire. 7218 7219 **Sequential Consistent Atomic** 7220 ------------------------------------------------------------------------------------ 7221 load atomic seq_cst - singlethread - global *Same as corresponding 7222 - wavefront - local load atomic acquire, 7223 - generic except must generated 7224 all instructions even 7225 for OpenCL.* 7226 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 7227 - generic vmcnt(0) & vscnt(0) 7228 7229 - If CU wavefront execution 7230 mode, omit vmcnt(0) and 7231 vscnt(0). 7232 - Could be split into 7233 separate s_waitcnt 7234 vmcnt(0), s_waitcnt 7235 vscnt(0), and s_waitcnt 7236 lgkmcnt(0) to allow 7237 them to be 7238 independently moved 7239 according to the 7240 following rules. 7241 - s_waitcnt lgkmcnt(0) must 7242 happen after 7243 preceding 7244 local/generic load 7245 atomic/store 7246 atomic/atomicrmw 7247 with memory 7248 ordering of seq_cst 7249 and with equal or 7250 wider sync scope. 7251 (Note that seq_cst 7252 fences have their 7253 own s_waitcnt 7254 lgkmcnt(0) and so do 7255 not need to be 7256 considered.) 7257 - s_waitcnt vmcnt(0) 7258 must happen after 7259 preceding 7260 global/generic load 7261 atomic/ 7262 atomicrmw-with-return-value 7263 with memory 7264 ordering of seq_cst 7265 and with equal or 7266 wider sync scope. 7267 (Note that seq_cst 7268 fences have their 7269 own s_waitcnt 7270 vmcnt(0) and so do 7271 not need to be 7272 considered.) 7273 - s_waitcnt vscnt(0) 7274 Must happen after 7275 preceding 7276 global/generic store 7277 atomic/ 7278 atomicrmw-no-return-value 7279 with memory 7280 ordering of seq_cst 7281 and with equal or 7282 wider sync scope. 7283 (Note that seq_cst 7284 fences have their 7285 own s_waitcnt 7286 vscnt(0) and so do 7287 not need to be 7288 considered.) 7289 - Ensures any 7290 preceding 7291 sequential 7292 consistent global/local 7293 memory instructions 7294 have completed 7295 before executing 7296 this sequentially 7297 consistent 7298 instruction. This 7299 prevents reordering 7300 a seq_cst store 7301 followed by a 7302 seq_cst load. (Note 7303 that seq_cst is 7304 stronger than 7305 acquire/release as 7306 the reordering of 7307 load acquire 7308 followed by a store 7309 release is 7310 prevented by the 7311 s_waitcnt of 7312 the release, but 7313 there is nothing 7314 preventing a store 7315 release followed by 7316 load acquire from 7317 completing out of 7318 order. The s_waitcnt 7319 could be placed after 7320 seq_store or before 7321 the seq_load. We 7322 choose the load to 7323 make the s_waitcnt be 7324 as late as possible 7325 so that the store 7326 may have already 7327 completed.) 7328 7329 2. *Following 7330 instructions same as 7331 corresponding load 7332 atomic acquire, 7333 except must generated 7334 all instructions even 7335 for OpenCL.* 7336 load atomic seq_cst - workgroup - local 7337 7338 1. s_waitcnt vmcnt(0) & vscnt(0) 7339 7340 - If CU wavefront execution 7341 mode, omit. 7342 - Could be split into 7343 separate s_waitcnt 7344 vmcnt(0) and s_waitcnt 7345 vscnt(0) to allow 7346 them to be 7347 independently moved 7348 according to the 7349 following rules. 7350 - s_waitcnt vmcnt(0) 7351 Must happen after 7352 preceding 7353 global/generic load 7354 atomic/ 7355 atomicrmw-with-return-value 7356 with memory 7357 ordering of seq_cst 7358 and with equal or 7359 wider sync scope. 7360 (Note that seq_cst 7361 fences have their 7362 own s_waitcnt 7363 vmcnt(0) and so do 7364 not need to be 7365 considered.) 7366 - s_waitcnt vscnt(0) 7367 Must happen after 7368 preceding 7369 global/generic store 7370 atomic/ 7371 atomicrmw-no-return-value 7372 with memory 7373 ordering of seq_cst 7374 and with equal or 7375 wider sync scope. 7376 (Note that seq_cst 7377 fences have their 7378 own s_waitcnt 7379 vscnt(0) and so do 7380 not need to be 7381 considered.) 7382 - Ensures any 7383 preceding 7384 sequential 7385 consistent global 7386 memory instructions 7387 have completed 7388 before executing 7389 this sequentially 7390 consistent 7391 instruction. This 7392 prevents reordering 7393 a seq_cst store 7394 followed by a 7395 seq_cst load. (Note 7396 that seq_cst is 7397 stronger than 7398 acquire/release as 7399 the reordering of 7400 load acquire 7401 followed by a store 7402 release is 7403 prevented by the 7404 s_waitcnt of 7405 the release, but 7406 there is nothing 7407 preventing a store 7408 release followed by 7409 load acquire from 7410 completing out of 7411 order. The s_waitcnt 7412 could be placed after 7413 seq_store or before 7414 the seq_load. We 7415 choose the load to 7416 make the s_waitcnt be 7417 as late as possible 7418 so that the store 7419 may have already 7420 completed.) 7421 7422 2. *Following 7423 instructions same as 7424 corresponding load 7425 atomic acquire, 7426 except must generated 7427 all instructions even 7428 for OpenCL.* 7429 7430 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 7431 - system - generic vmcnt(0) & vscnt(0) 7432 7433 - Could be split into 7434 separate s_waitcnt 7435 vmcnt(0), s_waitcnt 7436 vscnt(0) and s_waitcnt 7437 lgkmcnt(0) to allow 7438 them to be 7439 independently moved 7440 according to the 7441 following rules. 7442 - s_waitcnt lgkmcnt(0) 7443 must happen after 7444 preceding 7445 local load 7446 atomic/store 7447 atomic/atomicrmw 7448 with memory 7449 ordering of seq_cst 7450 and with equal or 7451 wider sync scope. 7452 (Note that seq_cst 7453 fences have their 7454 own s_waitcnt 7455 lgkmcnt(0) and so do 7456 not need to be 7457 considered.) 7458 - s_waitcnt vmcnt(0) 7459 must happen after 7460 preceding 7461 global/generic load 7462 atomic/ 7463 atomicrmw-with-return-value 7464 with memory 7465 ordering of seq_cst 7466 and with equal or 7467 wider sync scope. 7468 (Note that seq_cst 7469 fences have their 7470 own s_waitcnt 7471 vmcnt(0) and so do 7472 not need to be 7473 considered.) 7474 - s_waitcnt vscnt(0) 7475 Must happen after 7476 preceding 7477 global/generic store 7478 atomic/ 7479 atomicrmw-no-return-value 7480 with memory 7481 ordering of seq_cst 7482 and with equal or 7483 wider sync scope. 7484 (Note that seq_cst 7485 fences have their 7486 own s_waitcnt 7487 vscnt(0) and so do 7488 not need to be 7489 considered.) 7490 - Ensures any 7491 preceding 7492 sequential 7493 consistent global 7494 memory instructions 7495 have completed 7496 before executing 7497 this sequentially 7498 consistent 7499 instruction. This 7500 prevents reordering 7501 a seq_cst store 7502 followed by a 7503 seq_cst load. (Note 7504 that seq_cst is 7505 stronger than 7506 acquire/release as 7507 the reordering of 7508 load acquire 7509 followed by a store 7510 release is 7511 prevented by the 7512 s_waitcnt of 7513 the release, but 7514 there is nothing 7515 preventing a store 7516 release followed by 7517 load acquire from 7518 completing out of 7519 order. The s_waitcnt 7520 could be placed after 7521 seq_store or before 7522 the seq_load. We 7523 choose the load to 7524 make the s_waitcnt be 7525 as late as possible 7526 so that the store 7527 may have already 7528 completed.) 7529 7530 2. *Following 7531 instructions same as 7532 corresponding load 7533 atomic acquire, 7534 except must generated 7535 all instructions even 7536 for OpenCL.* 7537 store atomic seq_cst - singlethread - global *Same as corresponding 7538 - wavefront - local store atomic release, 7539 - workgroup - generic except must generated 7540 - agent all instructions even 7541 - system for OpenCL.* 7542 atomicrmw seq_cst - singlethread - global *Same as corresponding 7543 - wavefront - local atomicrmw acq_rel, 7544 - workgroup - generic except must generated 7545 - agent all instructions even 7546 - system for OpenCL.* 7547 fence seq_cst - singlethread *none* *Same as corresponding 7548 - wavefront fence acq_rel, 7549 - workgroup except must generated 7550 - agent all instructions even 7551 - system for OpenCL.* 7552 ============ ============ ============== ========== ================================ 7553 7554Trap Handler ABI 7555~~~~~~~~~~~~~~~~ 7556 7557For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 7558(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 7559the ``s_trap`` instruction with the following usage: 7560 7561 .. table:: AMDGPU Trap Handler for AMDHSA OS 7562 :name: amdgpu-trap-handler-for-amdhsa-os-table 7563 7564 =================== =============== =============== ======================= 7565 Usage Code Sequence Trap Handler Description 7566 Inputs 7567 =================== =============== =============== ======================= 7568 reserved ``s_trap 0x00`` Reserved by hardware. 7569 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 7570 ``queue_ptr`` ``debugtrap`` 7571 ``VGPR0``: intrinsic (not 7572 ``arg`` implemented). 7573 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 7574 ``queue_ptr`` terminated and its 7575 associated queue put 7576 into the error state. 7577 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 7578 installed then 7579 behaves as a 7580 no-operation. The 7581 trap handler is 7582 entered and 7583 immediately returns 7584 to continue 7585 execution of the 7586 wavefront. 7587 - If the debugger is 7588 installed, causes 7589 the debug trap to be 7590 reported by the 7591 debugger and the 7592 wavefront is put in 7593 the halt state until 7594 resumed by the 7595 debugger. 7596 reserved ``s_trap 0x04`` Reserved. 7597 reserved ``s_trap 0x05`` Reserved. 7598 reserved ``s_trap 0x06`` Reserved. 7599 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 7600 breakpoints. 7601 reserved ``s_trap 0x08`` Reserved. 7602 reserved ``s_trap 0xfe`` Reserved. 7603 reserved ``s_trap 0xff`` Reserved. 7604 =================== =============== =============== ======================= 7605 7606.. _amdgpu-amdhsa-function-call-convention: 7607 7608Call Convention 7609~~~~~~~~~~~~~~~ 7610 7611.. note:: 7612 7613 This section is currently incomplete and has inaccuracies. It is WIP that will 7614 be updated as information is determined. 7615 7616See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 7617addresses. Unswizzled addresses are normal linear addresses. 7618 7619.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 7620 7621Kernel Functions 7622++++++++++++++++ 7623 7624This section describes the call convention ABI for the outer kernel function. 7625 7626See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 7627convention. 7628 7629The following is not part of the AMDGPU kernel calling convention but describes 7630how the AMDGPU implements function calls: 7631 76321. Clang decides the kernarg layout to match the *HSA Programmer's Language 7633 Reference* [HSA]_. 7634 7635 - All structs are passed directly. 7636 - Lambda values are passed *TBA*. 7637 7638 .. TODO:: 7639 7640 - Does this really follow HSA rules? Or are structs >16 bytes passed 7641 by-value struct? 7642 - What is ABI for lambda values? 7643 76444. The kernel performs certain setup in its prolog, as described in 7645 :ref:`amdgpu-amdhsa-kernel-prolog`. 7646 7647.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 7648 7649Non-Kernel Functions 7650++++++++++++++++++++ 7651 7652This section describes the call convention ABI for functions other than the 7653outer kernel function. 7654 7655If a kernel has function calls then scratch is always allocated and used for 7656the call stack which grows from low address to high address using the swizzled 7657scratch address space. 7658 7659On entry to a function: 7660 76611. SGPR0-3 contain a V# with the following properties (see 7662 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 7663 7664 * Base address pointing to the beginning of the wavefront scratch backing 7665 memory. 7666 * Swizzled with dword element size and stride of wavefront size elements. 7667 76682. The FLAT_SCRATCH register pair is setup. See 7669 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 76703. GFX6-8: M0 register set to the size of LDS in bytes. See 7671 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 76724. The EXEC register is set to the lanes active on entry to the function. 76735. MODE register: *TBD* 76746. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 7675 below. 76767. SGPR30-31 return address (RA). The code address that the function must 7677 return to when it completes. The value is undefined if the function is *no 7678 return*. 76798. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 7680 offset relative to the beginning of the wavefront scratch backing memory. 7681 7682 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 7683 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 7684 manner. 7685 7686 The unswizzled SP value can be converted into the swizzled SP value by: 7687 7688 | swizzled SP = unswizzled SP / wavefront size 7689 7690 This may be used to obtain the private address space address of stack 7691 objects and to convert this address to a flat address by adding the flat 7692 scratch aperture base address. 7693 7694 The swizzled SP value is always 4 bytes aligned for the ``r600`` 7695 architecture and 16 byte aligned for the ``amdgcn`` architecture. 7696 7697 .. note:: 7698 7699 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 7700 OpenCL language which has the largest base type defined as 16 bytes. 7701 7702 On entry, the swizzled SP value is the address of the first function 7703 argument passed on the stack. Other stack passed arguments are positive 7704 offsets from the entry swizzled SP value. 7705 7706 The function may use positive offsets beyond the last stack passed argument 7707 for stack allocated local variables and register spill slots. If necessary, 7708 the function may align these to greater alignment than 16 bytes. After these 7709 the function may dynamically allocate space for such things as runtime sized 7710 ``alloca`` local allocations. 7711 7712 If the function calls another function, it will place any stack allocated 7713 arguments after the last local allocation and adjust SGPR32 to the address 7714 after the last local allocation. 7715 77169. All other registers are unspecified. 771710. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 7718 to the function. 7719 7720On exit from a function: 7721 77221. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 7723 described below. Any registers used are considered clobbered registers. 77242. The following registers are preserved and have the same value as on entry: 7725 7726 * FLAT_SCRATCH 7727 * EXEC 7728 * GFX6-8: M0 7729 * All SGPR registers except the clobbered registers of SGPR4-31. 7730 * VGPR40-47 7731 VGPR56-63 7732 VGPR72-79 7733 VGPR88-95 7734 VGPR104-111 7735 VGPR120-127 7736 VGPR136-143 7737 VGPR152-159 7738 VGPR168-175 7739 VGPR184-191 7740 VGPR200-207 7741 VGPR216-223 7742 VGPR232-239 7743 VGPR248-255 7744 7745 *Except the argument registers, the VGPR clobbered and the preserved 7746 registers are intermixed at regular intervals in order to 7747 get a better occupancy.* 7748 7749 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 7750 optimization may mark some of clobbered SGPR and VGPR registers as 7751 preserved if it can be determined that the called function does not change 7752 their value. 7753 77542. The PC is set to the RA provided on entry. 77553. MODE register: *TBD*. 77564. All other registers are clobbered. 77575. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 7758 function is available to the caller. 7759 7760.. TODO:: 7761 7762 - On gfx908 are all ACC registers clobbered? 7763 7764 - How are function results returned? The address of structured types is passed 7765 by reference, but what about other types? 7766 7767The function input arguments are made up of the formal arguments explicitly 7768declared by the source language function plus the implicit input arguments used 7769by the implementation. 7770 7771The source language input arguments are: 7772 77731. Any source language implicit ``this`` or ``self`` argument comes first as a 7774 pointer type. 77752. Followed by the function formal arguments in left to right source order. 7776 7777The source language result arguments are: 7778 77791. The function result argument. 7780 7781The source language input or result struct type arguments that are less than or 7782equal to 16 bytes, are decomposed recursively into their base type fields, and 7783each field is passed as if a separate argument. For input arguments, if the 7784called function requires the struct to be in memory, for example because its 7785address is taken, then the function body is responsible for allocating a stack 7786location and copying the field arguments into it. Clang terms this *direct 7787struct*. 7788 7789The source language input struct type arguments that are greater than 16 bytes, 7790are passed by reference. The caller is responsible for allocating a stack 7791location to make a copy of the struct value and pass the address as the input 7792argument. The called function is responsible to perform the dereference when 7793accessing the input argument. Clang terms this *by-value struct*. 7794 7795A source language result struct type argument that is greater than 16 bytes, is 7796returned by reference. The caller is responsible for allocating a stack location 7797to hold the result value and passes the address as the last input argument 7798(before the implicit input arguments). In this case there are no result 7799arguments. The called function is responsible to perform the dereference when 7800storing the result value. Clang terms this *structured return (sret)*. 7801 7802*TODO: correct the ``sret`` definition.* 7803 7804.. TODO:: 7805 7806 Is this definition correct? Or is ``sret`` only used if passing in registers, and 7807 pass as non-decomposed struct as stack argument? Or something else? Is the 7808 memory location in the caller stack frame, or a stack memory argument and so 7809 no address is passed as the caller can directly write to the argument stack 7810 location? But then the stack location is still live after return. If an 7811 argument stack location is it the first stack argument or the last one? 7812 7813Lambda argument types are treated as struct types with an implementation defined 7814set of fields. 7815 7816.. TODO:: 7817 7818 Need to specify the ABI for lambda types for AMDGPU. 7819 7820For AMDGPU backend all source language arguments (including the decomposed 7821struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 7822they are passed in SGPRs. 7823 7824The AMDGPU backend walks the function call graph from the leaves to determine 7825which implicit input arguments are used, propagating to each caller of the 7826function. The used implicit arguments are appended to the function arguments 7827after the source language arguments in the following order: 7828 7829.. TODO:: 7830 7831 Is recursion or external functions supported? 7832 78331. Work-Item ID (1 VGPR) 7834 7835 The X, Y and Z work-item ID are packed into a single VGRP with the following 7836 layout. Only fields actually used by the function are set. The other bits 7837 are undefined. 7838 7839 The values come from the initial kernel execution state. See 7840 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 7841 7842 .. table:: Work-item implicit argument layout 7843 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 7844 7845 ======= ======= ============== 7846 Bits Size Field Name 7847 ======= ======= ============== 7848 9:0 10 bits X Work-Item ID 7849 19:10 10 bits Y Work-Item ID 7850 29:20 10 bits Z Work-Item ID 7851 31:30 2 bits Unused 7852 ======= ======= ============== 7853 78542. Dispatch Ptr (2 SGPRs) 7855 7856 The value comes from the initial kernel execution state. See 7857 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7858 78593. Queue Ptr (2 SGPRs) 7860 7861 The value comes from the initial kernel execution state. See 7862 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7863 78644. Kernarg Segment Ptr (2 SGPRs) 7865 7866 The value comes from the initial kernel execution state. See 7867 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7868 78695. Dispatch id (2 SGPRs) 7870 7871 The value comes from the initial kernel execution state. See 7872 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7873 78746. Work-Group ID X (1 SGPR) 7875 7876 The value comes from the initial kernel execution state. See 7877 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7878 78797. Work-Group ID Y (1 SGPR) 7880 7881 The value comes from the initial kernel execution state. See 7882 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7883 78848. Work-Group ID Z (1 SGPR) 7885 7886 The value comes from the initial kernel execution state. See 7887 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7888 78899. Implicit Argument Ptr (2 SGPRs) 7890 7891 The value is computed by adding an offset to Kernarg Segment Ptr to get the 7892 global address space pointer to the first kernarg implicit argument. 7893 7894The input and result arguments are assigned in order in the following manner: 7895 7896.. note:: 7897 7898 There are likely some errors and omissions in the following description that 7899 need correction. 7900 7901 .. TODO:: 7902 7903 Check the clang source code to decipher how function arguments and return 7904 results are handled. Also see the AMDGPU specific values used. 7905 7906* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 7907 VGPR31. 7908 7909 If there are more arguments than will fit in these registers, the remaining 7910 arguments are allocated on the stack in order on naturally aligned 7911 addresses. 7912 7913 .. TODO:: 7914 7915 How are overly aligned structures allocated on the stack? 7916 7917* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 7918 SGPR29. 7919 7920 If there are more arguments than will fit in these registers, the remaining 7921 arguments are allocated on the stack in order on naturally aligned 7922 addresses. 7923 7924Note that decomposed struct type arguments may have some fields passed in 7925registers and some in memory. 7926 7927.. TODO:: 7928 7929 So, a struct which can pass some fields as decomposed register arguments, will 7930 pass the rest as decomposed stack elements? But an argument that will not start 7931 in registers will not be decomposed and will be passed as a non-decomposed 7932 stack value? 7933 7934The following is not part of the AMDGPU function calling convention but 7935describes how the AMDGPU implements function calls: 7936 79371. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 7938 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 7939 are used, or for the reasons defined in ``SIFrameLowering``. 79402. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 7941 to access the incoming stack arguments in the function. The BP is needed 7942 only when the function requires the runtime stack alignment. 7943 79443. Allocating SGPR arguments on the stack are not supported. 7945 79464. No CFI is currently generated. See 7947 :ref:`amdgpu-dwarf-call-frame-information`. 7948 7949 .. note:: 7950 7951 CFI will be generated that defines the CFA as the unswizzled address 7952 relative to the wave scratch base in the unswizzled private address space 7953 of the lowest address stack allocated local variable. 7954 7955 ``DW_AT_frame_base`` will be defined as the swizzled address in the 7956 swizzled private address space by dividing the CFA by the wavefront size 7957 (since CFA is always at least dword aligned which matches the scratch 7958 swizzle element size). 7959 7960 If no dynamic stack alignment was performed, the stack allocated arguments 7961 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 7962 local variables and register spill slots are accessed as positive offsets 7963 relative to ``DW_AT_frame_base``. 7964 79655. Function argument passing is implemented by copying the input physical 7966 registers to virtual registers on entry. The register allocator can spill if 7967 necessary. These are copied back to physical registers at call sites. The 7968 net effect is that each function call can have these values in entirely 7969 distinct locations. The IPRA can help avoid shuffling argument registers. 79706. Call sites are implemented by setting up the arguments at positive offsets 7971 from SP. Then SP is incremented to account for the known frame size before 7972 the call and decremented after the call. 7973 7974 .. note:: 7975 7976 The CFI will reflect the changed calculation needed to compute the CFA 7977 from SP. 7978 79797. 4 byte spill slots are used in the stack frame. One slot is allocated for an 7980 emergency spill slot. Buffer instructions are used for stack accesses and 7981 not the ``flat_scratch`` instruction. 7982 7983 .. TODO:: 7984 7985 Explain when the emergency spill slot is used. 7986 7987.. TODO:: 7988 7989 Possible broken issues: 7990 7991 - Stack arguments must be aligned to required alignment. 7992 - Stack is aligned to max(16, max formal argument alignment) 7993 - Direct argument < 64 bits should check register budget. 7994 - Register budget calculation should respect ``inreg`` for SGPR. 7995 - SGPR overflow is not handled. 7996 - struct with 1 member unpeeling is not checking size of member. 7997 - ``sret`` is after ``this`` pointer. 7998 - Caller is not implementing stack realignment: need an extra pointer. 7999 - Should say AMDGPU passes FP rather than SP. 8000 - Should CFI define CFA as address of locals or arguments. Difference is 8001 apparent when have implemented dynamic alignment. 8002 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 8003 highest address of stack frame and use negative offset for locals. Would 8004 allow SP to be the same as FP and could support signal-handler-like as now 8005 have a real SP for the top of the stack. 8006 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 8007 arguments? 8008 8009AMDPAL 8010------ 8011 8012This section provides code conventions used when the target triple OS is 8013``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 8014from the application/runtime to each invocation of a hardware shader. These 8015parameters include both generic, application-controlled parameters called 8016*user data* as well as system-generated parameters that are a product of the 8017draw or dispatch execution. 8018 8019User Data 8020~~~~~~~~~ 8021 8022Each hardware stage has a set of 32-bit *user data registers* which can be 8023written from a command buffer and then loaded into SGPRs when waves are launched 8024via a subsequent dispatch or draw operation. This is the way most arguments are 8025passed from the application/runtime to a hardware shader. 8026 8027Compute User Data 8028~~~~~~~~~~~~~~~~~ 8029 8030Compute shader user data mappings are simpler than graphics shaders and have a 8031fixed mapping. 8032 8033Note that there are always 10 available *user data entries* in registers - 8034entries beyond that limit must be fetched from memory (via the spill table 8035pointer) by the shader. 8036 8037 .. table:: PAL Compute Shader User Data Registers 8038 :name: pal-compute-user-data-registers 8039 8040 ============= ================================ 8041 User Register Description 8042 ============= ================================ 8043 0 Global Internal Table (32-bit pointer) 8044 1 Per-Shader Internal Table (32-bit pointer) 8045 2 - 11 Application-Controlled User Data (10 32-bit values) 8046 12 Spill Table (32-bit pointer) 8047 13 - 14 Thread Group Count (64-bit pointer) 8048 15 GDS Range 8049 ============= ================================ 8050 8051Graphics User Data 8052~~~~~~~~~~~~~~~~~~ 8053 8054Graphics pipelines support a much more flexible user data mapping: 8055 8056 .. table:: PAL Graphics Shader User Data Registers 8057 :name: pal-graphics-user-data-registers 8058 8059 ============= ================================ 8060 User Register Description 8061 ============= ================================ 8062 0 Global Internal Table (32-bit pointer) 8063 + Per-Shader Internal Table (32-bit pointer) 8064 + 1-15 Application Controlled User Data 8065 (1-15 Contiguous 32-bit Values in Registers) 8066 + Spill Table (32-bit pointer) 8067 + Draw Index (First Stage Only) 8068 + Vertex Offset (First Stage Only) 8069 + Instance Offset (First Stage Only) 8070 ============= ================================ 8071 8072 The placement of the global internal table remains fixed in the first *user 8073 data SGPR register*. Otherwise all parameters are optional, and can be mapped 8074 to any desired *user data SGPR register*, with the following restrictions: 8075 8076 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 8077 active hardware stage in a graphics pipeline (i.e. where the API vertex 8078 shader runs). 8079 8080 * Application-controlled user data must be mapped into a contiguous range of 8081 user data registers. 8082 8083 * The application-controlled user data range supports compaction remapping, so 8084 only *entries* that are actually consumed by the shader must be assigned to 8085 corresponding *registers*. Note that in order to support an efficient runtime 8086 implementation, the remapping must pack *registers* in the same order as 8087 *entries*, with unused *entries* removed. 8088 8089.. _pal_global_internal_table: 8090 8091Global Internal Table 8092~~~~~~~~~~~~~~~~~~~~~ 8093 8094The global internal table is a table of *shader resource descriptors* (SRDs) 8095that define how certain engine-wide, runtime-managed resources should be 8096accessed from a shader. The majority of these resources have HW-defined formats, 8097and it is up to the compiler to write/read data as required by the target 8098hardware. 8099 8100The following table illustrates the required format: 8101 8102 .. table:: PAL Global Internal Table 8103 :name: pal-git-table 8104 8105 ============= ================================ 8106 Offset Description 8107 ============= ================================ 8108 0-3 Graphics Scratch SRD 8109 4-7 Compute Scratch SRD 8110 8-11 ES/GS Ring Output SRD 8111 12-15 ES/GS Ring Input SRD 8112 16-19 GS/VS Ring Output #0 8113 20-23 GS/VS Ring Output #1 8114 24-27 GS/VS Ring Output #2 8115 28-31 GS/VS Ring Output #3 8116 32-35 GS/VS Ring Input SRD 8117 36-39 Tessellation Factor Buffer SRD 8118 40-43 Off-Chip LDS Buffer SRD 8119 44-47 Off-Chip Param Cache Buffer SRD 8120 48-51 Sample Position Buffer SRD 8121 52 vaRange::ShadowDescriptorTable High Bits 8122 ============= ================================ 8123 8124 The pointer to the global internal table passed to the shader as user data 8125 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 8126 the top 32 bits of the pipeline, so the shader may use the program 8127 counter's top 32 bits. 8128 8129.. _pal_call-convention: 8130 8131Call Convention 8132~~~~~~~~~~~~~~~ 8133 8134For graphics use cases, the calling convention is `amdgpu_gfx`. 8135 8136.. note:: 8137 8138 `amdgpu_gfx` Function calls are currently in development and are 8139 subject to major changes. 8140 8141This calling convention shares most properties with calling non-kernel 8142functions (see 8143:ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`). 8144Differences are: 8145 8146 - Currently there are none, differences will be listed here 8147 8148Unspecified OS 8149-------------- 8150 8151This section provides code conventions used when the target triple OS is 8152empty (see :ref:`amdgpu-target-triples`). 8153 8154Trap Handler ABI 8155~~~~~~~~~~~~~~~~ 8156 8157For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 8158not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 8159instructions are handled as follows: 8160 8161 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 8162 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 8163 8164 =============== =============== =========================================== 8165 Usage Code Sequence Description 8166 =============== =============== =========================================== 8167 llvm.trap s_endpgm Causes wavefront to be terminated. 8168 llvm.debugtrap *none* Compiler warning given that there is no 8169 trap handler installed. 8170 =============== =============== =========================================== 8171 8172Source Languages 8173================ 8174 8175.. _amdgpu-opencl: 8176 8177OpenCL 8178------ 8179 8180When the language is OpenCL the following differences occur: 8181 81821. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 81832. The AMDGPU backend appends additional arguments to the kernel's explicit 8184 arguments for the AMDHSA OS (see 8185 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 81863. Additional metadata is generated 8187 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 8188 8189 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 8190 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 8191 8192 ======== ==== ========= =========================================== 8193 Position Byte Byte Description 8194 Size Alignment 8195 ======== ==== ========= =========================================== 8196 1 8 8 OpenCL Global Offset X 8197 2 8 8 OpenCL Global Offset Y 8198 3 8 8 OpenCL Global Offset Z 8199 4 8 8 OpenCL address of printf buffer 8200 5 8 8 OpenCL address of virtual queue used by 8201 enqueue_kernel. 8202 6 8 8 OpenCL address of AqlWrap struct used by 8203 enqueue_kernel. 8204 7 8 8 Pointer argument used for Multi-gird 8205 synchronization. 8206 ======== ==== ========= =========================================== 8207 8208.. _amdgpu-hcc: 8209 8210HCC 8211--- 8212 8213When the language is HCC the following differences occur: 8214 82151. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 8216 8217.. _amdgpu-assembler: 8218 8219Assembler 8220--------- 8221 8222AMDGPU backend has LLVM-MC based assembler which is currently in development. 8223It supports AMDGCN GFX6-GFX10. 8224 8225This section describes general syntax for instructions and operands. 8226 8227Instructions 8228~~~~~~~~~~~~ 8229 8230An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 8231 8232 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 8233 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 8234 8235:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 8236:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 8237 8238The order of operands and modifiers is fixed. 8239Most modifiers are optional and may be omitted. 8240 8241Links to detailed instruction syntax description may be found in the following 8242table. Note that features under development are not included 8243in this description. 8244 8245 =================================== ======================================= 8246 Core ISA ISA Extensions 8247 =================================== ======================================= 8248 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 8249 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 8250 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 8251 8252 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 8253 8254 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 8255 8256 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 8257 8258 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 8259 8260 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 8261 8262 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 8263 8264 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 8265 =================================== ======================================= 8266 8267For more information about instructions, their semantics and supported 8268combinations of operands, refer to one of instruction set architecture manuals 8269[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 8270[AMD-GCN-GFX10]_. 8271 8272Operands 8273~~~~~~~~ 8274 8275Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 8276 8277Modifiers 8278~~~~~~~~~ 8279 8280Detailed description of modifiers may be found 8281:doc:`here<AMDGPUModifierSyntax>`. 8282 8283Instruction Examples 8284~~~~~~~~~~~~~~~~~~~~ 8285 8286DS 8287++ 8288 8289.. code-block:: nasm 8290 8291 ds_add_u32 v2, v4 offset:16 8292 ds_write_src2_b64 v2 offset0:4 offset1:8 8293 ds_cmpst_f32 v2, v4, v6 8294 ds_min_rtn_f64 v[8:9], v2, v[4:5] 8295 8296For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 8297Manual. 8298 8299FLAT 8300++++ 8301 8302.. code-block:: nasm 8303 8304 flat_load_dword v1, v[3:4] 8305 flat_store_dwordx3 v[3:4], v[5:7] 8306 flat_atomic_swap v1, v[3:4], v5 glc 8307 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 8308 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 8309 8310For full list of supported instructions, refer to "FLAT instructions" in ISA 8311Manual. 8312 8313MUBUF 8314+++++ 8315 8316.. code-block:: nasm 8317 8318 buffer_load_dword v1, off, s[4:7], s1 8319 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 8320 buffer_store_format_xy v[1:2], off, s[4:7], s1 8321 buffer_wbinvl1 8322 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 8323 8324For full list of supported instructions, refer to "MUBUF Instructions" in ISA 8325Manual. 8326 8327SMRD/SMEM 8328+++++++++ 8329 8330.. code-block:: nasm 8331 8332 s_load_dword s1, s[2:3], 0xfc 8333 s_load_dwordx8 s[8:15], s[2:3], s4 8334 s_load_dwordx16 s[88:103], s[2:3], s4 8335 s_dcache_inv_vol 8336 s_memtime s[4:5] 8337 8338For full list of supported instructions, refer to "Scalar Memory Operations" in 8339ISA Manual. 8340 8341SOP1 8342++++ 8343 8344.. code-block:: nasm 8345 8346 s_mov_b32 s1, s2 8347 s_mov_b64 s[0:1], 0x80000000 8348 s_cmov_b32 s1, 200 8349 s_wqm_b64 s[2:3], s[4:5] 8350 s_bcnt0_i32_b64 s1, s[2:3] 8351 s_swappc_b64 s[2:3], s[4:5] 8352 s_cbranch_join s[4:5] 8353 8354For full list of supported instructions, refer to "SOP1 Instructions" in ISA 8355Manual. 8356 8357SOP2 8358++++ 8359 8360.. code-block:: nasm 8361 8362 s_add_u32 s1, s2, s3 8363 s_and_b64 s[2:3], s[4:5], s[6:7] 8364 s_cselect_b32 s1, s2, s3 8365 s_andn2_b32 s2, s4, s6 8366 s_lshr_b64 s[2:3], s[4:5], s6 8367 s_ashr_i32 s2, s4, s6 8368 s_bfm_b64 s[2:3], s4, s6 8369 s_bfe_i64 s[2:3], s[4:5], s6 8370 s_cbranch_g_fork s[4:5], s[6:7] 8371 8372For full list of supported instructions, refer to "SOP2 Instructions" in ISA 8373Manual. 8374 8375SOPC 8376++++ 8377 8378.. code-block:: nasm 8379 8380 s_cmp_eq_i32 s1, s2 8381 s_bitcmp1_b32 s1, s2 8382 s_bitcmp0_b64 s[2:3], s4 8383 s_setvskip s3, s5 8384 8385For full list of supported instructions, refer to "SOPC Instructions" in ISA 8386Manual. 8387 8388SOPP 8389++++ 8390 8391.. code-block:: nasm 8392 8393 s_barrier 8394 s_nop 2 8395 s_endpgm 8396 s_waitcnt 0 ; Wait for all counters to be 0 8397 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 8398 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 8399 s_sethalt 9 8400 s_sleep 10 8401 s_sendmsg 0x1 8402 s_sendmsg sendmsg(MSG_INTERRUPT) 8403 s_trap 1 8404 8405For full list of supported instructions, refer to "SOPP Instructions" in ISA 8406Manual. 8407 8408Unless otherwise mentioned, little verification is performed on the operands 8409of SOPP Instructions, so it is up to the programmer to be familiar with the 8410range or acceptable values. 8411 8412VALU 8413++++ 8414 8415For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 8416the assembler will automatically use optimal encoding based on its operands. To 8417force specific encoding, one can add a suffix to the opcode of the instruction: 8418 8419* _e32 for 32-bit VOP1/VOP2/VOPC 8420* _e64 for 64-bit VOP3 8421* _dpp for VOP_DPP 8422* _sdwa for VOP_SDWA 8423 8424VOP1/VOP2/VOP3/VOPC examples: 8425 8426.. code-block:: nasm 8427 8428 v_mov_b32 v1, v2 8429 v_mov_b32_e32 v1, v2 8430 v_nop 8431 v_cvt_f64_i32_e32 v[1:2], v2 8432 v_floor_f32_e32 v1, v2 8433 v_bfrev_b32_e32 v1, v2 8434 v_add_f32_e32 v1, v2, v3 8435 v_mul_i32_i24_e64 v1, v2, 3 8436 v_mul_i32_i24_e32 v1, -3, v3 8437 v_mul_i32_i24_e32 v1, -100, v3 8438 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 8439 v_max_f16_e32 v1, v2, v3 8440 8441VOP_DPP examples: 8442 8443.. code-block:: nasm 8444 8445 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 8446 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 8447 v_mov_b32 v0, v0 wave_shl:1 8448 v_mov_b32 v0, v0 row_mirror 8449 v_mov_b32 v0, v0 row_bcast:31 8450 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 8451 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 8452 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 8453 8454VOP_SDWA examples: 8455 8456.. code-block:: nasm 8457 8458 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 8459 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 8460 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 8461 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 8462 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 8463 8464For full list of supported instructions, refer to "Vector ALU instructions". 8465 8466.. TODO:: 8467 8468 Remove once we switch to code object v3 by default. 8469 8470.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 8471 8472Code Object V2 Predefined Symbols (--amdhsa-code-object-version=2) 8473~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8474 8475.. warning:: Code Object V2 is not the default code object version emitted by 8476 this version of LLVM. For a description of the predefined symbols available 8477 with the default configuration (Code Object V3) see 8478 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 8479 8480The AMDGPU assembler defines and updates some symbols automatically. These 8481symbols do not affect code generation. 8482 8483.option.machine_version_major 8484+++++++++++++++++++++++++++++ 8485 8486Set to the GFX major generation number of the target being assembled for. For 8487example, when assembling for a "GFX9" target this will be set to the integer 8488value "9". The possible GFX major generation numbers are presented in 8489:ref:`amdgpu-processors`. 8490 8491.option.machine_version_minor 8492+++++++++++++++++++++++++++++ 8493 8494Set to the GFX minor generation number of the target being assembled for. For 8495example, when assembling for a "GFX810" target this will be set to the integer 8496value "1". The possible GFX minor generation numbers are presented in 8497:ref:`amdgpu-processors`. 8498 8499.option.machine_version_stepping 8500++++++++++++++++++++++++++++++++ 8501 8502Set to the GFX stepping generation number of the target being assembled for. 8503For example, when assembling for a "GFX704" target this will be set to the 8504integer value "4". The possible GFX stepping generation numbers are presented 8505in :ref:`amdgpu-processors`. 8506 8507.kernel.vgpr_count 8508++++++++++++++++++ 8509 8510Set to zero each time a 8511:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 8512encountered. At each instruction, if the current value of this symbol is less 8513than or equal to the maximum VGPR number explicitly referenced within that 8514instruction then the symbol value is updated to equal that VGPR number plus 8515one. 8516 8517.kernel.sgpr_count 8518++++++++++++++++++ 8519 8520Set to zero each time a 8521:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 8522encountered. At each instruction, if the current value of this symbol is less 8523than or equal to the maximum VGPR number explicitly referenced within that 8524instruction then the symbol value is updated to equal that SGPR number plus 8525one. 8526 8527.. _amdgpu-amdhsa-assembler-directives-v2: 8528 8529Code Object V2 Directives (--amdhsa-code-object-version=2) 8530~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8531 8532.. warning:: Code Object V2 is not the default code object version emitted by 8533 this version of LLVM. For a description of the directives supported with 8534 the default configuration (Code Object V3) see 8535 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 8536 8537AMDGPU ABI defines auxiliary data in output code object. In assembly source, 8538one can specify them with assembler directives. 8539 8540.hsa_code_object_version major, minor 8541+++++++++++++++++++++++++++++++++++++ 8542 8543*major* and *minor* are integers that specify the version of the HSA code 8544object that will be generated by the assembler. 8545 8546.hsa_code_object_isa [major, minor, stepping, vendor, arch] 8547+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 8548 8549 8550*major*, *minor*, and *stepping* are all integers that describe the instruction 8551set architecture (ISA) version of the assembly program. 8552 8553*vendor* and *arch* are quoted strings. *vendor* should always be equal to 8554"AMD" and *arch* should always be equal to "AMDGPU". 8555 8556By default, the assembler will derive the ISA version, *vendor*, and *arch* 8557from the value of the -mcpu option that is passed to the assembler. 8558 8559.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 8560 8561.amdgpu_hsa_kernel (name) 8562+++++++++++++++++++++++++ 8563 8564This directives specifies that the symbol with given name is a kernel entry 8565point (label) and the object should contain corresponding symbol of type 8566STT_AMDGPU_HSA_KERNEL. 8567 8568.amd_kernel_code_t 8569++++++++++++++++++ 8570 8571This directive marks the beginning of a list of key / value pairs that are used 8572to specify the amd_kernel_code_t object that will be emitted by the assembler. 8573The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 8574amd_kernel_code_t values that are unspecified a default value will be used. The 8575default value for all keys is 0, with the following exceptions: 8576 8577- *amd_code_version_major* defaults to 1. 8578- *amd_kernel_code_version_minor* defaults to 2. 8579- *amd_machine_kind* defaults to 1. 8580- *amd_machine_version_major*, *machine_version_minor*, and 8581 *amd_machine_version_stepping* are derived from the value of the -mcpu option 8582 that is passed to the assembler. 8583- *kernel_code_entry_byte_offset* defaults to 256. 8584- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 8585 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 8586 Note that wavefront size is specified as a power of two, so a value of **n** 8587 means a size of 2^ **n**. 8588- *call_convention* defaults to -1. 8589- *kernarg_segment_alignment*, *group_segment_alignment*, and 8590 *private_segment_alignment* default to 4. Note that alignments are specified 8591 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 8592- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 8593 GFX10 onwards. 8594- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 8595 8596The *.amd_kernel_code_t* directive must be placed immediately after the 8597function label and before any instructions. 8598 8599For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 8600comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 8601 8602.. _amdgpu-amdhsa-assembler-example-v2: 8603 8604Code Object V2 Example Source Code (--amdhsa-code-object-version=2) 8605~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8606 8607.. warning:: Code Object V2 is not the default code object version emitted by 8608 this version of LLVM. For a description of the directives supported with 8609 the default configuration (Code Object V3) see 8610 :ref:`amdgpu-amdhsa-assembler-example-v3`. 8611 8612Here is an example of a minimal assembly source file, defining one HSA kernel: 8613 8614.. code:: 8615 :number-lines: 8616 8617 .hsa_code_object_version 1,0 8618 .hsa_code_object_isa 8619 8620 .hsatext 8621 .globl hello_world 8622 .p2align 8 8623 .amdgpu_hsa_kernel hello_world 8624 8625 hello_world: 8626 8627 .amd_kernel_code_t 8628 enable_sgpr_kernarg_segment_ptr = 1 8629 is_ptr64 = 1 8630 compute_pgm_rsrc1_vgprs = 0 8631 compute_pgm_rsrc1_sgprs = 0 8632 compute_pgm_rsrc2_user_sgpr = 2 8633 compute_pgm_rsrc1_wgp_mode = 0 8634 compute_pgm_rsrc1_mem_ordered = 0 8635 compute_pgm_rsrc1_fwd_progress = 1 8636 .end_amd_kernel_code_t 8637 8638 s_load_dwordx2 s[0:1], s[0:1] 0x0 8639 v_mov_b32 v0, 3.14159 8640 s_waitcnt lgkmcnt(0) 8641 v_mov_b32 v1, s0 8642 v_mov_b32 v2, s1 8643 flat_store_dword v[1:2], v0 8644 s_endpgm 8645 .Lfunc_end0: 8646 .size hello_world, .Lfunc_end0-hello_world 8647 8648.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 8649 8650Code Object V3 Predefined Symbols (--amdhsa-code-object-version=3) 8651~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8652 8653The AMDGPU assembler defines and updates some symbols automatically. These 8654symbols do not affect code generation. 8655 8656.amdgcn.gfx_generation_number 8657+++++++++++++++++++++++++++++ 8658 8659Set to the GFX major generation number of the target being assembled for. For 8660example, when assembling for a "GFX9" target this will be set to the integer 8661value "9". The possible GFX major generation numbers are presented in 8662:ref:`amdgpu-processors`. 8663 8664.amdgcn.gfx_generation_minor 8665++++++++++++++++++++++++++++ 8666 8667Set to the GFX minor generation number of the target being assembled for. For 8668example, when assembling for a "GFX810" target this will be set to the integer 8669value "1". The possible GFX minor generation numbers are presented in 8670:ref:`amdgpu-processors`. 8671 8672.amdgcn.gfx_generation_stepping 8673+++++++++++++++++++++++++++++++ 8674 8675Set to the GFX stepping generation number of the target being assembled for. 8676For example, when assembling for a "GFX704" target this will be set to the 8677integer value "4". The possible GFX stepping generation numbers are presented 8678in :ref:`amdgpu-processors`. 8679 8680.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 8681 8682.amdgcn.next_free_vgpr 8683++++++++++++++++++++++ 8684 8685Set to zero before assembly begins. At each instruction, if the current value 8686of this symbol is less than or equal to the maximum VGPR number explicitly 8687referenced within that instruction then the symbol value is updated to equal 8688that VGPR number plus one. 8689 8690May be used to set the `.amdhsa_next_free_vgpr` directive in 8691:ref:`amdhsa-kernel-directives-table`. 8692 8693May be set at any time, e.g. manually set to zero at the start of each kernel. 8694 8695.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 8696 8697.amdgcn.next_free_sgpr 8698++++++++++++++++++++++ 8699 8700Set to zero before assembly begins. At each instruction, if the current value 8701of this symbol is less than or equal the maximum SGPR number explicitly 8702referenced within that instruction then the symbol value is updated to equal 8703that SGPR number plus one. 8704 8705May be used to set the `.amdhsa_next_free_spgr` directive in 8706:ref:`amdhsa-kernel-directives-table`. 8707 8708May be set at any time, e.g. manually set to zero at the start of each kernel. 8709 8710.. _amdgpu-amdhsa-assembler-directives-v3: 8711 8712Code Object V3 Directives (--amdhsa-code-object-version=3) 8713~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8714 8715Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 8716architecture processors, and are not OS-specific. Directives which begin with 8717``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 8718``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 8719:ref:`amdgpu-processors`. 8720 8721.amdgcn_target <target> 8722+++++++++++++++++++++++ 8723 8724Optional directive which declares the target supported by the containing 8725assembler source file. Valid values are described in 8726:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 8727to validate command-line options such as ``-triple``, ``-mcpu``, and those 8728which specify target features. 8729 8730.amdhsa_kernel <name> 8731+++++++++++++++++++++ 8732 8733Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 8734``<name>.kd``, in the current location of the current section. Only valid when 8735the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 8736instruction to execute, and does not need to be previously defined. 8737 8738Marks the beginning of a list of directives used to generate the bytes of a 8739kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 8740Directives which may appear in this list are described in 8741:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 8742be valid for the target being assembled for, and cannot be repeated. Directives 8743support the range of values specified by the field they reference in 8744:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 8745assumed to have its default value, unless it is marked as "Required", in which 8746case it is an error to omit the directive. This list of directives is 8747terminated by an ``.end_amdhsa_kernel`` directive. 8748 8749 .. table:: AMDHSA Kernel Assembler Directives 8750 :name: amdhsa-kernel-directives-table 8751 8752 ======================================================== =================== ============ =================== 8753 Directive Default Supported On Description 8754 ======================================================== =================== ============ =================== 8755 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 8756 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8757 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 8758 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8759 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 8760 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8761 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 8762 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8763 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 8764 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8765 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 8766 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8767 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 8768 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8769 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 8770 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8771 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 8772 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8773 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 8774 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8775 Specific 8776 (-wavefrontsize64) 8777 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 8778 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8779 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 8780 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8781 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 8782 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8783 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 8784 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8785 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 8786 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8787 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 8788 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8789 Possible values are defined in 8790 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 8791 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 8792 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 8793 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8794 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 8795 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 8796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8797 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 8798 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 8799 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8800 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 8801 scratch memory. Used to calculate 8802 GRANULATED_WAVEFRONT_SGPR_COUNT in 8803 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8804 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 8805 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 8806 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8807 (+xnack) 8808 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 8809 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8810 Possible values are defined in 8811 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 8812 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 8813 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8814 Possible values are defined in 8815 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 8816 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 8817 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8818 Possible values are defined in 8819 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 8820 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 8821 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8822 Possible values are defined in 8823 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 8824 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 8825 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8826 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 8827 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8828 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 8829 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8830 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 8831 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table-v3`. 8832 Specific 8833 (-cumode) 8834 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 8835 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8836 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 8837 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8838 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 8839 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8840 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 8841 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8842 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 8843 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8844 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 8845 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8846 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 8847 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8848 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 8849 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8850 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 8851 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8852 ======================================================== =================== ============ =================== 8853 8854.amdgpu_metadata 8855++++++++++++++++ 8856 8857Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 8858note record (see :ref:`amdgpu-elf-note-records-table-v3`). 8859 8860The contents must be in the [YAML]_ markup format, with the same structure and 8861semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 8862 8863This directive is terminated by an ``.end_amdgpu_metadata`` directive. 8864 8865.. _amdgpu-amdhsa-assembler-example-v3: 8866 8867Code Object V3 Example Source Code (--amdhsa-code-object-version=3) 8868~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8869 8870Here is an example of a minimal assembly source file, defining one HSA kernel: 8871 8872.. code:: 8873 :number-lines: 8874 8875 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 8876 8877 .text 8878 .globl hello_world 8879 .p2align 8 8880 .type hello_world,@function 8881 hello_world: 8882 s_load_dwordx2 s[0:1], s[0:1] 0x0 8883 v_mov_b32 v0, 3.14159 8884 s_waitcnt lgkmcnt(0) 8885 v_mov_b32 v1, s0 8886 v_mov_b32 v2, s1 8887 flat_store_dword v[1:2], v0 8888 s_endpgm 8889 .Lfunc_end0: 8890 .size hello_world, .Lfunc_end0-hello_world 8891 8892 .rodata 8893 .p2align 6 8894 .amdhsa_kernel hello_world 8895 .amdhsa_user_sgpr_kernarg_segment_ptr 1 8896 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 8897 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 8898 .end_amdhsa_kernel 8899 8900 .amdgpu_metadata 8901 --- 8902 amdhsa.version: 8903 - 1 8904 - 0 8905 amdhsa.kernels: 8906 - .name: hello_world 8907 .symbol: hello_world.kd 8908 .kernarg_segment_size: 48 8909 .group_segment_fixed_size: 0 8910 .private_segment_fixed_size: 0 8911 .kernarg_segment_align: 4 8912 .wavefront_size: 64 8913 .sgpr_count: 2 8914 .vgpr_count: 3 8915 .max_flat_workgroup_size: 256 8916 ... 8917 .end_amdgpu_metadata 8918 8919If an assembly source file contains multiple kernels and/or functions, the 8920:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 8921:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 8922the ``.set <symbol>, <expression>`` directive. For example, in the case of two 8923kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 8924to group the function with the kernel that calls it and reset the symbols 8925between the two connected components: 8926 8927.. code:: 8928 :number-lines: 8929 8930 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 8931 8932 // gpr tracking symbols are implicitly set to zero 8933 8934 .text 8935 .globl kern0 8936 .p2align 8 8937 .type kern0,@function 8938 kern0: 8939 // ... 8940 s_endpgm 8941 .Lkern0_end: 8942 .size kern0, .Lkern0_end-kern0 8943 8944 .rodata 8945 .p2align 6 8946 .amdhsa_kernel kern0 8947 // ... 8948 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 8949 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 8950 .end_amdhsa_kernel 8951 8952 // reset symbols to begin tracking usage in func1 and kern1 8953 .set .amdgcn.next_free_vgpr, 0 8954 .set .amdgcn.next_free_sgpr, 0 8955 8956 .text 8957 .hidden func1 8958 .global func1 8959 .p2align 2 8960 .type func1,@function 8961 func1: 8962 // ... 8963 s_setpc_b64 s[30:31] 8964 .Lfunc1_end: 8965 .size func1, .Lfunc1_end-func1 8966 8967 .globl kern1 8968 .p2align 8 8969 .type kern1,@function 8970 kern1: 8971 // ... 8972 s_getpc_b64 s[4:5] 8973 s_add_u32 s4, s4, func1@rel32@lo+4 8974 s_addc_u32 s5, s5, func1@rel32@lo+4 8975 s_swappc_b64 s[30:31], s[4:5] 8976 // ... 8977 s_endpgm 8978 .Lkern1_end: 8979 .size kern1, .Lkern1_end-kern1 8980 8981 .rodata 8982 .p2align 6 8983 .amdhsa_kernel kern1 8984 // ... 8985 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 8986 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 8987 .end_amdhsa_kernel 8988 8989These symbols cannot identify connected components in order to automatically 8990track the usage for each kernel. However, in some cases careful organization of 8991the kernels and functions in the source file means there is minimal additional 8992effort required to accurately calculate GPR usage. 8993 8994Additional Documentation 8995======================== 8996 8997.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 8998.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 8999.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 9000.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 9001.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 9002.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 9003.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 9004.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 9005.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 9006.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 9007.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 9008.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 9009.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 9010.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 9011.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 9012.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 9013.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 9014.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 9015.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 9016.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 9017