1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX10 19 AMDGPU/AMDGPUAsmGFX1011 20 AMDGPUModifierSyntax 21 AMDGPUOperandSyntax 22 AMDGPUInstructionSyntax 23 AMDGPUInstructionNotation 24 AMDGPUDwarfExtensionsForHeterogeneousDebugging 25 26Introduction 27============ 28 29The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 30R600 family up until the current GCN families. It lives in the 31``llvm/lib/Target/AMDGPU`` directory. 32 33LLVM 34==== 35 36.. _amdgpu-target-triples: 37 38Target Triples 39-------------- 40 41Use the ``clang -target <Architecture>-<Vendor>-<OS>-<Environment>`` option to 42specify the target triple: 43 44 .. table:: AMDGPU Architectures 45 :name: amdgpu-architecture-table 46 47 ============ ============================================================== 48 Architecture Description 49 ============ ============================================================== 50 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 51 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 52 ============ ============================================================== 53 54 .. table:: AMDGPU Vendors 55 :name: amdgpu-vendor-table 56 57 ============ ============================================================== 58 Vendor Description 59 ============ ============================================================== 60 ``amd`` Can be used for all AMD GPU usage. 61 ``mesa3d`` Can be used if the OS is ``mesa3d``. 62 ============ ============================================================== 63 64 .. table:: AMDGPU Operating Systems 65 :name: amdgpu-os-table 66 67 ============== ============================================================ 68 OS Description 69 ============== ============================================================ 70 *<empty>* Defaults to the *unknown* OS. 71 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 72 such as AMD's ROCm [AMD-ROCm]_. 73 ``amdpal`` Graphic shaders and compute kernels executed on AMD PAL 74 runtime. 75 ``mesa3d`` Graphic shaders and compute kernels executed on Mesa 3D 76 runtime. 77 ============== ============================================================ 78 79 .. table:: AMDGPU Environments 80 :name: amdgpu-environment-table 81 82 ============ ============================================================== 83 Environment Description 84 ============ ============================================================== 85 *<empty>* Default. 86 ============ ============================================================== 87 88.. _amdgpu-processors: 89 90Processors 91---------- 92 93Use the ``clang -mcpu <Processor>`` option to specify the AMDGPU processor. The 94names from both the *Processor* and *Alternative Processor* can be used. 95 96 .. table:: AMDGPU Processors 97 :name: amdgpu-processor-table 98 99 =========== =============== ============ ===== ============================= ======= ====================== 100 Processor Alternative Target dGPU/ Target ROCm Example 101 Processor Triple APU Features Support Products 102 Architecture Supported 103 [Default] 104 =========== =============== ============ ===== ============================= ======= ====================== 105 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 106 ----------------------------------------------------------------------------------------------------------- 107 ``r600`` ``r600`` dGPU 108 ``r630`` ``r600`` dGPU 109 ``rs880`` ``r600`` dGPU 110 ``rv670`` ``r600`` dGPU 111 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 112 ----------------------------------------------------------------------------------------------------------- 113 ``rv710`` ``r600`` dGPU 114 ``rv730`` ``r600`` dGPU 115 ``rv770`` ``r600`` dGPU 116 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 117 ----------------------------------------------------------------------------------------------------------- 118 ``cedar`` ``r600`` dGPU 119 ``cypress`` ``r600`` dGPU 120 ``juniper`` ``r600`` dGPU 121 ``redwood`` ``r600`` dGPU 122 ``sumo`` ``r600`` dGPU 123 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 124 ----------------------------------------------------------------------------------------------------------- 125 ``barts`` ``r600`` dGPU 126 ``caicos`` ``r600`` dGPU 127 ``cayman`` ``r600`` dGPU 128 ``turks`` ``r600`` dGPU 129 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 130 ----------------------------------------------------------------------------------------------------------- 131 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU 132 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU 133 - ``verde`` 134 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU 135 - ``oland`` 136 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 137 ----------------------------------------------------------------------------------------------------------- 138 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - A6-7000 139 - A6 Pro-7050B 140 - A8-7100 141 - A8 Pro-7150B 142 - A10-7300 143 - A10 Pro-7350B 144 - FX-7500 145 - A8-7200P 146 - A10-7400P 147 - FX-7600P 148 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU ROCm - FirePro W8100 149 - FirePro W9100 150 - FirePro S9150 151 - FirePro S9170 152 ``gfx702`` ``amdgcn`` dGPU ROCm - Radeon R9 290 153 - Radeon R9 290x 154 - Radeon R390 155 - Radeon R390x 156 ``gfx703`` - ``kabini`` ``amdgcn`` APU - E1-2100 157 - ``mullins`` - E1-2200 158 - E1-2500 159 - E2-3000 160 - E2-3800 161 - A4-5000 162 - A4-5100 163 - A6-5200 164 - A4 Pro-3340B 165 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Radeon HD 7790 166 - Radeon HD 8770 167 - R7 260 168 - R7 260X 169 ``gfx705`` ``amdgcn`` APU *TBA* 170 171 .. TODO:: 172 173 Add product 174 names. 175 176 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 177 ----------------------------------------------------------------------------------------------------------- 178 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - A6-8500P 179 [on] - Pro A6-8500B 180 - A8-8600P 181 - Pro A8-8600B 182 - FX-8800P 183 - Pro A12-8800B 184 \ ``amdgcn`` APU - xnack ROCm - A10-8700P 185 [on] - Pro A10-8700B 186 - A10-8780P 187 \ ``amdgcn`` APU - xnack - A10-9600P 188 [on] - A10-9630P 189 - A12-9700P 190 - A12-9730P 191 - FX-9800P 192 - FX-9830P 193 \ ``amdgcn`` APU - xnack - E2-9010 194 [on] - A6-9210 195 - A9-9410 196 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - xnack ROCm - Radeon R285 197 - ``tonga`` [off] - Radeon R9 380 198 - Radeon R9 385 199 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - xnack ROCm - Radeon R9 Nano 200 [off] - Radeon R9 Fury 201 - Radeon R9 FuryX 202 - Radeon Pro Duo 203 - FirePro S9300x2 204 - Radeon Instinct MI8 205 \ - ``polaris10`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 470 206 [off] - Radeon RX 480 207 - Radeon Instinct MI6 208 \ - ``polaris11`` ``amdgcn`` dGPU - xnack ROCm - Radeon RX 460 209 [off] 210 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - xnack ROCm - FirePro S7150 211 [off] - FirePro S7100 212 - FirePro W7100 213 - Mobile FirePro 214 M7170 215 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack *TBA* 216 [on] 217 .. TODO:: 218 219 Add product 220 names. 221 222 **GCN GFX9** [AMD-GCN-GFX9]_ 223 ----------------------------------------------------------------------------------------------------------- 224 ``gfx900`` ``amdgcn`` dGPU - xnack ROCm - Radeon Vega 225 [off] Frontier Edition 226 - Radeon RX Vega 56 227 - Radeon RX Vega 64 228 - Radeon RX Vega 64 229 Liquid 230 - Radeon Instinct MI25 231 ``gfx902`` ``amdgcn`` APU - xnack - Ryzen 3 2200G 232 [on] - Ryzen 5 2400G 233 ``gfx904`` ``amdgcn`` dGPU - xnack *TBA* 234 [off] 235 .. TODO:: 236 237 Add product 238 names. 239 240 ``gfx906`` ``amdgcn`` dGPU - xnack - Radeon Instinct MI50 241 [off] - Radeon Instinct MI60 242 - sram-ecc - Radeon VII 243 [off] - Radeon Pro VII 244 ``gfx908`` ``amdgcn`` dGPU - xnack *TBA* 245 [off] 246 - sram-ecc 247 [on] 248 .. TODO:: 249 250 Add product 251 names. 252 253 ``gfx909`` ``amdgcn`` APU - xnack *TBA* 254 [off] 255 .. TODO:: 256 257 Add product 258 names. 259 260 ``gfx90c`` ``amdgcn`` APU - xnack - Ryzen 7 4700G 261 [on] - Ryzen 7 4700GE 262 - Ryzen 7 4700G 263 - Ryzen 7 4700GE 264 - Ryzen 5 4600G 265 - Ryzen 5 4600GE 266 - Ryzen 3 4300G 267 - Ryzen 3 4300GE 268 - Ryzen Pro 4000G 269 - Ryzen 7 Pro 4700G 270 - Ryzen 7 Pro 4750GE 271 - Ryzen 5 Pro 4650G 272 - Ryzen 5 Pro 4650GE 273 - Ryzen 3 Pro 4350G 274 - Ryzen 3 Pro 4350GE 275 276 **GCN GFX10** [AMD-GCN-GFX10]_ 277 ----------------------------------------------------------------------------------------------------------- 278 ``gfx1010`` ``amdgcn`` dGPU - xnack - Radeon RX 5700 279 [off] - Radeon RX 5700 XT 280 - wavefrontsize64 - Radeon Pro 5600 XT 281 [off] - Radeon Pro 5600M 282 - cumode 283 [off] 284 ``gfx1011`` ``amdgcn`` dGPU - xnack *TBA* 285 [off] 286 - wavefrontsize64 287 [off] 288 - cumode 289 [off] 290 .. TODO:: 291 292 Add product 293 names. 294 295 ``gfx1012`` ``amdgcn`` dGPU - xnack - Radeon RX 5500 296 [off] - Radeon RX 5500 XT 297 - wavefrontsize64 298 [off] 299 - cumode 300 [off] 301 ``gfx1030`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 302 [off] 303 - cumode 304 [off] 305 .. TODO:: 306 307 Add product 308 names. 309 310 ``gfx1031`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 311 [off] 312 - cumode 313 [off] 314 .. TODO:: 315 316 Add product 317 names. 318 319 ``gfx1032`` ``amdgcn`` dGPU - wavefrontsize64 *TBA* 320 [off] 321 - cumode 322 [off] 323 .. TODO:: 324 Add product 325 names. 326 ``gfx1033`` ``amdgcn`` APU - wavefrontsize64 *TBA* 327 [off] 328 - cumode 329 [off] 330 .. TODO:: 331 Add product 332 names. 333 334 =========== =============== ============ ===== ============================= ======= ====================== 335 336.. _amdgpu-target-features: 337 338Target Features 339--------------- 340 341Target features control how code is generated to support certain 342processor specific features. Not all target features are supported by 343all processors. The runtime must ensure that the features supported by 344the device used to execute the code match the features enabled when 345generating the code. A mismatch of features may result in incorrect 346execution, or a reduction in performance. 347 348The target features supported by each processor, and the default value 349used if not specified explicitly, is listed in 350:ref:`amdgpu-processor-table`. 351 352Use the ``clang -m[no-]<TargetFeature>`` option to specify the AMDGPU 353target features. 354 355For example: 356 357``-mxnack`` 358 Enable the ``xnack`` feature. 359``-mno-xnack`` 360 Disable the ``xnack`` feature. 361 362 .. table:: AMDGPU Target Features 363 :name: amdgpu-target-feature-table 364 365 ====================== ================================================== 366 Target Feature Description 367 ====================== ================================================== 368 -m[no-]xnack Enable/disable generating code that has 369 memory clauses that are compatible with 370 having XNACK replay enabled. 371 372 This is used for demand paging and page 373 migration. If XNACK replay is enabled in 374 the device, then if a page fault occurs 375 the code may execute incorrectly if the 376 ``xnack`` feature is not enabled. Executing 377 code that has the feature enabled on a 378 device that does not have XNACK replay 379 enabled will execute correctly but may 380 be less performant than code with the 381 feature disabled. 382 383 -m[no-]sram-ecc Enable/disable generating code that assumes SRAM 384 ECC is enabled/disabled. 385 386 -m[no-]wavefrontsize64 Control the default wavefront size used when 387 generating code for kernels. When disabled 388 native wavefront size 32 is used, when enabled 389 wavefront size 64 is used. 390 391 -m[no-]cumode Control the default wavefront execution mode used 392 when generating code for kernels. When disabled 393 native WGP wavefront execution mode is used, 394 when enabled CU wavefront execution mode is used 395 (see :ref:`amdgpu-amdhsa-memory-model`). 396 ====================== ================================================== 397 398.. _amdgpu-address-spaces: 399 400Address Spaces 401-------------- 402 403The AMDGPU architecture supports a number of memory address spaces. The address 404space names use the OpenCL standard names, with some additions. 405 406The AMDGPU address spaces correspond to target architecture specific LLVM 407address space numbers used in LLVM IR. 408 409The AMDGPU address spaces are described in 410:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 411supported for the ``amdgcn`` target. 412 413 .. table:: AMDGPU Address Spaces 414 :name: amdgpu-address-spaces-table 415 416 ================================= =============== =========== ================ ======= ============================ 417 .. 64-Bit Process Address Space 418 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 419 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 420 Space Number Name Name Size 421 ================================= =============== =========== ================ ======= ============================ 422 Generic 0 flat flat 64 0x0000000000000000 423 Global 1 global global 64 0x0000000000000000 424 Region 2 N/A GDS 32 *not implemented for AMDHSA* 425 Local 3 group LDS 32 0xFFFFFFFF 426 Constant 4 constant *same as global* 64 0x0000000000000000 427 Private 5 private scratch 32 0xFFFFFFFF 428 Constant 32-bit 6 *TODO* 0x00000000 429 Buffer Fat Pointer (experimental) 7 *TODO* 430 ================================= =============== =========== ================ ======= ============================ 431 432**Generic** 433 The generic address space uses the hardware flat address support available in 434 GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 435 local apertures), that are outside the range of addressable global memory, to 436 map from a flat address to a private or local address. 437 438 FLAT instructions can take a flat address and access global, private 439 (scratch), and group (LDS) memory depending on if the address is within one 440 of the aperture ranges. Flat access to scratch requires hardware aperture 441 setup and setup in the kernel prologue (see 442 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 443 hardware aperture setup and M0 (GFX7-GFX8) register setup (see 444 :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 445 446 To convert between a private or group address space address (termed a segment 447 address) and a flat address the base address of the corresponding aperture 448 can be used. For GFX7-GFX8 these are available in the 449 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 450 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 451 GFX9-GFX10 the aperture base addresses are directly available as inline 452 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 453 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 454 aligned to 2^32 which makes it easier to convert from flat to segment or 455 segment to flat. 456 457 A global address space address has the same value when used as a flat address 458 so no conversion is needed. 459 460**Global and Constant** 461 The global and constant address spaces both use global virtual addresses, 462 which are the same virtual address space used by the CPU. However, some 463 virtual addresses may only be accessible to the CPU, some only accessible 464 by the GPU, and some by both. 465 466 Using the constant address space indicates that the data will not change 467 during the execution of the kernel. This allows scalar read instructions to 468 be used. As the constant address space could only be modified on the host 469 side, a generic pointer loaded from the constant address space is safe to be 470 assumed as a global pointer since only the device global memory is visible 471 and managed on the host side. The vector and scalar L1 caches are invalidated 472 of volatile data before each kernel dispatch execution to allow constant 473 memory to change values between kernel dispatches. 474 475**Region** 476 The region address space uses the hardware Global Data Store (GDS). All 477 wavefronts executing on the same device will access the same memory for any 478 given region address. However, the same region address accessed by wavefronts 479 executing on different devices will access different memory. It is higher 480 performance than global memory. It is allocated by the runtime. The data 481 store (DS) instructions can be used to access it. 482 483**Local** 484 The local address space uses the hardware Local Data Store (LDS) which is 485 automatically allocated when the hardware creates the wavefronts of a 486 work-group, and freed when all the wavefronts of a work-group have 487 terminated. All wavefronts belonging to the same work-group will access the 488 same memory for any given local address. However, the same local address 489 accessed by wavefronts belonging to different work-groups will access 490 different memory. It is higher performance than global memory. The data store 491 (DS) instructions can be used to access it. 492 493**Private** 494 The private address space uses the hardware scratch memory support which 495 automatically allocates memory when it creates a wavefront and frees it when 496 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 497 given private address will be different to the memory accessed by another lane 498 of the same or different wavefront for the same private address. 499 500 If a kernel dispatch uses scratch, then the hardware allocates memory from a 501 pool of backing memory allocated by the runtime for each wavefront. The lanes 502 of the wavefront access this using dword (4 byte) interleaving. The mapping 503 used from private address to backing memory address is: 504 505 ``wavefront-scratch-base + 506 ((private-address / 4) * wavefront-size * 4) + 507 (wavefront-lane-id * 4) + (private-address % 4)`` 508 509 If each lane of a wavefront accesses the same private address, the 510 interleaving results in adjacent dwords being accessed and hence requires 511 fewer cache lines to be fetched. 512 513 There are different ways that the wavefront scratch base address is 514 determined by a wavefront (see 515 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 516 517 Scratch memory can be accessed in an interleaved manner using buffer 518 instructions with the scratch buffer descriptor and per wavefront scratch 519 offset, by the scratch instructions, or by flat instructions. Multi-dword 520 access is not supported except by flat and scratch instructions in 521 GFX9-GFX10. 522 523**Constant 32-bit** 524 *TODO* 525 526**Buffer Fat Pointer** 527 The buffer fat pointer is an experimental address space that is currently 528 unsupported in the backend. It exposes a non-integral pointer that is in 529 the future intended to support the modelling of 128-bit buffer descriptors 530 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 531 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 532 model the buffer descriptors used heavily in graphics workloads targeting 533 the backend. 534 535.. _amdgpu-memory-scopes: 536 537Memory Scopes 538------------- 539 540This section provides LLVM memory synchronization scopes supported by the AMDGPU 541backend memory model when the target triple OS is ``amdhsa`` (see 542:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 543 544The memory model supported is based on the HSA memory model [HSA]_ which is 545based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 546relation is transitive over the synchronizes-with relation independent of scope 547and synchronizes-with allows the memory scope instances to be inclusive (see 548table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 549 550This is different to the OpenCL [OpenCL]_ memory model which does not have scope 551inclusion and requires the memory scopes to exactly match. However, this 552is conservatively correct for OpenCL. 553 554 .. table:: AMDHSA LLVM Sync Scopes 555 :name: amdgpu-amdhsa-llvm-sync-scopes-table 556 557 ======================= =================================================== 558 LLVM Sync Scope Description 559 ======================= =================================================== 560 *none* The default: ``system``. 561 562 Synchronizes with, and participates in modification 563 and seq_cst total orderings with, other operations 564 (except image operations) for all address spaces 565 (except private, or generic that accesses private) 566 provided the other operation's sync scope is: 567 568 - ``system``. 569 - ``agent`` and executed by a thread on the same 570 agent. 571 - ``workgroup`` and executed by a thread in the 572 same work-group. 573 - ``wavefront`` and executed by a thread in the 574 same wavefront. 575 576 ``agent`` Synchronizes with, and participates in modification 577 and seq_cst total orderings with, other operations 578 (except image operations) for all address spaces 579 (except private, or generic that accesses private) 580 provided the other operation's sync scope is: 581 582 - ``system`` or ``agent`` and executed by a thread 583 on the same agent. 584 - ``workgroup`` and executed by a thread in the 585 same work-group. 586 - ``wavefront`` and executed by a thread in the 587 same wavefront. 588 589 ``workgroup`` Synchronizes with, and participates in modification 590 and seq_cst total orderings with, other operations 591 (except image operations) for all address spaces 592 (except private, or generic that accesses private) 593 provided the other operation's sync scope is: 594 595 - ``system``, ``agent`` or ``workgroup`` and 596 executed by a thread in the same work-group. 597 - ``wavefront`` and executed by a thread in the 598 same wavefront. 599 600 ``wavefront`` Synchronizes with, and participates in modification 601 and seq_cst total orderings with, other operations 602 (except image operations) for all address spaces 603 (except private, or generic that accesses private) 604 provided the other operation's sync scope is: 605 606 - ``system``, ``agent``, ``workgroup`` or 607 ``wavefront`` and executed by a thread in the 608 same wavefront. 609 610 ``singlethread`` Only synchronizes with and participates in 611 modification and seq_cst total orderings with, 612 other operations (except image operations) running 613 in the same thread for all address spaces (for 614 example, in signal handlers). 615 616 ``one-as`` Same as ``system`` but only synchronizes with other 617 operations within the same address space. 618 619 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 620 operations within the same address space. 621 622 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 623 other operations within the same address space. 624 625 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 626 other operations within the same address space. 627 628 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 629 other operations within the same address space. 630 ======================= =================================================== 631 632LLVM IR Intrinsics 633------------------ 634 635The AMDGPU backend implements the following LLVM IR intrinsics. 636 637*This section is WIP.* 638 639.. TODO:: 640 641 List AMDGPU intrinsics. 642 643LLVM IR Attributes 644------------------ 645 646The AMDGPU backend supports the following LLVM IR attributes. 647 648 .. table:: AMDGPU LLVM IR Attributes 649 :name: amdgpu-llvm-ir-attributes-table 650 651 ======================================= ========================================================== 652 LLVM Attribute Description 653 ======================================= ========================================================== 654 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 655 will be specified when the kernel is dispatched. Generated 656 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 657 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 658 argument block size for the implicit arguments. This 659 varies by OS and language (for OpenCL see 660 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 661 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 662 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 663 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 664 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 665 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 666 execution unit. Generated by the ``amdgpu_waves_per_eu`` 667 CLANG attribute [CLANG-ATTR]_. 668 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 669 mode register to be set on entry. Overrides the default for 670 the calling convention. 671 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 672 the mode register to be set on entry. Overrides the default 673 for the calling convention. 674 ======================================= ========================================================== 675 676.. _amdgpu-elf-code-object: 677 678ELF Code Object 679=============== 680 681The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 682can be linked by ``lld`` to produce a standard ELF shared code object which can 683be loaded and executed on an AMDGPU target. 684 685.. _amdgpu-elf-header: 686 687Header 688------ 689 690The AMDGPU backend uses the following ELF header: 691 692 .. table:: AMDGPU ELF Header 693 :name: amdgpu-elf-header-table 694 695 ========================== =============================== 696 Field Value 697 ========================== =============================== 698 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 699 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 700 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 701 - ``ELFOSABI_AMDGPU_HSA`` 702 - ``ELFOSABI_AMDGPU_PAL`` 703 - ``ELFOSABI_AMDGPU_MESA3D`` 704 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA`` 705 - ``ELFABIVERSION_AMDGPU_PAL`` 706 - ``ELFABIVERSION_AMDGPU_MESA3D`` 707 ``e_type`` - ``ET_REL`` 708 - ``ET_DYN`` 709 ``e_machine`` ``EM_AMDGPU`` 710 ``e_entry`` 0 711 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-table` 712 ========================== =============================== 713 714.. 715 716 .. table:: AMDGPU ELF Header Enumeration Values 717 :name: amdgpu-elf-header-enumeration-values-table 718 719 =============================== ===== 720 Name Value 721 =============================== ===== 722 ``EM_AMDGPU`` 224 723 ``ELFOSABI_NONE`` 0 724 ``ELFOSABI_AMDGPU_HSA`` 64 725 ``ELFOSABI_AMDGPU_PAL`` 65 726 ``ELFOSABI_AMDGPU_MESA3D`` 66 727 ``ELFABIVERSION_AMDGPU_HSA`` 1 728 ``ELFABIVERSION_AMDGPU_PAL`` 0 729 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 730 =============================== ===== 731 732``e_ident[EI_CLASS]`` 733 The ELF class is: 734 735 * ``ELFCLASS32`` for ``r600`` architecture. 736 737 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 738 process address space applications. 739 740``e_ident[EI_DATA]`` 741 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 742 743``e_ident[EI_OSABI]`` 744 One of the following AMDGPU target architecture specific OS ABIs 745 (see :ref:`amdgpu-os-table`): 746 747 * ``ELFOSABI_NONE`` for *unknown* OS. 748 749 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 750 751 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 752 753 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 754 755``e_ident[EI_ABIVERSION]`` 756 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 757 object conforms: 758 759 * ``ELFABIVERSION_AMDGPU_HSA`` is used to specify the version of AMD HSA 760 runtime ABI. 761 762 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 763 runtime ABI. 764 765 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 766 3D runtime ABI. 767 768``e_type`` 769 Can be one of the following values: 770 771 772 ``ET_REL`` 773 The type produced by the AMDGPU backend compiler as it is relocatable code 774 object. 775 776 ``ET_DYN`` 777 The type produced by the linker as it is a shared code object. 778 779 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 780 781``e_machine`` 782 The value ``EM_AMDGPU`` is used for the machine for all processors supported 783 by the ``r600`` and ``amdgcn`` architectures (see 784 :ref:`amdgpu-processor-table`). The specific processor is specified in the 785 ``EF_AMDGPU_MACH`` bit field of the ``e_flags`` (see 786 :ref:`amdgpu-elf-header-e_flags-table`). 787 788``e_entry`` 789 The entry point is 0 as the entry points for individual kernels must be 790 selected in order to invoke them through AQL packets. 791 792``e_flags`` 793 The AMDGPU backend uses the following ELF header flags: 794 795 .. table:: AMDGPU ELF Header ``e_flags`` 796 :name: amdgpu-elf-header-e_flags-table 797 798 ================================= ========== ============================= 799 Name Value Description 800 ================================= ========== ============================= 801 **AMDGPU Processor Flag** See :ref:`amdgpu-processor-table`. 802 -------------------------------------------- ----------------------------- 803 ``EF_AMDGPU_MACH`` 0x000000ff AMDGPU processor selection 804 mask for 805 ``EF_AMDGPU_MACH_xxx`` values 806 defined in 807 :ref:`amdgpu-ef-amdgpu-mach-table`. 808 ``EF_AMDGPU_XNACK`` 0x00000100 Indicates if the ``xnack`` 809 target feature is 810 enabled for all code 811 contained in the code object. 812 If the processor 813 does not support the 814 ``xnack`` target 815 feature then must 816 be 0. 817 See 818 :ref:`amdgpu-target-features`. 819 ``EF_AMDGPU_SRAM_ECC`` 0x00000200 Indicates if the ``sram-ecc`` 820 target feature is 821 enabled for all code 822 contained in the code object. 823 If the processor 824 does not support the 825 ``sram-ecc`` target 826 feature then must 827 be 0. 828 See 829 :ref:`amdgpu-target-features`. 830 ================================= ========== ============================= 831 832 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 833 :name: amdgpu-ef-amdgpu-mach-table 834 835 ==================================== ========== ============================= 836 Name Value Description (see 837 :ref:`amdgpu-processor-table`) 838 ==================================== ========== ============================= 839 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 840 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 841 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 842 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 843 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 844 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 845 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 846 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 847 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 848 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 849 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 850 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 851 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 852 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 853 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 854 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 855 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 856 *reserved* 0x011 - Reserved for ``r600`` 857 0x01f architecture processors. 858 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 859 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 860 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 861 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 862 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 863 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 864 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 865 *reserved* 0x027 Reserved. 866 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 867 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 868 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 869 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 870 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 871 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 872 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 873 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 874 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 875 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 876 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 877 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 878 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 879 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 880 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 881 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 882 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 883 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 884 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 885 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 886 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 887 ==================================== ========== ============================= 888 889Sections 890-------- 891 892An AMDGPU target ELF code object has the standard ELF sections which include: 893 894 .. table:: AMDGPU ELF Sections 895 :name: amdgpu-elf-sections-table 896 897 ================== ================ ================================= 898 Name Type Attributes 899 ================== ================ ================================= 900 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 901 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 902 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 903 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 904 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 905 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 906 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 907 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 908 ``.note`` ``SHT_NOTE`` *none* 909 ``.rela``\ *name* ``SHT_RELA`` *none* 910 ``.rela.dyn`` ``SHT_RELA`` *none* 911 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 912 ``.shstrtab`` ``SHT_STRTAB`` *none* 913 ``.strtab`` ``SHT_STRTAB`` *none* 914 ``.symtab`` ``SHT_SYMTAB`` *none* 915 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 916 ================== ================ ================================= 917 918These sections have their standard meanings (see [ELF]_) and are only generated 919if needed. 920 921``.debug``\ *\** 922 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 923 information on the DWARF produced by the AMDGPU backend. 924 925``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 926 The standard sections used by a dynamic loader. 927 928``.note`` 929 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 930 backend. 931 932``.rela``\ *name*, ``.rela.dyn`` 933 For relocatable code objects, *name* is the name of the section that the 934 relocation records apply. For example, ``.rela.text`` is the section name for 935 relocation records associated with the ``.text`` section. 936 937 For linked shared code objects, ``.rela.dyn`` contains all the relocation 938 records from each of the relocatable code object's ``.rela``\ *name* sections. 939 940 See :ref:`amdgpu-relocation-records` for the relocation records supported by 941 the AMDGPU backend. 942 943``.text`` 944 The executable machine code for the kernels and functions they call. Generated 945 as position independent code. See :ref:`amdgpu-code-conventions` for 946 information on conventions used in the isa generation. 947 948.. _amdgpu-note-records: 949 950Note Records 951------------ 952 953The AMDGPU backend code object contains ELF note records in the ``.note`` 954section. The set of generated notes and their semantics depend on the code 955object version; see :ref:`amdgpu-note-records-v2` and 956:ref:`amdgpu-note-records-v3`. 957 958As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 959must be generated after the ``name`` field to ensure the ``desc`` field is 4 960byte aligned. In addition, minimal zero-byte padding must be generated to 961ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 962field of the ``.note`` section must be at least 4 to indicate at least 8 byte 963alignment. 964 965.. _amdgpu-note-records-v2: 966 967Code Object V2 Note Records (--amdhsa-code-object-version=2) 968~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 969 970.. warning:: Code Object V2 is not the default code object version emitted by 971 this version of LLVM. For a description of the notes generated with the 972 default configuration (Code Object V3) see :ref:`amdgpu-note-records-v3`. 973 974The AMDGPU backend code object uses the following ELF note record in the 975``.note`` section when compiling for Code Object V2 976(--amdhsa-code-object-version=2). 977 978Additional note records may be present, but any which are not documented here 979are deprecated and should not be used. 980 981 .. table:: AMDGPU Code Object V2 ELF Note Records 982 :name: amdgpu-elf-note-records-table-v2 983 984 ===== ============================== ====================================== 985 Name Type Description 986 ===== ============================== ====================================== 987 "AMD" ``NT_AMD_AMDGPU_HSA_METADATA`` <metadata null terminated string> 988 ===== ============================== ====================================== 989 990.. 991 992 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 993 :name: amdgpu-elf-note-record-enumeration-values-table-v2 994 995 ============================== ===== 996 Name Value 997 ============================== ===== 998 *reserved* 0-9 999 ``NT_AMD_AMDGPU_HSA_METADATA`` 10 1000 *reserved* 11 1001 ============================== ===== 1002 1003``NT_AMD_AMDGPU_HSA_METADATA`` 1004 Specifies extensible metadata associated with the code objects executed on HSA 1005 [HSA]_ compatible runtimes such as AMD's ROCm [AMD-ROCm]_. It is required when 1006 the target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1007 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code 1008 object metadata string. 1009 1010.. _amdgpu-note-records-v3: 1011 1012Code Object V3 Note Records (--amdhsa-code-object-version=3) 1013~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1014 1015The AMDGPU backend code object uses the following ELF note record in the 1016``.note`` section when compiling for Code Object V3 1017(--amdhsa-code-object-version=3). 1018 1019Additional note records may be present, but any which are not documented here 1020are deprecated and should not be used. 1021 1022 .. table:: AMDGPU Code Object V3 ELF Note Records 1023 :name: amdgpu-elf-note-records-table-v3 1024 1025 ======== ============================== ====================================== 1026 Name Type Description 1027 ======== ============================== ====================================== 1028 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1029 binary format. 1030 ======== ============================== ====================================== 1031 1032.. 1033 1034 .. table:: AMDGPU Code Object V3 ELF Note Record Enumeration Values 1035 :name: amdgpu-elf-note-record-enumeration-values-table-v3 1036 1037 ============================== ===== 1038 Name Value 1039 ============================== ===== 1040 *reserved* 0-31 1041 ``NT_AMDGPU_METADATA`` 32 1042 ============================== ===== 1043 1044``NT_AMDGPU_METADATA`` 1045 Specifies extensible metadata associated with an AMDGPU code 1046 object. It is encoded as a map in the Message Pack [MsgPack]_ binary 1047 data format. See :ref:`amdgpu-amdhsa-code-object-metadata-v3` for the 1048 map keys defined for the ``amdhsa`` OS. 1049 1050.. _amdgpu-symbols: 1051 1052Symbols 1053------- 1054 1055Symbols include the following: 1056 1057 .. table:: AMDGPU ELF Symbols 1058 :name: amdgpu-elf-symbols-table 1059 1060 ===================== ================== ================ ================== 1061 Name Type Section Description 1062 ===================== ================== ================ ================== 1063 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1064 - ``.rodata`` 1065 - ``.bss`` 1066 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1067 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1068 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1069 ===================== ================== ================ ================== 1070 1071Global variable 1072 Global variables both used and defined by the compilation unit. 1073 1074 If the symbol is defined in the compilation unit then it is allocated in the 1075 appropriate section according to if it has initialized data or is readonly. 1076 1077 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1078 will resolve relocations using the definition provided by another code object 1079 or explicitly defined by the runtime. 1080 1081 If the symbol resides in local/group memory (LDS) then its section is the 1082 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1083 ``st_value`` field describes alignment requirements as it does for common 1084 symbols. 1085 1086 .. TODO:: 1087 1088 Add description of linked shared object symbols. Seems undefined symbols 1089 are marked as STT_NOTYPE. 1090 1091Kernel descriptor 1092 Every HSA kernel has an associated kernel descriptor. It is the address of the 1093 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1094 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1095 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1096 1097Kernel entry point 1098 Every HSA kernel also has a symbol for its machine code entry point. 1099 1100.. _amdgpu-relocation-records: 1101 1102Relocation Records 1103------------------ 1104 1105AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1106relocatable fields are: 1107 1108``word32`` 1109 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1110 alignment. These values use the same byte order as other word values in the 1111 AMDGPU architecture. 1112 1113``word64`` 1114 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1115 alignment. These values use the same byte order as other word values in the 1116 AMDGPU architecture. 1117 1118Following notations are used for specifying relocation calculations: 1119 1120**A** 1121 Represents the addend used to compute the value of the relocatable field. 1122 1123**G** 1124 Represents the offset into the global offset table at which the relocation 1125 entry's symbol will reside during execution. 1126 1127**GOT** 1128 Represents the address of the global offset table. 1129 1130**P** 1131 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1132 of the storage unit being relocated (computed using ``r_offset``). 1133 1134**S** 1135 Represents the value of the symbol whose index resides in the relocation 1136 entry. Relocations not using this must specify a symbol index of 1137 ``STN_UNDEF``. 1138 1139**B** 1140 Represents the base address of a loaded executable or shared object which is 1141 the difference between the ELF address and the actual load address. 1142 Relocations using this are only valid in executable or shared objects. 1143 1144The following relocation types are supported: 1145 1146 .. table:: AMDGPU ELF Relocation Records 1147 :name: amdgpu-elf-relocation-records-table 1148 1149 ========================== ======= ===== ========== ============================== 1150 Relocation Type Kind Value Field Calculation 1151 ========================== ======= ===== ========== ============================== 1152 ``R_AMDGPU_NONE`` 0 *none* *none* 1153 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1154 Dynamic 1155 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1156 Dynamic 1157 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1158 Dynamic 1159 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1160 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1161 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1162 Dynamic 1163 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1164 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1165 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1166 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1167 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1168 *reserved* 12 1169 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1170 ========================== ======= ===== ========== ============================== 1171 1172``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1173the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1174 1175There is no current OS loader support for 32-bit programs and so 1176``R_AMDGPU_ABS32`` is not used. 1177 1178.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1179 1180Loaded Code Object Path Uniform Resource Identifier (URI) 1181--------------------------------------------------------- 1182 1183The AMD GPU code object loader represents the path of the ELF shared object from 1184which the code object was loaded as a textual Unifom Resource Identifier (URI). 1185Note that the code object is the in memory loaded relocated form of the ELF 1186shared object. Multiple code objects may be loaded at different memory 1187addresses in the same process from the same ELF shared object. 1188 1189The loaded code object path URI syntax is defined by the following BNF syntax: 1190 1191.. code:: 1192 1193 code_object_uri ::== file_uri | memory_uri 1194 file_uri ::== "file://" file_path [ range_specifier ] 1195 memory_uri ::== "memory://" process_id range_specifier 1196 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1197 file_path ::== URI_ENCODED_OS_FILE_PATH 1198 process_id ::== DECIMAL_NUMBER 1199 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1200 1201**number** 1202 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1203 and octal values by "0". 1204 1205**file_path** 1206 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1207 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1208 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1209 the path are separated by "/". 1210 1211**offset** 1212 Is a 0-based byte offset to the start of the code object. For a file URI, it 1213 is from the start of the file specified by the ``file_path``, and if omitted 1214 defaults to 0. For a memory URI, it is the memory address and is required. 1215 1216**size** 1217 Is the number of bytes in the code object. For a file URI, if omitted it 1218 defaults to the size of the file. It is required for a memory URI. 1219 1220**process_id** 1221 Is the identity of the process owning the memory. For Linux it is the C 1222 unsigned integral decimal literal for the process ID (PID). 1223 1224For example: 1225 1226.. code:: 1227 1228 file:///dir1/dir2/file1 1229 file:///dir3/dir4/file2#offset=0x2000&size=3000 1230 memory://1234#offset=0x20000&size=3000 1231 1232.. _amdgpu-dwarf-debug-information: 1233 1234DWARF Debug Information 1235======================= 1236 1237.. warning:: 1238 1239 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1240 is not currently fully implemented and is subject to change. 1241 1242AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1243:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1244object executable code and data to the source language constructs. It can be 1245used by tools such as debuggers and profilers. It uses features defined in 1246:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1247DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1248 1249This section defines the AMDGPU target architecture specific DWARF mappings. 1250 1251.. _amdgpu-dwarf-register-identifier: 1252 1253Register Identifier 1254------------------- 1255 1256This section defines the AMDGPU target architecture register numbers used in 1257DWARF operation expressions (see DWARF Version 5 section 2.5 and 1258:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1259instructions (see DWARF Version 5 section 6.4 and 1260:ref:`amdgpu-dwarf-call-frame-information`). 1261 1262A single code object can contain code for kernels that have different wavefront 1263sizes. The vector registers and some scalar registers are based on the wavefront 1264size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1265simplifies the consumer of the DWARF so that each register has a fixed size, 1266rather than being dynamic according to the wavefront size mode. Similarly, 1267distinct DWARF registers are defined for those registers that vary in size 1268according to the process address size. This allows a consumer to treat a 1269specific AMDGPU processor as a single architecture regardless of how it is 1270configured at run time. The compiler explicitly specifies the DWARF registers 1271that match the mode in which the code it is generating will be executed. 1272 1273DWARF registers are encoded as numbers, which are mapped to architecture 1274registers. The mapping for AMDGPU is defined in 1275:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1276mapping. 1277 1278.. table:: AMDGPU DWARF Register Mapping 1279 :name: amdgpu-dwarf-register-mapping-table 1280 1281 ============== ================= ======== ================================== 1282 DWARF Register AMDGPU Register Bit Size Description 1283 ============== ================= ======== ================================== 1284 0 PC_32 32 Program Counter (PC) when 1285 executing in a 32-bit process 1286 address space. Used in the CFI to 1287 describe the PC of the calling 1288 frame. 1289 1 EXEC_MASK_32 32 Execution Mask Register when 1290 executing in wavefront 32 mode. 1291 2-15 *Reserved* *Reserved for highly accessed 1292 registers using DWARF shortcut.* 1293 16 PC_64 64 Program Counter (PC) when 1294 executing in a 64-bit process 1295 address space. Used in the CFI to 1296 describe the PC of the calling 1297 frame. 1298 17 EXEC_MASK_64 64 Execution Mask Register when 1299 executing in wavefront 64 mode. 1300 18-31 *Reserved* *Reserved for highly accessed 1301 registers using DWARF shortcut.* 1302 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1303 Registers. 1304 96-127 *Reserved* *Reserved for frequently accessed 1305 registers using DWARF 1-byte ULEB.* 1306 128 STATUS 32 Status Register. 1307 129-511 *Reserved* *Reserved for future Scalar 1308 Architectural Registers.* 1309 512 VCC_32 32 Vector Condition Code Register 1310 when executing in wavefront 32 1311 mode. 1312 513-1023 *Reserved* *Reserved for future Vector 1313 Architectural Registers when 1314 executing in wavefront 32 mode.* 1315 768 VCC_64 64 Vector Condition Code Register 1316 when executing in wavefront 64 1317 mode. 1318 769-1023 *Reserved* *Reserved for future Vector 1319 Architectural Registers when 1320 executing in wavefront 64 mode.* 1321 1024-1087 *Reserved* *Reserved for padding.* 1322 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1323 1130-1535 *Reserved* *Reserved for future Scalar 1324 General Purpose Registers.* 1325 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1326 when executing in wavefront 32 1327 mode. 1328 1792-2047 *Reserved* *Reserved for future Vector 1329 General Purpose Registers when 1330 executing in wavefront 32 mode.* 1331 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1332 when executing in wavefront 32 1333 mode. 1334 2304-2559 *Reserved* *Reserved for future Vector 1335 Accumulation Registers when 1336 executing in wavefront 32 mode.* 1337 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1338 when executing in wavefront 64 1339 mode. 1340 2816-3071 *Reserved* *Reserved for future Vector 1341 General Purpose Registers when 1342 executing in wavefront 64 mode.* 1343 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1344 when executing in wavefront 64 1345 mode. 1346 3328-3583 *Reserved* *Reserved for future Vector 1347 Accumulation Registers when 1348 executing in wavefront 64 mode.* 1349 ============== ================= ======== ================================== 1350 1351The vector registers are represented as the full size for the wavefront. They 1352are organized as consecutive dwords (32-bits), one per lane, with the dword at 1353the least significant bit position corresponding to lane 0 and so forth. DWARF 1354location expressions involving the ``DW_OP_LLVM_offset`` and 1355``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1356register corresponding to the lane that is executing the current thread of 1357execution in languages that are implemented using a SIMD or SIMT execution 1358model. 1359 1360If the wavefront size is 32 lanes then the wavefront 32 mode register 1361definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1362mode register definitions are used. Some AMDGPU targets support executing in 1363both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1364to the wavefront mode of the generated code will be used. 1365 1366If code is generated to execute in a 32-bit process address space, then the 136732-bit process address space register definitions are used. If code is generated 1368to execute in a 64-bit process address space, then the 64-bit process address 1369space register definitions are used. The ``amdgcn`` target only supports the 137064-bit process address space. 1371 1372.. _amdgpu-dwarf-address-class-identifier: 1373 1374Address Class Identifier 1375------------------------ 1376 1377The DWARF address class represents the source language memory space. See DWARF 1378Version 5 section 2.12 which is updated by the *DWARF Extensions For 1379Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1380 1381The DWARF address class mapping used for AMDGPU is defined in 1382:ref:`amdgpu-dwarf-address-class-mapping-table`. 1383 1384.. table:: AMDGPU DWARF Address Class Mapping 1385 :name: amdgpu-dwarf-address-class-mapping-table 1386 1387 ========================= ====== ================= 1388 DWARF AMDGPU 1389 -------------------------------- ----------------- 1390 Address Class Name Value Address Space 1391 ========================= ====== ================= 1392 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1393 ``DW_ADDR_LLVM_global`` 0x0001 Global 1394 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1395 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1396 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1397 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1398 ========================= ====== ================= 1399 1400The DWARF address class values defined in the *DWARF Extensions For 1401Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1402 1403In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1404available for use for the AMD extension for access to the hardware GDS memory 1405which is scratchpad memory allocated per device. 1406 1407For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1408address class of ``DW_ADDR_none`` is used. 1409 1410See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1411mapping of DWARF address classes to DWARF address spaces, including address size 1412and NULL value. 1413 1414.. _amdgpu-dwarf-address-space-identifier: 1415 1416Address Space Identifier 1417------------------------ 1418 1419DWARF address spaces correspond to target architecture specific linear 1420addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1421For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1422 1423The DWARF address space mapping used for AMDGPU is defined in 1424:ref:`amdgpu-dwarf-address-space-mapping-table`. 1425 1426.. table:: AMDGPU DWARF Address Space Mapping 1427 :name: amdgpu-dwarf-address-space-mapping-table 1428 1429 ======================================= ===== ======= ======== ================= ======================= 1430 DWARF AMDGPU Notes 1431 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1432 Address Space Name Value Address Bit Size Address Space 1433 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1434 .. 64-bit 32-bit 1435 process process 1436 address address 1437 space space 1438 ======================================= ===== ======= ======== ================= ======================= 1439 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1440 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1441 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1442 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1443 *Reserved* 0x04 1444 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1445 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1446 ======================================= ===== ======= ======== ================= ======================= 1447 1448See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1449including address size and NULL value. 1450 1451The ``DW_ASPACE_none`` address space is the default target architecture address 1452space used in DWARF operations that do not specify an address space. It 1453therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1454related operations can refer to addresses in the program code. 1455 1456The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1457specify the flat address space. If the address corresponds to an address in the 1458local address space, then it corresponds to the wavefront that is executing the 1459focused thread of execution. If the address corresponds to an address in the 1460private address space, then it corresponds to the lane that is executing the 1461focused thread of execution for languages that are implemented using a SIMD or 1462SIMT execution model. 1463 1464.. note:: 1465 1466 CUDA-like languages such as HIP that do not have address spaces in the 1467 language type system, but do allow variables to be allocated in different 1468 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1469 address space in the DWARF expression operations as the default address space 1470 is the global address space. 1471 1472The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 1473specify the local address space corresponding to the wavefront that is executing 1474the focused thread of execution. 1475 1476The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 1477to specify the private address space corresponding to the lane that is executing 1478the focused thread of execution for languages that are implemented using a SIMD 1479or SIMT execution model. 1480 1481The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 1482to specify the unswizzled private address space corresponding to the wavefront 1483that is executing the focused thread of execution. The wavefront view of private 1484memory is the per wavefront unswizzled backing memory layout defined in 1485:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 1486location for the backing memory of the wavefront (namely the address is not 1487offset by ``wavefront-scratch-base``). The following formula can be used to 1488convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 1489``DW_ASPACE_AMDGPU_private_wave`` address: 1490 1491:: 1492 1493 private-address-wavefront = 1494 ((private-address-lane / 4) * wavefront-size * 4) + 1495 (wavefront-lane-id * 4) + (private-address-lane % 4) 1496 1497If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 1498of the dwords for each lane starting with lane 0 is required, then this 1499simplifies to: 1500 1501:: 1502 1503 private-address-wavefront = 1504 private-address-lane * wavefront-size 1505 1506A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 1507complete spilled vector register back into a complete vector register in the 1508CFI. The frame pointer can be a private lane address which is dword aligned, 1509which can be shifted to multiply by the wavefront size, and then used to form a 1510private wavefront address that gives a location for a contiguous set of dwords, 1511one per lane, where the vector register dwords are spilled. The compiler knows 1512the wavefront size since it generates the code. Note that the type of the 1513address may have to be converted as the size of a 1514``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 1515``DW_ASPACE_AMDGPU_private_wave`` address. 1516 1517.. _amdgpu-dwarf-lane-identifier: 1518 1519Lane identifier 1520--------------- 1521 1522DWARF lane identifies specify a target architecture lane position for hardware 1523that executes in a SIMD or SIMT manner, and on which a source language maps its 1524threads of execution onto those lanes. The DWARF lane identifier is pushed by 1525the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 1526section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 1527section :ref:`amdgpu-dwarf-operation-expressions`. 1528 1529For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 1530wavefront. It is numbered from 0 to the wavefront size minus 1. 1531 1532Operation Expressions 1533--------------------- 1534 1535DWARF expressions are used to compute program values and the locations of 1536program objects. See DWARF Version 5 section 2.5 and 1537:ref:`amdgpu-dwarf-operation-expressions`. 1538 1539DWARF location descriptions describe how to access storage which includes memory 1540and registers. When accessing storage on AMDGPU, bytes are ordered with least 1541significant bytes first, and bits are ordered within bytes with least 1542significant bits first. 1543 1544For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 1545unwinding vector registers that are spilled under the execution mask to memory: 1546the zero-single location description is the vector register, and the one-single 1547location description is the spilled memory location description. The 1548``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 1549memory location description. 1550 1551In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 1552``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 1553controlled by the execution mask. An undefined location description together 1554with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 1555to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 1556 1557Debugger Information Entry Attributes 1558------------------------------------- 1559 1560This section describes how certain debugger information entry attributes are 1561used by AMDGPU. See the sections in DWARF Version 5 section 2 which are updated 1562by *DWARF Extensions For Heterogeneous Debugging* section 1563:ref:`amdgpu-dwarf-debugging-information-entry-attributes`. 1564 1565.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 1566 1567``DW_AT_LLVM_lane_pc`` 1568~~~~~~~~~~~~~~~~~~~~~~ 1569 1570For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 1571location of the separate lanes of a SIMT thread. 1572 1573If the lane is an active lane then this will be the same as the current program 1574location. 1575 1576If the lane is inactive, but was active on entry to the subprogram, then this is 1577the program location in the subprogram at which execution of the lane is 1578conceptual positioned. 1579 1580If the lane was not active on entry to the subprogram, then this will be the 1581undefined location. A client debugger can check if the lane is part of a valid 1582work-group by checking that the lane is in the range of the associated 1583work-group within the grid, accounting for partial work-groups. If it is not, 1584then the debugger can omit any information for the lane. Otherwise, the debugger 1585may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 1586calling subprogram until it finds a non-undefined location. Conceptually the 1587lane only has the call frames that it has a non-undefined 1588``DW_AT_LLVM_lane_pc``. 1589 1590The following example illustrates how the AMDGPU backend can generate a DWARF 1591location list expression for the nested ``IF/THEN/ELSE`` structures of the 1592following subprogram pseudo code for a target with 64 lanes per wavefront. 1593 1594.. code:: 1595 :number-lines: 1596 1597 SUBPROGRAM X 1598 BEGIN 1599 a; 1600 IF (c1) THEN 1601 b; 1602 IF (c2) THEN 1603 c; 1604 ELSE 1605 d; 1606 ENDIF 1607 e; 1608 ELSE 1609 f; 1610 ENDIF 1611 g; 1612 END 1613 1614The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 1615execution mask (``EXEC``) to linearize the control flow. The condition is 1616evaluated to make a mask of the lanes for which the condition evaluates to true. 1617First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 1618logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 1619``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 1620the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 1621region the ``EXEC`` mask is restored to the value it had at the beginning of the 1622region. This is shown below. Other approaches are possible, but the basic 1623concept is the same. 1624 1625.. code:: 1626 :number-lines: 1627 1628 $lex_start: 1629 a; 1630 %1 = EXEC 1631 %2 = c1 1632 $lex_1_start: 1633 EXEC = %1 & %2 1634 $if_1_then: 1635 b; 1636 %3 = EXEC 1637 %4 = c2 1638 $lex_1_1_start: 1639 EXEC = %3 & %4 1640 $lex_1_1_then: 1641 c; 1642 EXEC = ~EXEC & %3 1643 $lex_1_1_else: 1644 d; 1645 EXEC = %3 1646 $lex_1_1_end: 1647 e; 1648 EXEC = ~EXEC & %1 1649 $lex_1_else: 1650 f; 1651 EXEC = %1 1652 $lex_1_end: 1653 g; 1654 $lex_end: 1655 1656To create the DWARF location list expression that defines the location 1657description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 1658pseudo instruction can be used to annotate the linearized control flow. This can 1659be done by defining an artificial variable for the lane PC. The DWARF location 1660list expression created for it is used as the value of the 1661``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 1662 1663A DWARF procedure is defined for each well nested structured control flow region 1664which provides the conceptual lane program location for a lane if it is not 1665active (namely it is divergent). The DWARF operation expression for each region 1666conceptually inherits the value of the immediately enclosing region and modifies 1667it according to the semantics of the region. 1668 1669For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 1670the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 1671region the divergent program location is at the end of the ``IF/THEN/ELSE`` 1672region since the ``THEN`` region has completed. 1673 1674The lane PC artificial variable is assigned at each region transition. It uses 1675the immediately enclosing region's DWARF procedure to compute the program 1676location for each lane assuming they are divergent, and then modifies the result 1677by inserting the current program location for each lane that the ``EXEC`` mask 1678indicates is active. 1679 1680By having separate DWARF procedures for each region, they can be reused to 1681define the value for any nested region. This reduces the total size of the DWARF 1682operation expressions. 1683 1684The following provides an example using pseudo LLVM MIR. 1685 1686.. code:: 1687 :number-lines: 1688 1689 $lex_start: 1690 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 1691 DW_AT_name = "__uint64"; 1692 DW_AT_byte_size = 8; 1693 DW_AT_encoding = DW_ATE_unsigned; 1694 ]; 1695 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 1696 DW_AT_name = "__active_lane_pc"; 1697 DW_AT_location = [ 1698 DW_OP_regx PC; 1699 DW_OP_LLVM_extend 64, 64; 1700 DW_OP_regval_type EXEC, %uint_64; 1701 DW_OP_LLVM_select_bit_piece 64, 64; 1702 ]; 1703 ]; 1704 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 1705 DW_AT_name = "__divergent_lane_pc"; 1706 DW_AT_location = [ 1707 DW_OP_LLVM_undefined; 1708 DW_OP_LLVM_extend 64, 64; 1709 ]; 1710 ]; 1711 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1712 DW_OP_call_ref %__divergent_lane_pc; 1713 DW_OP_call_ref %__active_lane_pc; 1714 ]; 1715 a; 1716 %1 = EXEC; 1717 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 1718 %2 = c1; 1719 $lex_1_start: 1720 EXEC = %1 & %2; 1721 $lex_1_then: 1722 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 1723 DW_AT_name = "__divergent_lane_pc_1_then"; 1724 DW_AT_location = DIExpression[ 1725 DW_OP_call_ref %__divergent_lane_pc; 1726 DW_OP_addrx &lex_1_start; 1727 DW_OP_stack_value; 1728 DW_OP_LLVM_extend 64, 64; 1729 DW_OP_call_ref %__lex_1_save_exec; 1730 DW_OP_deref_type 64, %__uint_64; 1731 DW_OP_LLVM_select_bit_piece 64, 64; 1732 ]; 1733 ]; 1734 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1735 DW_OP_call_ref %__divergent_lane_pc_1_then; 1736 DW_OP_call_ref %__active_lane_pc; 1737 ]; 1738 b; 1739 %3 = EXEC; 1740 DBG_VALUE %3, %__lex_1_1_save_exec; 1741 %4 = c2; 1742 $lex_1_1_start: 1743 EXEC = %3 & %4; 1744 $lex_1_1_then: 1745 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 1746 DW_AT_name = "__divergent_lane_pc_1_1_then"; 1747 DW_AT_location = DIExpression[ 1748 DW_OP_call_ref %__divergent_lane_pc_1_then; 1749 DW_OP_addrx &lex_1_1_start; 1750 DW_OP_stack_value; 1751 DW_OP_LLVM_extend 64, 64; 1752 DW_OP_call_ref %__lex_1_1_save_exec; 1753 DW_OP_deref_type 64, %__uint_64; 1754 DW_OP_LLVM_select_bit_piece 64, 64; 1755 ]; 1756 ]; 1757 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1758 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 1759 DW_OP_call_ref %__active_lane_pc; 1760 ]; 1761 c; 1762 EXEC = ~EXEC & %3; 1763 $lex_1_1_else: 1764 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 1765 DW_AT_name = "__divergent_lane_pc_1_1_else"; 1766 DW_AT_location = DIExpression[ 1767 DW_OP_call_ref %__divergent_lane_pc_1_then; 1768 DW_OP_addrx &lex_1_1_end; 1769 DW_OP_stack_value; 1770 DW_OP_LLVM_extend 64, 64; 1771 DW_OP_call_ref %__lex_1_1_save_exec; 1772 DW_OP_deref_type 64, %__uint_64; 1773 DW_OP_LLVM_select_bit_piece 64, 64; 1774 ]; 1775 ]; 1776 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1777 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 1778 DW_OP_call_ref %__active_lane_pc; 1779 ]; 1780 d; 1781 EXEC = %3; 1782 $lex_1_1_end: 1783 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1784 DW_OP_call_ref %__divergent_lane_pc; 1785 DW_OP_call_ref %__active_lane_pc; 1786 ]; 1787 e; 1788 EXEC = ~EXEC & %1; 1789 $lex_1_else: 1790 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 1791 DW_AT_name = "__divergent_lane_pc_1_else"; 1792 DW_AT_location = DIExpression[ 1793 DW_OP_call_ref %__divergent_lane_pc; 1794 DW_OP_addrx &lex_1_end; 1795 DW_OP_stack_value; 1796 DW_OP_LLVM_extend 64, 64; 1797 DW_OP_call_ref %__lex_1_save_exec; 1798 DW_OP_deref_type 64, %__uint_64; 1799 DW_OP_LLVM_select_bit_piece 64, 64; 1800 ]; 1801 ]; 1802 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 1803 DW_OP_call_ref %__divergent_lane_pc_1_else; 1804 DW_OP_call_ref %__active_lane_pc; 1805 ]; 1806 f; 1807 EXEC = %1; 1808 $lex_1_end: 1809 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 1810 DW_OP_call_ref %__divergent_lane_pc; 1811 DW_OP_call_ref %__active_lane_pc; 1812 ]; 1813 g; 1814 $lex_end: 1815 1816The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 1817that are active, with the current program location. 1818 1819Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 1820the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 1821instruction, location list entries will be created that describe where the 1822artificial variables are allocated at any given program location. The compiler 1823may allocate them to registers or spill them to memory. 1824 1825The DWARF procedures for each region use the values of the saved execution mask 1826artificial variables to only update the lanes that are active on entry to the 1827region. All other lanes retain the value of the enclosing region where they were 1828last active. If they were not active on entry to the subprogram, then will have 1829the undefined location description. 1830 1831Other structured control flow regions can be handled similarly. For example, 1832loops would set the divergent program location for the region at the end of the 1833loop. Any lanes active will be in the loop, and any lanes not active must have 1834exited the loop. 1835 1836An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 1837``IF/THEN/ELSE`` regions. 1838 1839The DWARF procedures can use the active lane artificial variable described in 1840:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 1841``EXEC`` mask in order to support whole or quad wavefront mode. 1842 1843.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 1844 1845``DW_AT_LLVM_active_lane`` 1846~~~~~~~~~~~~~~~~~~~~~~~~~~ 1847 1848The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 1849entry is used to specify the lanes that are conceptually active for a SIMT 1850thread. 1851 1852The execution mask may be modified to implement whole or quad wavefront mode 1853operations. For example, all lanes may need to temporarily be made active to 1854execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 1855update it to enable the necessary lanes, perform the operations, and then 1856restore the ``EXEC`` mask from the saved value. While executing the whole 1857wavefront region, the conceptual execution mask is the saved value, not the 1858``EXEC`` value. 1859 1860This is handled by defining an artificial variable for the active lane mask. The 1861active lane mask artificial variable would be the actual ``EXEC`` mask for 1862normal regions, and the saved execution mask for regions where the mask is 1863temporarily updated. The location list expression created for this artificial 1864variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 1865attribute. 1866 1867``DW_AT_LLVM_augmentation`` 1868~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1869 1870For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 1871debugger information entry has the following value for the augmentation string: 1872 1873:: 1874 1875 [amdgpu:v0.0] 1876 1877The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1878extensions used in the DWARF of the compilation unit. The version number 1879conforms to [SEMVER]_. 1880 1881Call Frame Information 1882---------------------- 1883 1884DWARF Call Frame Information (CFI) describes how a consumer can virtually 1885*unwind* call frames in a running process or core dump. See DWARF Version 5 1886section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 1887 1888For AMDGPU, the Common Information Entry (CIE) fields have the following values: 1889 18901. ``augmentation`` string contains the following null-terminated UTF-8 string: 1891 1892 :: 1893 1894 [amd:v0.0] 1895 1896 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 1897 extensions used in this CIE or to the FDEs that use it. The version number 1898 conforms to [SEMVER]_. 1899 19002. ``address_size`` for the ``Global`` address space is defined in 1901 :ref:`amdgpu-dwarf-address-space-identifier`. 1902 19033. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 1904 19054. ``code_alignment_factor`` is 4 bytes. 1906 1907 .. TODO:: 1908 1909 Add to :ref:`amdgpu-processor-table` table. 1910 19115. ``data_alignment_factor`` is 4 bytes. 1912 1913 .. TODO:: 1914 1915 Add to :ref:`amdgpu-processor-table` table. 1916 19176. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 1918 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 1919 19207. ``initial_instructions`` Since a subprogram X with fewer registers can be 1921 called from subprogram Y that has more allocated, X will not change any of 1922 the extra registers as it cannot access them. Therefore, the default rule 1923 for all columns is ``same value``. 1924 1925For AMDGPU the register number follows the numbering defined in 1926:ref:`amdgpu-dwarf-register-identifier`. 1927 1928For AMDGPU the instructions are variable size. A consumer can subtract 1 from 1929the return address to get the address of a byte within the call site 1930instructions. See DWARF Version 5 section 6.4.4. 1931 1932Accelerated Access 1933------------------ 1934 1935See DWARF Version 5 section 6.1. 1936 1937Lookup By Name Section Header 1938~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1939 1940See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 1941 1942For AMDGPU the lookup by name section header table: 1943 1944``augmentation_string_size`` (uword) 1945 1946 Set to the length of the ``augmentation_string`` value which is always a 1947 multiple of 4. 1948 1949``augmentation_string`` (sequence of UTF-8 characters) 1950 1951 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 1952 1953 :: 1954 1955 [amdgpu:v0.0] 1956 1957 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 1958 extensions used in the DWARF of this index. The version number conforms to 1959 [SEMVER]_. 1960 1961 .. note:: 1962 1963 This is different to the DWARF Version 5 definition that requires the first 1964 4 characters to be the vendor ID. But this is consistent with the other 1965 augmentation strings and does allow multiple vendor contributions. However, 1966 backwards compatibility may be more desirable. 1967 1968Lookup By Address Section Header 1969~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1970 1971See DWARF Version 5 section 6.1.2. 1972 1973For AMDGPU the lookup by address section header table: 1974 1975``address_size`` (ubyte) 1976 1977 Match the address size for the ``Global`` address space defined in 1978 :ref:`amdgpu-dwarf-address-space-identifier`. 1979 1980``segment_selector_size`` (ubyte) 1981 1982 AMDGPU does not use a segment selector so this is 0. The entries in the 1983 ``.debug_aranges`` do not have a segment selector. 1984 1985Line Number Information 1986----------------------- 1987 1988See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 1989 1990AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 1991The instruction set must be obtained from the ELF file header ``e_flags`` field 1992in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 1993<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 1994 1995.. TODO:: 1996 1997 Should the ``isa`` state machine register be used to indicate if the code is 1998 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 1999 2000For AMDGPU the line number program header fields have the following values (see 2001DWARF Version 5 section 6.2.4): 2002 2003``address_size`` (ubyte) 2004 Matches the address size for the ``Global`` address space defined in 2005 :ref:`amdgpu-dwarf-address-space-identifier`. 2006 2007``segment_selector_size`` (ubyte) 2008 AMDGPU does not use a segment selector so this is 0. 2009 2010``minimum_instruction_length`` (ubyte) 2011 For GFX9-GFX10 this is 4. 2012 2013``maximum_operations_per_instruction`` (ubyte) 2014 For GFX9-GFX10 this is 1. 2015 2016Source text for online-compiled programs (for example, those compiled by the 2017OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2018See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2019Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2020<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2021 2022The Clang option used to control source embedding in AMDGPU is defined in 2023:ref:`amdgpu-clang-debug-options-table`. 2024 2025 .. table:: AMDGPU Clang Debug Options 2026 :name: amdgpu-clang-debug-options-table 2027 2028 ==================== ================================================== 2029 Debug Flag Description 2030 ==================== ================================================== 2031 -g[no-]embed-source Enable/disable embedding source text in DWARF 2032 debug sections. Useful for environments where 2033 source cannot be written to disk, such as 2034 when performing online compilation. 2035 ==================== ================================================== 2036 2037For example: 2038 2039``-gembed-source`` 2040 Enable the embedded source. 2041 2042``-gno-embed-source`` 2043 Disable the embedded source. 2044 204532-Bit and 64-Bit DWARF Formats 2046------------------------------- 2047 2048See DWARF Version 5 section 7.4 and 2049:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2050 2051For AMDGPU: 2052 2053* For the ``amdgcn`` target architecture only the 64-bit process address space 2054 is supported. 2055 2056* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2057 the 32-bit DWARF format. 2058 2059Unit Headers 2060------------ 2061 2062For AMDGPU the following values apply for each of the unit headers described in 2063DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2064 2065``address_size`` (ubyte) 2066 Matches the address size for the ``Global`` address space defined in 2067 :ref:`amdgpu-dwarf-address-space-identifier`. 2068 2069.. _amdgpu-code-conventions: 2070 2071Code Conventions 2072================ 2073 2074This section provides code conventions used for each supported target triple OS 2075(see :ref:`amdgpu-target-triples`). 2076 2077AMDHSA 2078------ 2079 2080This section provides code conventions used when the target triple OS is 2081``amdhsa`` (see :ref:`amdgpu-target-triples`). 2082 2083.. _amdgpu-amdhsa-code-object-target-identification: 2084 2085Code Object Target Identification 2086~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2087 2088The AMDHSA OS uses the following syntax to specify the code object 2089target as a single string: 2090 2091 ``<Architecture>-<Vendor>-<OS>-<Environment>-<Processor><Target Features>`` 2092 2093Where: 2094 2095 - ``<Architecture>``, ``<Vendor>``, ``<OS>`` and ``<Environment>`` 2096 are the same as the *Target Triple* (see 2097 :ref:`amdgpu-target-triples`). 2098 2099 - ``<Processor>`` is the same as the *Processor* (see 2100 :ref:`amdgpu-processors`). 2101 2102 - ``<Target Features>`` is a list of the enabled *Target Features* 2103 (see :ref:`amdgpu-target-features`), each prefixed by a plus, that 2104 apply to *Processor*. The list must be in the same order as listed 2105 in the table :ref:`amdgpu-target-feature-table`. Note that *Target 2106 Features* must be included in the list if they are enabled even if 2107 that is the default for *Processor*. 2108 2109For example: 2110 2111 ``"amdgcn-amd-amdhsa--gfx902+xnack"`` 2112 2113.. _amdgpu-amdhsa-code-object-metadata: 2114 2115Code Object Metadata 2116~~~~~~~~~~~~~~~~~~~~ 2117 2118The code object metadata specifies extensible metadata associated with the code 2119objects executed on HSA [HSA]_ compatible runtimes such as AMD's ROCm 2120[AMD-ROCm]_. The encoding and semantics of this metadata depends on the code 2121object version; see :ref:`amdgpu-amdhsa-code-object-metadata-v2` and 2122:ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2123 2124Code object metadata is specified in a note record (see 2125:ref:`amdgpu-note-records`) and is required when the target triple OS is 2126``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2127information necessary to support the ROCM kernel queries. For example, the 2128segment sizes needed in a dispatch packet. In addition, a high-level language 2129runtime may require other information to be included. For example, the AMD 2130OpenCL runtime records kernel argument information. 2131 2132.. _amdgpu-amdhsa-code-object-metadata-v2: 2133 2134Code Object V2 Metadata (--amdhsa-code-object-version=2) 2135++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2136 2137.. warning:: Code Object V2 is not the default code object version emitted by 2138 this version of LLVM. For a description of the metadata generated with the 2139 default configuration (Code Object V3) see 2140 :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 2141 2142Code object V2 metadata is specified by the ``NT_AMD_AMDGPU_METADATA`` note 2143record (see :ref:`amdgpu-note-records-v2`). 2144 2145The metadata is specified as a YAML formatted string (see [YAML]_ and 2146:doc:`YamlIO`). 2147 2148.. TODO:: 2149 2150 Is the string null terminated? It probably should not if YAML allows it to 2151 contain null characters, otherwise it should be. 2152 2153The metadata is represented as a single YAML document comprised of the mapping 2154defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v2` and 2155referenced tables. 2156 2157For boolean values, the string values of ``false`` and ``true`` are used for 2158false and true respectively. 2159 2160Additional information can be added to the mappings. To avoid conflicts, any 2161non-AMD key names should be prefixed by "*vendor-name*.". 2162 2163 .. table:: AMDHSA Code Object V2 Metadata Map 2164 :name: amdgpu-amdhsa-code-object-metadata-map-table-v2 2165 2166 ========== ============== ========= ======================================= 2167 String Key Value Type Required? Description 2168 ========== ============== ========= ======================================= 2169 "Version" sequence of Required - The first integer is the major 2170 2 integers version. Currently 1. 2171 - The second integer is the minor 2172 version. Currently 0. 2173 "Printf" sequence of Each string is encoded information 2174 strings about a printf function call. The 2175 encoded information is organized as 2176 fields separated by colon (':'): 2177 2178 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2179 2180 where: 2181 2182 ``ID`` 2183 A 32-bit integer as a unique id for 2184 each printf function call 2185 2186 ``N`` 2187 A 32-bit integer equal to the number 2188 of arguments of printf function call 2189 minus 1 2190 2191 ``S[i]`` (where i = 0, 1, ... , N-1) 2192 32-bit integers for the size in bytes 2193 of the i-th FormatString argument of 2194 the printf function call 2195 2196 FormatString 2197 The format string passed to the 2198 printf function call. 2199 "Kernels" sequence of Required Sequence of the mappings for each 2200 mapping kernel in the code object. See 2201 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2` 2202 for the definition of the mapping. 2203 ========== ============== ========= ======================================= 2204 2205.. 2206 2207 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2208 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v2 2209 2210 ================= ============== ========= ================================ 2211 String Key Value Type Required? Description 2212 ================= ============== ========= ================================ 2213 "Name" string Required Source name of the kernel. 2214 "SymbolName" string Required Name of the kernel 2215 descriptor ELF symbol. 2216 "Language" string Source language of the kernel. 2217 Values include: 2218 2219 - "OpenCL C" 2220 - "OpenCL C++" 2221 - "HCC" 2222 - "OpenMP" 2223 2224 "LanguageVersion" sequence of - The first integer is the major 2225 2 integers version. 2226 - The second integer is the 2227 minor version. 2228 "Attrs" mapping Mapping of kernel attributes. 2229 See 2230 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2` 2231 for the mapping definition. 2232 "Args" sequence of Sequence of mappings of the 2233 mapping kernel arguments. See 2234 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2` 2235 for the definition of the mapping. 2236 "CodeProps" mapping Mapping of properties related to 2237 the kernel code. See 2238 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2` 2239 for the mapping definition. 2240 ================= ============== ========= ================================ 2241 2242.. 2243 2244 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2245 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-table-v2 2246 2247 =================== ============== ========= ============================== 2248 String Key Value Type Required? Description 2249 =================== ============== ========= ============================== 2250 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2251 3 integers must be >=1 and the dispatch 2252 work-group size X, Y, Z must 2253 correspond to the specified 2254 values. Defaults to 0, 0, 0. 2255 2256 Corresponds to the OpenCL 2257 ``reqd_work_group_size`` 2258 attribute. 2259 "WorkGroupSizeHint" sequence of The dispatch work-group size 2260 3 integers X, Y, Z is likely to be the 2261 specified values. 2262 2263 Corresponds to the OpenCL 2264 ``work_group_size_hint`` 2265 attribute. 2266 "VecTypeHint" string The name of a scalar or vector 2267 type. 2268 2269 Corresponds to the OpenCL 2270 ``vec_type_hint`` attribute. 2271 2272 "RuntimeHandle" string The external symbol name 2273 associated with a kernel. 2274 OpenCL runtime allocates a 2275 global buffer for the symbol 2276 and saves the kernel's address 2277 to it, which is used for 2278 device side enqueueing. Only 2279 available for device side 2280 enqueued kernels. 2281 =================== ============== ========= ============================== 2282 2283.. 2284 2285 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2286 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v2 2287 2288 ================= ============== ========= ================================ 2289 String Key Value Type Required? Description 2290 ================= ============== ========= ================================ 2291 "Name" string Kernel argument name. 2292 "TypeName" string Kernel argument type name. 2293 "Size" integer Required Kernel argument size in bytes. 2294 "Align" integer Required Kernel argument alignment in 2295 bytes. Must be a power of two. 2296 "ValueKind" string Required Kernel argument kind that 2297 specifies how to set up the 2298 corresponding argument. 2299 Values include: 2300 2301 "ByValue" 2302 The argument is copied 2303 directly into the kernarg. 2304 2305 "GlobalBuffer" 2306 A global address space pointer 2307 to the buffer data is passed 2308 in the kernarg. 2309 2310 "DynamicSharedPointer" 2311 A group address space pointer 2312 to dynamically allocated LDS 2313 is passed in the kernarg. 2314 2315 "Sampler" 2316 A global address space 2317 pointer to a S# is passed in 2318 the kernarg. 2319 2320 "Image" 2321 A global address space 2322 pointer to a T# is passed in 2323 the kernarg. 2324 2325 "Pipe" 2326 A global address space pointer 2327 to an OpenCL pipe is passed in 2328 the kernarg. 2329 2330 "Queue" 2331 A global address space pointer 2332 to an OpenCL device enqueue 2333 queue is passed in the 2334 kernarg. 2335 2336 "HiddenGlobalOffsetX" 2337 The OpenCL grid dispatch 2338 global offset for the X 2339 dimension is passed in the 2340 kernarg. 2341 2342 "HiddenGlobalOffsetY" 2343 The OpenCL grid dispatch 2344 global offset for the Y 2345 dimension is passed in the 2346 kernarg. 2347 2348 "HiddenGlobalOffsetZ" 2349 The OpenCL grid dispatch 2350 global offset for the Z 2351 dimension is passed in the 2352 kernarg. 2353 2354 "HiddenNone" 2355 An argument that is not used 2356 by the kernel. Space needs to 2357 be left for it, but it does 2358 not need to be set up. 2359 2360 "HiddenPrintfBuffer" 2361 A global address space pointer 2362 to the runtime printf buffer 2363 is passed in kernarg. 2364 2365 "HiddenHostcallBuffer" 2366 A global address space pointer 2367 to the runtime hostcall buffer 2368 is passed in kernarg. 2369 2370 "HiddenDefaultQueue" 2371 A global address space pointer 2372 to the OpenCL device enqueue 2373 queue that should be used by 2374 the kernel by default is 2375 passed in the kernarg. 2376 2377 "HiddenCompletionAction" 2378 A global address space pointer 2379 to help link enqueued kernels into 2380 the ancestor tree for determining 2381 when the parent kernel has finished. 2382 2383 "HiddenMultiGridSyncArg" 2384 A global address space pointer for 2385 multi-grid synchronization is 2386 passed in the kernarg. 2387 2388 "ValueType" string Unused and deprecated. This should no longer 2389 be emitted, but is accepted for compatibility. 2390 2391 2392 "PointeeAlign" integer Alignment in bytes of pointee 2393 type for pointer type kernel 2394 argument. Must be a power 2395 of 2. Only present if 2396 "ValueKind" is 2397 "DynamicSharedPointer". 2398 "AddrSpaceQual" string Kernel argument address space 2399 qualifier. Only present if 2400 "ValueKind" is "GlobalBuffer" or 2401 "DynamicSharedPointer". Values 2402 are: 2403 2404 - "Private" 2405 - "Global" 2406 - "Constant" 2407 - "Local" 2408 - "Generic" 2409 - "Region" 2410 2411 .. TODO:: 2412 2413 Is GlobalBuffer only Global 2414 or Constant? Is 2415 DynamicSharedPointer always 2416 Local? Can HCC allow Generic? 2417 How can Private or Region 2418 ever happen? 2419 2420 "AccQual" string Kernel argument access 2421 qualifier. Only present if 2422 "ValueKind" is "Image" or 2423 "Pipe". Values 2424 are: 2425 2426 - "ReadOnly" 2427 - "WriteOnly" 2428 - "ReadWrite" 2429 2430 .. TODO:: 2431 2432 Does this apply to 2433 GlobalBuffer? 2434 2435 "ActualAccQual" string The actual memory accesses 2436 performed by the kernel on the 2437 kernel argument. Only present if 2438 "ValueKind" is "GlobalBuffer", 2439 "Image", or "Pipe". This may be 2440 more restrictive than indicated 2441 by "AccQual" to reflect what the 2442 kernel actual does. If not 2443 present then the runtime must 2444 assume what is implied by 2445 "AccQual" and "IsConst". Values 2446 are: 2447 2448 - "ReadOnly" 2449 - "WriteOnly" 2450 - "ReadWrite" 2451 2452 "IsConst" boolean Indicates if the kernel argument 2453 is const qualified. Only present 2454 if "ValueKind" is 2455 "GlobalBuffer". 2456 2457 "IsRestrict" boolean Indicates if the kernel argument 2458 is restrict qualified. Only 2459 present if "ValueKind" is 2460 "GlobalBuffer". 2461 2462 "IsVolatile" boolean Indicates if the kernel argument 2463 is volatile qualified. Only 2464 present if "ValueKind" is 2465 "GlobalBuffer". 2466 2467 "IsPipe" boolean Indicates if the kernel argument 2468 is pipe qualified. Only present 2469 if "ValueKind" is "Pipe". 2470 2471 .. TODO:: 2472 2473 Can GlobalBuffer be pipe 2474 qualified? 2475 2476 ================= ============== ========= ================================ 2477 2478.. 2479 2480 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2481 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-table-v2 2482 2483 ============================ ============== ========= ===================== 2484 String Key Value Type Required? Description 2485 ============================ ============== ========= ===================== 2486 "KernargSegmentSize" integer Required The size in bytes of 2487 the kernarg segment 2488 that holds the values 2489 of the arguments to 2490 the kernel. 2491 "GroupSegmentFixedSize" integer Required The amount of group 2492 segment memory 2493 required by a 2494 work-group in 2495 bytes. This does not 2496 include any 2497 dynamically allocated 2498 group segment memory 2499 that may be added 2500 when the kernel is 2501 dispatched. 2502 "PrivateSegmentFixedSize" integer Required The amount of fixed 2503 private address space 2504 memory required for a 2505 work-item in 2506 bytes. If the kernel 2507 uses a dynamic call 2508 stack then additional 2509 space must be added 2510 to this value for the 2511 call stack. 2512 "KernargSegmentAlign" integer Required The maximum byte 2513 alignment of 2514 arguments in the 2515 kernarg segment. Must 2516 be a power of 2. 2517 "WavefrontSize" integer Required Wavefront size. Must 2518 be a power of 2. 2519 "NumSGPRs" integer Required Number of scalar 2520 registers used by a 2521 wavefront for 2522 GFX6-GFX10. This 2523 includes the special 2524 SGPRs for VCC, Flat 2525 Scratch (GFX7-GFX10) 2526 and XNACK (for 2527 GFX8-GFX10). It does 2528 not include the 16 2529 SGPR added if a trap 2530 handler is 2531 enabled. It is not 2532 rounded up to the 2533 allocation 2534 granularity. 2535 "NumVGPRs" integer Required Number of vector 2536 registers used by 2537 each work-item for 2538 GFX6-GFX10 2539 "MaxFlatWorkGroupSize" integer Required Maximum flat 2540 work-group size 2541 supported by the 2542 kernel in work-items. 2543 Must be >=1 and 2544 consistent with 2545 ReqdWorkGroupSize if 2546 not 0, 0, 0. 2547 "NumSpilledSGPRs" integer Number of stores from 2548 a scalar register to 2549 a register allocator 2550 created spill 2551 location. 2552 "NumSpilledVGPRs" integer Number of stores from 2553 a vector register to 2554 a register allocator 2555 created spill 2556 location. 2557 ============================ ============== ========= ===================== 2558 2559.. _amdgpu-amdhsa-code-object-metadata-v3: 2560 2561Code Object V3 Metadata (--amdhsa-code-object-version=3) 2562++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2563 2564Code object V3 metadata is specified by the ``NT_AMDGPU_METADATA`` note record 2565(see :ref:`amdgpu-note-records-v3`). 2566 2567The metadata is represented as Message Pack formatted binary data (see 2568[MsgPack]_). The top level is a Message Pack map that includes the 2569keys defined in table 2570:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 2571tables. 2572 2573Additional information can be added to the maps. To avoid conflicts, 2574any key names should be prefixed by "*vendor-name*." where 2575``vendor-name`` can be the name of the vendor and specific vendor 2576tool that generates the information. The prefix is abbreviated to 2577simply "." when it appears within a map that has been added by the 2578same *vendor-name*. 2579 2580 .. table:: AMDHSA Code Object V3 Metadata Map 2581 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 2582 2583 ================= ============== ========= ======================================= 2584 String Key Value Type Required? Description 2585 ================= ============== ========= ======================================= 2586 "amdhsa.version" sequence of Required - The first integer is the major 2587 2 integers version. Currently 1. 2588 - The second integer is the minor 2589 version. Currently 0. 2590 "amdhsa.printf" sequence of Each string is encoded information 2591 strings about a printf function call. The 2592 encoded information is organized as 2593 fields separated by colon (':'): 2594 2595 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2596 2597 where: 2598 2599 ``ID`` 2600 A 32-bit integer as a unique id for 2601 each printf function call 2602 2603 ``N`` 2604 A 32-bit integer equal to the number 2605 of arguments of printf function call 2606 minus 1 2607 2608 ``S[i]`` (where i = 0, 1, ... , N-1) 2609 32-bit integers for the size in bytes 2610 of the i-th FormatString argument of 2611 the printf function call 2612 2613 FormatString 2614 The format string passed to the 2615 printf function call. 2616 "amdhsa.kernels" sequence of Required Sequence of the maps for each 2617 map kernel in the code object. See 2618 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 2619 for the definition of the keys included 2620 in that map. 2621 ================= ============== ========= ======================================= 2622 2623.. 2624 2625 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 2626 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 2627 2628 =================================== ============== ========= ================================ 2629 String Key Value Type Required? Description 2630 =================================== ============== ========= ================================ 2631 ".name" string Required Source name of the kernel. 2632 ".symbol" string Required Name of the kernel 2633 descriptor ELF symbol. 2634 ".language" string Source language of the kernel. 2635 Values include: 2636 2637 - "OpenCL C" 2638 - "OpenCL C++" 2639 - "HCC" 2640 - "HIP" 2641 - "OpenMP" 2642 - "Assembler" 2643 2644 ".language_version" sequence of - The first integer is the major 2645 2 integers version. 2646 - The second integer is the 2647 minor version. 2648 ".args" sequence of Sequence of maps of the 2649 map kernel arguments. See 2650 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 2651 for the definition of the keys 2652 included in that map. 2653 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 2654 3 integers must be >=1 and the dispatch 2655 work-group size X, Y, Z must 2656 correspond to the specified 2657 values. Defaults to 0, 0, 0. 2658 2659 Corresponds to the OpenCL 2660 ``reqd_work_group_size`` 2661 attribute. 2662 ".workgroup_size_hint" sequence of The dispatch work-group size 2663 3 integers X, Y, Z is likely to be the 2664 specified values. 2665 2666 Corresponds to the OpenCL 2667 ``work_group_size_hint`` 2668 attribute. 2669 ".vec_type_hint" string The name of a scalar or vector 2670 type. 2671 2672 Corresponds to the OpenCL 2673 ``vec_type_hint`` attribute. 2674 2675 ".device_enqueue_symbol" string The external symbol name 2676 associated with a kernel. 2677 OpenCL runtime allocates a 2678 global buffer for the symbol 2679 and saves the kernel's address 2680 to it, which is used for 2681 device side enqueueing. Only 2682 available for device side 2683 enqueued kernels. 2684 ".kernarg_segment_size" integer Required The size in bytes of 2685 the kernarg segment 2686 that holds the values 2687 of the arguments to 2688 the kernel. 2689 ".group_segment_fixed_size" integer Required The amount of group 2690 segment memory 2691 required by a 2692 work-group in 2693 bytes. This does not 2694 include any 2695 dynamically allocated 2696 group segment memory 2697 that may be added 2698 when the kernel is 2699 dispatched. 2700 ".private_segment_fixed_size" integer Required The amount of fixed 2701 private address space 2702 memory required for a 2703 work-item in 2704 bytes. If the kernel 2705 uses a dynamic call 2706 stack then additional 2707 space must be added 2708 to this value for the 2709 call stack. 2710 ".kernarg_segment_align" integer Required The maximum byte 2711 alignment of 2712 arguments in the 2713 kernarg segment. Must 2714 be a power of 2. 2715 ".wavefront_size" integer Required Wavefront size. Must 2716 be a power of 2. 2717 ".sgpr_count" integer Required Number of scalar 2718 registers required by a 2719 wavefront for 2720 GFX6-GFX9. A register 2721 is required if it is 2722 used explicitly, or 2723 if a higher numbered 2724 register is used 2725 explicitly. This 2726 includes the special 2727 SGPRs for VCC, Flat 2728 Scratch (GFX7-GFX9) 2729 and XNACK (for 2730 GFX8-GFX9). It does 2731 not include the 16 2732 SGPR added if a trap 2733 handler is 2734 enabled. It is not 2735 rounded up to the 2736 allocation 2737 granularity. 2738 ".vgpr_count" integer Required Number of vector 2739 registers required by 2740 each work-item for 2741 GFX6-GFX9. A register 2742 is required if it is 2743 used explicitly, or 2744 if a higher numbered 2745 register is used 2746 explicitly. 2747 ".max_flat_workgroup_size" integer Required Maximum flat 2748 work-group size 2749 supported by the 2750 kernel in work-items. 2751 Must be >=1 and 2752 consistent with 2753 ReqdWorkGroupSize if 2754 not 0, 0, 0. 2755 ".sgpr_spill_count" integer Number of stores from 2756 a scalar register to 2757 a register allocator 2758 created spill 2759 location. 2760 ".vgpr_spill_count" integer Number of stores from 2761 a vector register to 2762 a register allocator 2763 created spill 2764 location. 2765 =================================== ============== ========= ================================ 2766 2767.. 2768 2769 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 2770 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 2771 2772 ====================== ============== ========= ================================ 2773 String Key Value Type Required? Description 2774 ====================== ============== ========= ================================ 2775 ".name" string Kernel argument name. 2776 ".type_name" string Kernel argument type name. 2777 ".size" integer Required Kernel argument size in bytes. 2778 ".offset" integer Required Kernel argument offset in 2779 bytes. The offset must be a 2780 multiple of the alignment 2781 required by the argument. 2782 ".value_kind" string Required Kernel argument kind that 2783 specifies how to set up the 2784 corresponding argument. 2785 Values include: 2786 2787 "by_value" 2788 The argument is copied 2789 directly into the kernarg. 2790 2791 "global_buffer" 2792 A global address space pointer 2793 to the buffer data is passed 2794 in the kernarg. 2795 2796 "dynamic_shared_pointer" 2797 A group address space pointer 2798 to dynamically allocated LDS 2799 is passed in the kernarg. 2800 2801 "sampler" 2802 A global address space 2803 pointer to a S# is passed in 2804 the kernarg. 2805 2806 "image" 2807 A global address space 2808 pointer to a T# is passed in 2809 the kernarg. 2810 2811 "pipe" 2812 A global address space pointer 2813 to an OpenCL pipe is passed in 2814 the kernarg. 2815 2816 "queue" 2817 A global address space pointer 2818 to an OpenCL device enqueue 2819 queue is passed in the 2820 kernarg. 2821 2822 "hidden_global_offset_x" 2823 The OpenCL grid dispatch 2824 global offset for the X 2825 dimension is passed in the 2826 kernarg. 2827 2828 "hidden_global_offset_y" 2829 The OpenCL grid dispatch 2830 global offset for the Y 2831 dimension is passed in the 2832 kernarg. 2833 2834 "hidden_global_offset_z" 2835 The OpenCL grid dispatch 2836 global offset for the Z 2837 dimension is passed in the 2838 kernarg. 2839 2840 "hidden_none" 2841 An argument that is not used 2842 by the kernel. Space needs to 2843 be left for it, but it does 2844 not need to be set up. 2845 2846 "hidden_printf_buffer" 2847 A global address space pointer 2848 to the runtime printf buffer 2849 is passed in kernarg. 2850 2851 "hidden_hostcall_buffer" 2852 A global address space pointer 2853 to the runtime hostcall buffer 2854 is passed in kernarg. 2855 2856 "hidden_default_queue" 2857 A global address space pointer 2858 to the OpenCL device enqueue 2859 queue that should be used by 2860 the kernel by default is 2861 passed in the kernarg. 2862 2863 "hidden_completion_action" 2864 A global address space pointer 2865 to help link enqueued kernels into 2866 the ancestor tree for determining 2867 when the parent kernel has finished. 2868 2869 "hidden_multigrid_sync_arg" 2870 A global address space pointer for 2871 multi-grid synchronization is 2872 passed in the kernarg. 2873 2874 ".value_type" string Unused and deprecated. This should no longer 2875 be emitted, but is accepted for compatibility. 2876 2877 ".pointee_align" integer Alignment in bytes of pointee 2878 type for pointer type kernel 2879 argument. Must be a power 2880 of 2. Only present if 2881 ".value_kind" is 2882 "dynamic_shared_pointer". 2883 ".address_space" string Kernel argument address space 2884 qualifier. Only present if 2885 ".value_kind" is "global_buffer" or 2886 "dynamic_shared_pointer". Values 2887 are: 2888 2889 - "private" 2890 - "global" 2891 - "constant" 2892 - "local" 2893 - "generic" 2894 - "region" 2895 2896 .. TODO:: 2897 2898 Is "global_buffer" only "global" 2899 or "constant"? Is 2900 "dynamic_shared_pointer" always 2901 "local"? Can HCC allow "generic"? 2902 How can "private" or "region" 2903 ever happen? 2904 2905 ".access" string Kernel argument access 2906 qualifier. Only present if 2907 ".value_kind" is "image" or 2908 "pipe". Values 2909 are: 2910 2911 - "read_only" 2912 - "write_only" 2913 - "read_write" 2914 2915 .. TODO:: 2916 2917 Does this apply to 2918 "global_buffer"? 2919 2920 ".actual_access" string The actual memory accesses 2921 performed by the kernel on the 2922 kernel argument. Only present if 2923 ".value_kind" is "global_buffer", 2924 "image", or "pipe". This may be 2925 more restrictive than indicated 2926 by ".access" to reflect what the 2927 kernel actual does. If not 2928 present then the runtime must 2929 assume what is implied by 2930 ".access" and ".is_const" . Values 2931 are: 2932 2933 - "read_only" 2934 - "write_only" 2935 - "read_write" 2936 2937 ".is_const" boolean Indicates if the kernel argument 2938 is const qualified. Only present 2939 if ".value_kind" is 2940 "global_buffer". 2941 2942 ".is_restrict" boolean Indicates if the kernel argument 2943 is restrict qualified. Only 2944 present if ".value_kind" is 2945 "global_buffer". 2946 2947 ".is_volatile" boolean Indicates if the kernel argument 2948 is volatile qualified. Only 2949 present if ".value_kind" is 2950 "global_buffer". 2951 2952 ".is_pipe" boolean Indicates if the kernel argument 2953 is pipe qualified. Only present 2954 if ".value_kind" is "pipe". 2955 2956 .. TODO:: 2957 2958 Can "global_buffer" be pipe 2959 qualified? 2960 2961 ====================== ============== ========= ================================ 2962 2963.. 2964 2965Kernel Dispatch 2966~~~~~~~~~~~~~~~ 2967 2968The HSA architected queuing language (AQL) defines a user space memory interface 2969that can be used to control the dispatch of kernels, in an agent independent 2970way. An agent can have zero or more AQL queues created for it using the ROCm 2971runtime, in which AQL packets (all of which are 64 bytes) can be placed. See the 2972*HSA Platform System Architecture Specification* [HSA]_ for the AQL queue 2973mechanics and packet layouts. 2974 2975The packet processor of a kernel agent is responsible for detecting and 2976dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 2977packet processor is implemented by the hardware command processor (CP), 2978asynchronous dispatch controller (ADC) and shader processor input controller 2979(SPI). 2980 2981The ROCm runtime can be used to allocate an AQL queue object. It uses the kernel 2982mode driver to initialize and register the AQL queue with CP. 2983 2984To dispatch a kernel the following actions are performed. This can occur in the 2985CPU host program, or from an HSA kernel executing on a GPU. 2986 29871. A pointer to an AQL queue for the kernel agent on which the kernel is to be 2988 executed is obtained. 29892. A pointer to the kernel descriptor (see 2990 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 2991 It must be for a kernel that is contained in a code object that that was 2992 loaded by the ROCm runtime on the kernel agent with which the AQL queue is 2993 associated. 29943. Space is allocated for the kernel arguments using the ROCm runtime allocator 2995 for a memory region with the kernarg property for the kernel agent that will 2996 execute the kernel. It must be at least 16-byte aligned. 29974. Kernel argument values are assigned to the kernel argument memory 2998 allocation. The layout is defined in the *HSA Programmer's Language 2999 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3000 kernel argument memory in the same way constant memory is accessed. (Note 3001 that the HSA specification allows an implementation to copy the kernel 3002 argument contents to another location that is accessed by the kernel.) 30035. An AQL kernel dispatch packet is created on the AQL queue. The ROCm runtime 3004 api uses 64-bit atomic operations to reserve space in the AQL queue for the 3005 packet. The packet must be set up, and the final write must use an atomic 3006 store release to set the packet kind to ensure the packet contents are 3007 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3008 notify the kernel agent that the AQL queue has been updated. These rules, and 3009 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3010 System Architecture Specification* [HSA]_. 30116. A kernel dispatch packet includes information about the actual dispatch, 3012 such as grid and work-group size, together with information from the code 3013 object about the kernel, such as segment sizes. The ROCm runtime queries on 3014 the kernel symbol can be used to obtain the code object values which are 3015 recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 30167. CP executes micro-code and is responsible for detecting and setting up the 3017 GPU to execute the wavefronts of a kernel dispatch. 30188. CP ensures that when the a wavefront starts executing the kernel machine 3019 code, the scalar general purpose registers (SGPR) and vector general purpose 3020 registers (VGPR) are set up as required by the machine code. The required 3021 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3022 register state is defined in 3023 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 30249. The prolog of the kernel machine code (see 3025 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3026 before continuing executing the machine code that corresponds to the kernel. 302710. When the kernel dispatch has completed execution, CP signals the completion 3028 signal specified in the kernel dispatch packet if not 0. 3029 3030.. _amdgpu-amdhsa-memory-spaces: 3031 3032Memory Spaces 3033~~~~~~~~~~~~~ 3034 3035The memory space properties are: 3036 3037 .. table:: AMDHSA Memory Spaces 3038 :name: amdgpu-amdhsa-memory-spaces-table 3039 3040 ================= =========== ======== ======= ================== 3041 Memory Space Name HSA Segment Hardware Address NULL Value 3042 Name Name Size 3043 ================= =========== ======== ======= ================== 3044 Private private scratch 32 0x00000000 3045 Local group LDS 32 0xFFFFFFFF 3046 Global global global 64 0x0000000000000000 3047 Constant constant *same as 64 0x0000000000000000 3048 global* 3049 Generic flat flat 64 0x0000000000000000 3050 Region N/A GDS 32 *not implemented 3051 for AMDHSA* 3052 ================= =========== ======== ======= ================== 3053 3054The global and constant memory spaces both use global virtual addresses, which 3055are the same virtual address space used by the CPU. However, some virtual 3056addresses may only be accessible to the CPU, some only accessible by the GPU, 3057and some by both. 3058 3059Using the constant memory space indicates that the data will not change during 3060the execution of the kernel. This allows scalar read instructions to be 3061used. The vector and scalar L1 caches are invalidated of volatile data before 3062each kernel dispatch execution to allow constant memory to change values between 3063kernel dispatches. 3064 3065The local memory space uses the hardware Local Data Store (LDS) which is 3066automatically allocated when the hardware creates work-groups of wavefronts, and 3067freed when all the wavefronts of a work-group have terminated. The data store 3068(DS) instructions can be used to access it. 3069 3070The private memory space uses the hardware scratch memory support. If the kernel 3071uses scratch, then the hardware allocates memory that is accessed using 3072wavefront lane dword (4 byte) interleaving. The mapping used from private 3073address to physical address is: 3074 3075 ``wavefront-scratch-base + 3076 (private-address * wavefront-size * 4) + 3077 (wavefront-lane-id * 4)`` 3078 3079There are different ways that the wavefront scratch base address is determined 3080by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3081memory can be accessed in an interleaved manner using buffer instruction with 3082the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3083instructions, or by flat instructions. If each lane of a wavefront accesses the 3084same private address, the interleaving results in adjacent dwords being accessed 3085and hence requires fewer cache lines to be fetched. Multi-dword access is not 3086supported except by flat and scratch instructions in GFX9-GFX10. 3087 3088The generic address space uses the hardware flat address support available in 3089GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3090local apertures), that are outside the range of addressible global memory, to 3091map from a flat address to a private or local address. 3092 3093FLAT instructions can take a flat address and access global, private (scratch) 3094and group (LDS) memory depending in if the address is within one of the 3095aperture ranges. Flat access to scratch requires hardware aperture setup and 3096setup in the kernel prologue (see 3097:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3098hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3099:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3100 3101To convert between a segment address and a flat address the base address of the 3102apertures address can be used. For GFX7-GFX8 these are available in the 3103:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3104Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3105GFX9-GFX10 the aperture base addresses are directly available as inline constant 3106registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3107address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3108which makes it easier to convert from flat to segment or segment to flat. 3109 3110Image and Samplers 3111~~~~~~~~~~~~~~~~~~ 3112 3113Image and sample handles created by the ROCm runtime are 64-bit addresses of a 3114hardware 32-byte V# and 48 byte S# object respectively. In order to support the 3115HSA ``query_sampler`` operations two extra dwords are used to store the HSA BRIG 3116enumeration values for the queries that are not trivially deducible from the S# 3117representation. 3118 3119HSA Signals 3120~~~~~~~~~~~ 3121 3122HSA signal handles created by the ROCm runtime are 64-bit addresses of a 3123structure allocated in memory accessible from both the CPU and GPU. The 3124structure is defined by the ROCm runtime and subject to change between releases 3125(see [AMD-ROCm-github]_). 3126 3127.. _amdgpu-amdhsa-hsa-aql-queue: 3128 3129HSA AQL Queue 3130~~~~~~~~~~~~~ 3131 3132The HSA AQL queue structure is defined by the ROCm runtime and subject to change 3133between releases (see [AMD-ROCm-github]_). For some processors it contains 3134fields needed to implement certain language features such as the flat address 3135aperture bases. It also contains fields used by CP such as managing the 3136allocation of scratch memory. 3137 3138.. _amdgpu-amdhsa-kernel-descriptor: 3139 3140Kernel Descriptor 3141~~~~~~~~~~~~~~~~~ 3142 3143A kernel descriptor consists of the information needed by CP to initiate the 3144execution of a kernel, including the entry point address of the machine code 3145that implements the kernel. 3146 3147Kernel Descriptor for GFX6-GFX10 3148++++++++++++++++++++++++++++++++ 3149 3150CP microcode requires the Kernel descriptor to be allocated on 64-byte 3151alignment. 3152 3153 .. table:: Kernel Descriptor for GFX6-GFX10 3154 :name: amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table 3155 3156 ======= ======= =============================== ============================ 3157 Bits Size Field Name Description 3158 ======= ======= =============================== ============================ 3159 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3160 address space memory 3161 required for a work-group 3162 in bytes. This does not 3163 include any dynamically 3164 allocated local address 3165 space memory that may be 3166 added when the kernel is 3167 dispatched. 3168 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3169 private address space 3170 memory required for a 3171 work-item in bytes. If 3172 is_dynamic_callstack is 1 3173 then additional space must 3174 be added to this value for 3175 the call stack. 3176 127:64 8 bytes Reserved, must be 0. 3177 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3178 negative) from base 3179 address of kernel 3180 descriptor to kernel's 3181 entry point instruction 3182 which must be 256 byte 3183 aligned. 3184 351:272 20 Reserved, must be 0. 3185 bytes 3186 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-9 3187 Reserved, must be 0. 3188 GFX10 3189 Compute Shader (CS) 3190 program settings used by 3191 CP to set up 3192 ``COMPUTE_PGM_RSRC3`` 3193 configuration 3194 register. See 3195 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3196 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3197 program settings used by 3198 CP to set up 3199 ``COMPUTE_PGM_RSRC1`` 3200 configuration 3201 register. See 3202 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3203 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3204 program settings used by 3205 CP to set up 3206 ``COMPUTE_PGM_RSRC2`` 3207 configuration 3208 register. See 3209 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3210 448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3211 _BUFFER SGPR user data registers 3212 (see 3213 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3214 3215 The total number of SGPR 3216 user data registers 3217 requested must not exceed 3218 16 and match value in 3219 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3220 Any requests beyond 16 3221 will be ignored. 3222 449 1 bit ENABLE_SGPR_DISPATCH_PTR *see above* 3223 450 1 bit ENABLE_SGPR_QUEUE_PTR *see above* 3224 451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR *see above* 3225 452 1 bit ENABLE_SGPR_DISPATCH_ID *see above* 3226 453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT *see above* 3227 454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT *see above* 3228 _SIZE 3229 457:455 3 bits Reserved, must be 0. 3230 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-9 3231 Reserved, must be 0. 3232 GFX10 3233 - If 0 execute in 3234 wavefront size 64 mode. 3235 - If 1 execute in 3236 native wavefront size 3237 32 mode. 3238 463:459 5 bits Reserved, must be 0. 3239 511:464 6 bytes Reserved, must be 0. 3240 512 **Total size 64 bytes.** 3241 ======= ==================================================================== 3242 3243.. 3244 3245 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 3246 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 3247 3248 ======= ======= =============================== =========================================================================== 3249 Bits Size Field Name Description 3250 ======= ======= =============================== =========================================================================== 3251 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 3252 blocks used by each work-item; 3253 granularity is device 3254 specific: 3255 3256 GFX6-GFX9 3257 - vgprs_used 0..256 3258 - max(0, ceil(vgprs_used / 4) - 1) 3259 GFX10 (wavefront size 64) 3260 - max_vgpr 1..256 3261 - max(0, ceil(vgprs_used / 4) - 1) 3262 GFX10 (wavefront size 32) 3263 - max_vgpr 1..256 3264 - max(0, ceil(vgprs_used / 8) - 1) 3265 3266 Where vgprs_used is defined 3267 as the highest VGPR number 3268 explicitly referenced plus 3269 one. 3270 3271 Used by CP to set up 3272 ``COMPUTE_PGM_RSRC1.VGPRS``. 3273 3274 The 3275 :ref:`amdgpu-assembler` 3276 calculates this 3277 automatically for the 3278 selected processor from 3279 values provided to the 3280 `.amdhsa_kernel` directive 3281 by the 3282 `.amdhsa_next_free_vgpr` 3283 nested directive (see 3284 :ref:`amdhsa-kernel-directives-table`). 3285 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 3286 blocks used by a wavefront; 3287 granularity is device 3288 specific: 3289 3290 GFX6-GFX8 3291 - sgprs_used 0..112 3292 - max(0, ceil(sgprs_used / 8) - 1) 3293 GFX9 3294 - sgprs_used 0..112 3295 - 2 * max(0, ceil(sgprs_used / 16) - 1) 3296 GFX10 3297 Reserved, must be 0. 3298 (128 SGPRs always 3299 allocated.) 3300 3301 Where sgprs_used is 3302 defined as the highest 3303 SGPR number explicitly 3304 referenced plus one, plus 3305 a target specific number 3306 of additional special 3307 SGPRs for VCC, 3308 FLAT_SCRATCH (GFX7+) and 3309 XNACK_MASK (GFX8+), and 3310 any additional 3311 target specific 3312 limitations. It does not 3313 include the 16 SGPRs added 3314 if a trap handler is 3315 enabled. 3316 3317 The target specific 3318 limitations and special 3319 SGPR layout are defined in 3320 the hardware 3321 documentation, which can 3322 be found in the 3323 :ref:`amdgpu-processors` 3324 table. 3325 3326 Used by CP to set up 3327 ``COMPUTE_PGM_RSRC1.SGPRS``. 3328 3329 The 3330 :ref:`amdgpu-assembler` 3331 calculates this 3332 automatically for the 3333 selected processor from 3334 values provided to the 3335 `.amdhsa_kernel` directive 3336 by the 3337 `.amdhsa_next_free_sgpr` 3338 and `.amdhsa_reserve_*` 3339 nested directives (see 3340 :ref:`amdhsa-kernel-directives-table`). 3341 11:10 2 bits PRIORITY Must be 0. 3342 3343 Start executing wavefront 3344 at the specified priority. 3345 3346 CP is responsible for 3347 filling in 3348 ``COMPUTE_PGM_RSRC1.PRIORITY``. 3349 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 3350 with specified rounding 3351 mode for single (32 3352 bit) floating point 3353 precision floating point 3354 operations. 3355 3356 Floating point rounding 3357 mode values are defined in 3358 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3359 3360 Used by CP to set up 3361 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3362 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 3363 with specified rounding 3364 denorm mode for half/double (16 3365 and 64-bit) floating point 3366 precision floating point 3367 operations. 3368 3369 Floating point rounding 3370 mode values are defined in 3371 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 3372 3373 Used by CP to set up 3374 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3375 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 3376 with specified denorm mode 3377 for single (32 3378 bit) floating point 3379 precision floating point 3380 operations. 3381 3382 Floating point denorm mode 3383 values are defined in 3384 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3385 3386 Used by CP to set up 3387 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3388 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 3389 with specified denorm mode 3390 for half/double (16 3391 and 64-bit) floating point 3392 precision floating point 3393 operations. 3394 3395 Floating point denorm mode 3396 values are defined in 3397 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 3398 3399 Used by CP to set up 3400 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 3401 20 1 bit PRIV Must be 0. 3402 3403 Start executing wavefront 3404 in privilege trap handler 3405 mode. 3406 3407 CP is responsible for 3408 filling in 3409 ``COMPUTE_PGM_RSRC1.PRIV``. 3410 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 3411 with DX10 clamp mode 3412 enabled. Used by the vector 3413 ALU to force DX10 style 3414 treatment of NaN's (when 3415 set, clamp NaN to zero, 3416 otherwise pass NaN 3417 through). 3418 3419 Used by CP to set up 3420 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 3421 22 1 bit DEBUG_MODE Must be 0. 3422 3423 Start executing wavefront 3424 in single step mode. 3425 3426 CP is responsible for 3427 filling in 3428 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 3429 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 3430 with IEEE mode 3431 enabled. Floating point 3432 opcodes that support 3433 exception flag gathering 3434 will quiet and propagate 3435 signaling-NaN inputs per 3436 IEEE 754-2008. Min_dx10 and 3437 max_dx10 become IEEE 3438 754-2008 compliant due to 3439 signaling-NaN propagation 3440 and quieting. 3441 3442 Used by CP to set up 3443 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 3444 24 1 bit BULKY Must be 0. 3445 3446 Only one work-group allowed 3447 to execute on a compute 3448 unit. 3449 3450 CP is responsible for 3451 filling in 3452 ``COMPUTE_PGM_RSRC1.BULKY``. 3453 25 1 bit CDBG_USER Must be 0. 3454 3455 Flag that can be used to 3456 control debugging code. 3457 3458 CP is responsible for 3459 filling in 3460 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 3461 26 1 bit FP16_OVFL GFX6-GFX8 3462 Reserved, must be 0. 3463 GFX9-GFX10 3464 Wavefront starts execution 3465 with specified fp16 overflow 3466 mode. 3467 3468 - If 0, fp16 overflow generates 3469 +/-INF values. 3470 - If 1, fp16 overflow that is the 3471 result of an +/-INF input value 3472 or divide by 0 produces a +/-INF, 3473 otherwise clamps computed 3474 overflow to +/-MAX_FP16 as 3475 appropriate. 3476 3477 Used by CP to set up 3478 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 3479 28:27 2 bits Reserved, must be 0. 3480 29 1 bit WGP_MODE GFX6-GFX9 3481 Reserved, must be 0. 3482 GFX10 3483 - If 0 execute work-groups in 3484 CU wavefront execution mode. 3485 - If 1 execute work-groups on 3486 in WGP wavefront execution mode. 3487 3488 See :ref:`amdgpu-amdhsa-memory-model`. 3489 3490 Used by CP to set up 3491 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 3492 30 1 bit MEM_ORDERED GFX6-9 3493 Reserved, must be 0. 3494 GFX10 3495 Controls the behavior of the 3496 s_waitcnt's vmcnt and vscnt 3497 counters. 3498 3499 - If 0 vmcnt reports completion 3500 of load and atomic with return 3501 out of order with sample 3502 instructions, and the vscnt 3503 reports the completion of 3504 store and atomic without 3505 return in order. 3506 - If 1 vmcnt reports completion 3507 of load, atomic with return 3508 and sample instructions in 3509 order, and the vscnt reports 3510 the completion of store and 3511 atomic without return in order. 3512 3513 Used by CP to set up 3514 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 3515 31 1 bit FWD_PROGRESS GFX6-9 3516 Reserved, must be 0. 3517 GFX10 3518 - If 0 execute SIMD wavefronts 3519 using oldest first policy. 3520 - If 1 execute SIMD wavefronts to 3521 ensure wavefronts will make some 3522 forward progress. 3523 3524 Used by CP to set up 3525 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 3526 32 **Total size 4 bytes** 3527 ======= =================================================================================================================== 3528 3529.. 3530 3531 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 3532 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 3533 3534 ======= ======= =============================== =========================================================================== 3535 Bits Size Field Name Description 3536 ======= ======= =============================== =========================================================================== 3537 0 1 bit ENABLE_SGPR_PRIVATE_SEGMENT Enable the setup of the 3538 _WAVEFRONT_OFFSET SGPR wavefront scratch offset 3539 system register (see 3540 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3541 3542 Used by CP to set up 3543 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 3544 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 3545 user data registers 3546 requested. This number must 3547 match the number of user 3548 data registers enabled. 3549 3550 Used by CP to set up 3551 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 3552 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 3553 3554 This bit represents 3555 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 3556 which is set by the CP if 3557 the runtime has installed a 3558 trap handler. 3559 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 3560 system SGPR register for 3561 the work-group id in the X 3562 dimension (see 3563 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3564 3565 Used by CP to set up 3566 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 3567 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 3568 system SGPR register for 3569 the work-group id in the Y 3570 dimension (see 3571 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3572 3573 Used by CP to set up 3574 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 3575 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 3576 system SGPR register for 3577 the work-group id in the Z 3578 dimension (see 3579 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3580 3581 Used by CP to set up 3582 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 3583 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 3584 system SGPR register for 3585 work-group information (see 3586 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3587 3588 Used by CP to set up 3589 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 3590 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 3591 VGPR system registers used 3592 for the work-item ID. 3593 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 3594 defines the values. 3595 3596 Used by CP to set up 3597 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 3598 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 3599 3600 Wavefront starts execution 3601 with address watch 3602 exceptions enabled which 3603 are generated when L1 has 3604 witnessed a thread access 3605 an *address of 3606 interest*. 3607 3608 CP is responsible for 3609 filling in the address 3610 watch bit in 3611 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3612 according to what the 3613 runtime requests. 3614 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 3615 3616 Wavefront starts execution 3617 with memory violation 3618 exceptions exceptions 3619 enabled which are generated 3620 when a memory violation has 3621 occurred for this wavefront from 3622 L1 or LDS 3623 (write-to-read-only-memory, 3624 mis-aligned atomic, LDS 3625 address out of range, 3626 illegal address, etc.). 3627 3628 CP sets the memory 3629 violation bit in 3630 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 3631 according to what the 3632 runtime requests. 3633 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 3634 3635 CP uses the rounded value 3636 from the dispatch packet, 3637 not this value, as the 3638 dispatch may contain 3639 dynamically allocated group 3640 segment memory. CP writes 3641 directly to 3642 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 3643 3644 Amount of group segment 3645 (LDS) to allocate for each 3646 work-group. Granularity is 3647 device specific: 3648 3649 GFX6: 3650 roundup(lds-size / (64 * 4)) 3651 GFX7-GFX10: 3652 roundup(lds-size / (128 * 4)) 3653 3654 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 3655 _INVALID_OPERATION with specified exceptions 3656 enabled. 3657 3658 Used by CP to set up 3659 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 3660 (set from bits 0..6). 3661 3662 IEEE 754 FP Invalid 3663 Operation 3664 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 3665 _SOURCE input operands is a 3666 denormal number 3667 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 3668 _DIVISION_BY_ZERO Zero 3669 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 3670 _OVERFLOW 3671 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 3672 _UNDERFLOW 3673 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 3674 _INEXACT 3675 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 3676 _ZERO (rcp_iflag_f32 instruction 3677 only) 3678 31 1 bit Reserved, must be 0. 3679 32 **Total size 4 bytes.** 3680 ======= =================================================================================================================== 3681 3682.. 3683 3684 .. table:: compute_pgm_rsrc3 for GFX10 3685 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 3686 3687 ======= ======= =============================== =========================================================================== 3688 Bits Size Field Name Description 3689 ======= ======= =============================== =========================================================================== 3690 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPRs for wavefront size 64. Granularity 8. Value 0-120. 3691 compute_pgm_rsrc1.vgprs + shared_vgpr_cnt cannot exceed 64. 3692 31:4 28 Reserved, must be 0. 3693 bits 3694 32 **Total size 4 bytes.** 3695 ======= =================================================================================================================== 3696 3697.. 3698 3699 .. table:: Floating Point Rounding Mode Enumeration Values 3700 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 3701 3702 ====================================== ===== ============================== 3703 Enumeration Name Value Description 3704 ====================================== ===== ============================== 3705 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 3706 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 3707 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 3708 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 3709 ====================================== ===== ============================== 3710 3711.. 3712 3713 .. table:: Floating Point Denorm Mode Enumeration Values 3714 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 3715 3716 ====================================== ===== ============================== 3717 Enumeration Name Value Description 3718 ====================================== ===== ============================== 3719 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 3720 Denorms 3721 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 3722 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 3723 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 3724 ====================================== ===== ============================== 3725 3726.. 3727 3728 .. table:: System VGPR Work-Item ID Enumeration Values 3729 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 3730 3731 ======================================== ===== ============================ 3732 Enumeration Name Value Description 3733 ======================================== ===== ============================ 3734 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 3735 ID. 3736 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 3737 dimensions ID. 3738 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 3739 dimensions ID. 3740 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 3741 ======================================== ===== ============================ 3742 3743.. _amdgpu-amdhsa-initial-kernel-execution-state: 3744 3745Initial Kernel Execution State 3746~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 3747 3748This section defines the register state that will be set up by the packet 3749processor prior to the start of execution of every wavefront. This is limited by 3750the constraints of the hardware controllers of CP/ADC/SPI. 3751 3752The order of the SGPR registers is defined, but the compiler can specify which 3753ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 3754fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3755for enabled registers are dense starting at SGPR0: the first enabled register is 3756SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 3757an SGPR number. 3758 3759The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 3760all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 3761using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 3762actually initialized. These are then immediately followed by the System SGPRs 3763that are set up by ADC/SPI and can have different values for each wavefront of 3764the grid dispatch. 3765 3766SGPR register initial state is defined in 3767:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 3768 3769 .. table:: SGPR Register Set Up Order 3770 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 3771 3772 ========== ========================== ====== ============================== 3773 SGPR Order Name Number Description 3774 (kernel descriptor enable of 3775 field) SGPRs 3776 ========== ========================== ====== ============================== 3777 First Private Segment Buffer 4 V# that can be used, together 3778 (enable_sgpr_private with Scratch Wavefront Offset 3779 _segment_buffer) as an offset, to access the 3780 private memory space using a 3781 segment address. 3782 3783 CP uses the value provided by 3784 the runtime. 3785 then Dispatch Ptr 2 64-bit address of AQL dispatch 3786 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 3787 actually executing. 3788 then Queue Ptr 2 64-bit address of amd_queue_t 3789 (enable_sgpr_queue_ptr) object for AQL queue on which 3790 the dispatch packet was 3791 queued. 3792 then Kernarg Segment Ptr 2 64-bit address of Kernarg 3793 (enable_sgpr_kernarg segment. This is directly 3794 _segment_ptr) copied from the 3795 kernarg_address in the kernel 3796 dispatch packet. 3797 3798 Having CP load it once avoids 3799 loading it at the beginning of 3800 every wavefront. 3801 then Dispatch Id 2 64-bit Dispatch ID of the 3802 (enable_sgpr_dispatch_id) dispatch packet being 3803 executed. 3804 then Flat Scratch Init 2 This is 2 SGPRs: 3805 (enable_sgpr_flat_scratch 3806 _init) GFX6 3807 Not supported. 3808 GFX7-GFX8 3809 The first SGPR is a 32-bit 3810 byte offset from 3811 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3812 to per SPI base of memory 3813 for scratch for the queue 3814 executing the kernel 3815 dispatch. CP obtains this 3816 from the runtime. (The 3817 Scratch Segment Buffer base 3818 address is 3819 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3820 plus this offset.) The value 3821 of Scratch Wavefront Offset must 3822 be added to this offset by 3823 the kernel machine code, 3824 right shifted by 8, and 3825 moved to the FLAT_SCRATCH_HI 3826 SGPR register. 3827 FLAT_SCRATCH_HI corresponds 3828 to SGPRn-4 on GFX7, and 3829 SGPRn-6 on GFX8 (where SGPRn 3830 is the highest numbered SGPR 3831 allocated to the wavefront). 3832 FLAT_SCRATCH_HI is 3833 multiplied by 256 (as it is 3834 in units of 256 bytes) and 3835 added to 3836 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` 3837 to calculate the per wavefront 3838 FLAT SCRATCH BASE in flat 3839 memory instructions that 3840 access the scratch 3841 aperture. 3842 3843 The second SGPR is 32-bit 3844 byte size of a single 3845 work-item's scratch memory 3846 usage. CP obtains this from 3847 the runtime, and it is 3848 always a multiple of DWORD. 3849 CP checks that the value in 3850 the kernel dispatch packet 3851 Private Segment Byte Size is 3852 not larger and requests the 3853 runtime to increase the 3854 queue's scratch size if 3855 necessary. The kernel code 3856 must move it to 3857 FLAT_SCRATCH_LO which is 3858 SGPRn-3 on GFX7 and SGPRn-5 3859 on GFX8. FLAT_SCRATCH_LO is 3860 used as the FLAT SCRATCH 3861 SIZE in flat memory 3862 instructions. Having CP load 3863 it once avoids loading it at 3864 the beginning of every 3865 wavefront. 3866 GFX9-GFX10 3867 This is the 3868 64-bit base address of the 3869 per SPI scratch backing 3870 memory managed by SPI for 3871 the queue executing the 3872 kernel dispatch. CP obtains 3873 this from the runtime (and 3874 divides it if there are 3875 multiple Shader Arrays each 3876 with its own SPI). The value 3877 of Scratch Wavefront Offset must 3878 be added by the kernel 3879 machine code and the result 3880 moved to the FLAT_SCRATCH 3881 SGPR which is SGPRn-6 and 3882 SGPRn-5. It is used as the 3883 FLAT SCRATCH BASE in flat 3884 memory instructions. 3885 then Private Segment Size 1 The 32-bit byte size of a 3886 (enable_sgpr_private single 3887 work-item's 3888 scratch_segment_size) memory 3889 allocation. This is the 3890 value from the kernel 3891 dispatch packet Private 3892 Segment Byte Size rounded up 3893 by CP to a multiple of 3894 DWORD. 3895 3896 Having CP load it once avoids 3897 loading it at the beginning of 3898 every wavefront. 3899 3900 This is not used for 3901 GFX7-GFX8 since it is the same 3902 value as the second SGPR of 3903 Flat Scratch Init. However, it 3904 may be needed for GFX9-GFX10 which 3905 changes the meaning of the 3906 Flat Scratch Init value. 3907 then Grid Work-Group Count X 1 32-bit count of the number of 3908 (enable_sgpr_grid work-groups in the X dimension 3909 _workgroup_count_X) for the grid being 3910 executed. Computed from the 3911 fields in the kernel dispatch 3912 packet as ((grid_size.x + 3913 workgroup_size.x - 1) / 3914 workgroup_size.x). 3915 then Grid Work-Group Count Y 1 32-bit count of the number of 3916 (enable_sgpr_grid work-groups in the Y dimension 3917 _workgroup_count_Y && for the grid being 3918 less than 16 previous executed. Computed from the 3919 SGPRs) fields in the kernel dispatch 3920 packet as ((grid_size.y + 3921 workgroup_size.y - 1) / 3922 workgroupSize.y). 3923 3924 Only initialized if <16 3925 previous SGPRs initialized. 3926 then Grid Work-Group Count Z 1 32-bit count of the number of 3927 (enable_sgpr_grid work-groups in the Z dimension 3928 _workgroup_count_Z && for the grid being 3929 less than 16 previous executed. Computed from the 3930 SGPRs) fields in the kernel dispatch 3931 packet as ((grid_size.z + 3932 workgroup_size.z - 1) / 3933 workgroupSize.z). 3934 3935 Only initialized if <16 3936 previous SGPRs initialized. 3937 then Work-Group Id X 1 32-bit work-group id in X 3938 (enable_sgpr_workgroup_id dimension of grid for 3939 _X) wavefront. 3940 then Work-Group Id Y 1 32-bit work-group id in Y 3941 (enable_sgpr_workgroup_id dimension of grid for 3942 _Y) wavefront. 3943 then Work-Group Id Z 1 32-bit work-group id in Z 3944 (enable_sgpr_workgroup_id dimension of grid for 3945 _Z) wavefront. 3946 then Work-Group Info 1 {first_wavefront, 14'b0000, 3947 (enable_sgpr_workgroup ordered_append_term[10:0], 3948 _info) threadgroup_size_in_wavefronts[5:0]} 3949 then Scratch Wavefront Offset 1 32-bit byte offset from base 3950 (enable_sgpr_private of scratch base of queue 3951 _segment_wavefront_offset) executing the kernel 3952 dispatch. Must be used as an 3953 offset with Private 3954 segment address when using 3955 Scratch Segment Buffer. It 3956 must be used to set up FLAT 3957 SCRATCH for flat addressing 3958 (see 3959 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 3960 ========== ========================== ====== ============================== 3961 3962The order of the VGPR registers is defined, but the compiler can specify which 3963ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 3964fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 3965for enabled registers are dense starting at VGPR0: the first enabled register is 3966VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 3967VGPR number. 3968 3969VGPR register initial state is defined in 3970:ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 3971 3972 .. table:: VGPR Register Set Up Order 3973 :name: amdgpu-amdhsa-vgpr-register-set-up-order-table 3974 3975 ========== ========================== ====== ============================== 3976 VGPR Order Name Number Description 3977 (kernel descriptor enable of 3978 field) VGPRs 3979 ========== ========================== ====== ============================== 3980 First Work-Item Id X 1 32-bit work-item id in X 3981 (Always initialized) dimension of work-group for 3982 wavefront lane. 3983 then Work-Item Id Y 1 32-bit work-item id in Y 3984 (enable_vgpr_workitem_id dimension of work-group for 3985 > 0) wavefront lane. 3986 then Work-Item Id Z 1 32-bit work-item id in Z 3987 (enable_vgpr_workitem_id dimension of work-group for 3988 > 1) wavefront lane. 3989 ========== ========================== ====== ============================== 3990 3991The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 3992 39931. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 3994 registers. 39952. Work-group Id registers X, Y, Z are set by ADC which supports any 3996 combination including none. 39973. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 3998 its value cannot be included with the flat scratch init value which is per 3999 queue. 40004. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4001 or (X, Y, Z). 4002 4003Flat Scratch register pair are adjacent SGPRs so they can be moved as a 64-bit 4004value to the hardware required SGPRn-3 and SGPRn-4 respectively. 4005 4006The global segment can be accessed either using buffer instructions (GFX6 which 4007has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4008instructions (GFX9-GFX10). 4009 4010If buffer operations are used, then the compiler can generate a V# with the 4011following properties: 4012 4013* base address of 0 4014* no swizzle 4015* ATC: 1 if IOMMU present (such as APU) 4016* ptr64: 1 4017* MTYPE set to support memory coherence that matches the runtime (such as CC for 4018 APU and NC for dGPU). 4019 4020.. _amdgpu-amdhsa-kernel-prolog: 4021 4022Kernel Prolog 4023~~~~~~~~~~~~~ 4024 4025The compiler performs initialization in the kernel prologue depending on the 4026target and information about things like stack usage in the kernel and called 4027functions. Some of this initialization requires the compiler to request certain 4028User and System SGPRs be present in the 4029:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4030:ref:`amdgpu-amdhsa-kernel-descriptor`. 4031 4032.. _amdgpu-amdhsa-kernel-prolog-cfi: 4033 4034CFI 4035+++ 4036 40371. The CFI return address is undefined. 4038 40392. The CFI CFA is defined using an expression which evaluates to a location 4040 description that comprises one memory location description for the 4041 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4042 4043.. _amdgpu-amdhsa-kernel-prolog-m0: 4044 4045M0 4046++ 4047 4048GFX6-GFX8 4049 The M0 register must be initialized with a value at least the total LDS size 4050 if the kernel may access LDS via DS or flat operations. Total LDS size is 4051 available in dispatch packet. For M0, it is also possible to use maximum 4052 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4053 GFX7-GFX8). 4054GFX9-GFX10 4055 The M0 register is not used for range checking LDS accesses and so does not 4056 need to be initialized in the prolog. 4057 4058.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4059 4060Stack Pointer 4061+++++++++++++ 4062 4063If the kernel has function calls it must set up the ABI stack pointer described 4064in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4065SGPR32 to the unswizzled scratch offset of the address past the last local 4066allocation. 4067 4068.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4069 4070Frame Pointer 4071+++++++++++++ 4072 4073If the kernel needs a frame pointer for the reasons defined in 4074``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4075kernel prolog. If a frame pointer is not required then all uses of the frame 4076pointer are replaced with immediate ``0`` offsets. 4077 4078.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4079 4080Flat Scratch 4081++++++++++++ 4082 4083If the kernel or any function it calls may use flat operations to access 4084scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4085(FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4086uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4087:ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4088 4089GFX6 4090 Flat scratch is not supported. 4091 4092GFX7-GFX8 4093 4094 1. The low word of Flat Scratch Init is 32-bit byte offset from 4095 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4096 being managed by SPI for the queue executing the kernel dispatch. This is 4097 the same value used in the Scratch Segment Buffer V# base address. The 4098 prolog must add the value of Scratch Wavefront Offset to get the 4099 wavefront's byte scratch backing memory offset from 4100 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. Since FLAT_SCRATCH_LO is in units of 256 4101 bytes, the offset must be right shifted by 8 before moving into 4102 FLAT_SCRATCH_LO. 4103 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4104 work-items scratch memory usage. This is directly loaded from the kernel 4105 dispatch packet Private Segment Byte Size and rounded up to a multiple of 4106 DWORD. Having CP load it once avoids loading it at the beginning of every 4107 wavefront. The prolog must move it to FLAT_SCRATCH_LO for use as FLAT 4108 SCRATCH SIZE. 4109 4110GFX9-GFX10 4111 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4112 memory being managed by SPI for the queue executing the kernel dispatch. The 4113 prolog must add the value of Scratch Wavefront Offset and moved to the 4114 FLAT_SCRATCH pair for use as the flat scratch base in flat memory 4115 instructions. 4116 4117.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4118 4119Private Segment Buffer 4120++++++++++++++++++++++ 4121 4122A set of four SGPRs beginning at a four-aligned SGPR index are always selected 4123to serve as the scratch V# for the kernel as follows: 4124 4125 - If it is known during instruction selection that there is stack usage, 4126 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4127 optimizations are disabled (``-O0``), if stack objects already exist (for 4128 locals, etc.), or if there are any function calls. 4129 4130 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4131 are reserved for the tentative scratch V#. These will be used if it is 4132 determined that spilling is needed. 4133 4134 - If no use is made of the tentative scratch V#, then it is unreserved, 4135 and the register count is determined ignoring it. 4136 - If use is made of the tentative scratch V#, then its register numbers 4137 are shifted to the first four-aligned SGPR index after the highest one 4138 allocated by the register allocator, and all uses are updated. The 4139 register count includes them in the shifted location. 4140 - In either case, if the processor has the SGPR allocation bug, the 4141 tentative allocation is not shifted or unreserved in order to ensure 4142 the register count is higher to workaround the bug. 4143 4144 .. note:: 4145 4146 This approach of using a tentative scratch V# and shifting the register 4147 numbers if used avoids having to perform register allocation a second 4148 time if the tentative V# is eliminated. This is more efficient and 4149 avoids the problem that the second register allocation may perform 4150 spilling which will fail as there is no longer a scratch V#. 4151 4152When the kernel prolog code is being emitted it is known whether the scratch V# 4153described above is actually used. If it is, the prolog code must set it up by 4154copying the Private Segment Buffer to the scratch V# registers and then adding 4155the Private Segment Wavefront Offset to the queue base address in the V#. The 4156result is a V# with a base address pointing to the beginning of the wavefront 4157scratch backing memory. 4158 4159The Private Segment Buffer is always requested, but the Private Segment 4160Wavefront Offset is only requested if it is used (see 4161:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4162 4163.. _amdgpu-amdhsa-memory-model: 4164 4165Memory Model 4166~~~~~~~~~~~~ 4167 4168This section describes the mapping of the LLVM memory model onto AMDGPU machine 4169code (see :ref:`memmodel`). 4170 4171The AMDGPU backend supports the memory synchronization scopes specified in 4172:ref:`amdgpu-memory-scopes`. 4173 4174The code sequences used to implement the memory model specify the order of 4175instructions that a single thread must execute. The ``s_waitcnt`` and cache 4176management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4177to other memory instructions executed by the same thread. This allows them to be 4178moved earlier or later which can allow them to be combined with other instances 4179of the same instruction, or hoisted/sunk out of loops to improve performance. 4180Only the instructions related to the memory model are given; additional 4181``s_waitcnt`` instructions are required to ensure registers are defined before 4182being used. These may be able to be combined with the memory model ``s_waitcnt`` 4183instructions as described above. 4184 4185The AMDGPU backend supports the following memory models: 4186 4187 HSA Memory Model [HSA]_ 4188 The HSA memory model uses a single happens-before relation for all address 4189 spaces (see :ref:`amdgpu-address-spaces`). 4190 OpenCL Memory Model [OpenCL]_ 4191 The OpenCL memory model which has separate happens-before relations for the 4192 global and local address spaces. Only a fence specifying both global and 4193 local address space, and seq_cst instructions join the relationships. Since 4194 the LLVM ``memfence`` instruction does not allow an address space to be 4195 specified the OpenCL fence has to conservatively assume both local and 4196 global address space was specified. However, optimizations can often be 4197 done to eliminate the additional ``s_waitcnt`` instructions when there are 4198 no intervening memory instructions which access the corresponding address 4199 space. The code sequences in the table indicate what can be omitted for the 4200 OpenCL memory. The target triple environment is used to determine if the 4201 source language is OpenCL (see :ref:`amdgpu-opencl`). 4202 4203``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4204operations. 4205 4206``buffer/global/flat_load/store/atomic`` instructions to global memory are 4207termed vector memory operations. 4208 4209Private address space uses ``buffer_load/store`` using the scratch V# 4210(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 4211is accessing the memory, atomic memory orderings are not meaningful, and all 4212accesses are treated as non-atomic. 4213 4214Constant address space uses ``buffer/global_load`` instructions (or equivalent 4215scalar memory instructions). Since the constant address space contents do not 4216change during the execution of a kernel dispatch it is not legal to perform 4217stores, and atomic memory orderings are not meaningful, and all accesses are 4218treated as non-atomic. 4219 4220A memory synchronization scope wider than work-group is not meaningful for the 4221group (LDS) address space and is treated as work-group. 4222 4223The memory model does not support the region address space which is treated as 4224non-atomic. 4225 4226Acquire memory ordering is not meaningful on store atomic instructions and is 4227treated as non-atomic. 4228 4229Release memory ordering is not meaningful on load atomic instructions and is 4230treated a non-atomic. 4231 4232Acquire-release memory ordering is not meaningful on load or store atomic 4233instructions and is treated as acquire and release respectively. 4234 4235The memory order also adds the single thread optimization constraints defined in 4236table 4237:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 4238 4239 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 4240 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 4241 4242 ============ ============================================================== 4243 LLVM Memory Optimization Constraints 4244 Ordering 4245 ============ ============================================================== 4246 unordered *none* 4247 monotonic *none* 4248 acquire - If a load atomic/atomicrmw then no following load/load 4249 atomic/store/store atomic/atomicrmw/fence instruction can be 4250 moved before the acquire. 4251 - If a fence then same as load atomic, plus no preceding 4252 associated fence-paired-atomic can be moved after the fence. 4253 release - If a store atomic/atomicrmw then no preceding load/load 4254 atomic/store/store atomic/atomicrmw/fence instruction can be 4255 moved after the release. 4256 - If a fence then same as store atomic, plus no following 4257 associated fence-paired-atomic can be moved before the 4258 fence. 4259 acq_rel Same constraints as both acquire and release. 4260 seq_cst - If a load atomic then same constraints as acquire, plus no 4261 preceding sequentially consistent load atomic/store 4262 atomic/atomicrmw/fence instruction can be moved after the 4263 seq_cst. 4264 - If a store atomic then the same constraints as release, plus 4265 no following sequentially consistent load atomic/store 4266 atomic/atomicrmw/fence instruction can be moved before the 4267 seq_cst. 4268 - If an atomicrmw/fence then same constraints as acq_rel. 4269 ============ ============================================================== 4270 4271The code sequences used to implement the memory model are defined in the 4272following sections: 4273 4274* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 4275* :ref:`amdgpu-amdhsa-memory-model-gfx10` 4276 4277.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 4278 4279Memory Model GFX6-GFX9 4280++++++++++++++++++++++ 4281 4282For GFX6-GFX9: 4283 4284* Each agent has multiple shader arrays (SA). 4285* Each SA has multiple compute units (CU). 4286* Each CU has multiple SIMDs that execute wavefronts. 4287* The wavefronts for a single work-group are executed in the same CU but may be 4288 executed by different SIMDs. 4289* Each CU has a single LDS memory shared by the wavefronts of the work-groups 4290 executing on it. 4291* All LDS operations of a CU are performed as wavefront wide operations in a 4292 global order and involve no caching. Completion is reported to a wavefront in 4293 execution order. 4294* The LDS memory has multiple request queues shared by the SIMDs of a 4295 CU. Therefore, the LDS operations performed by different wavefronts of a 4296 work-group can be reordered relative to each other, which can result in 4297 reordering the visibility of vector memory operations with respect to LDS 4298 operations of other wavefronts in the same work-group. A ``s_waitcnt 4299 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 4300 vector memory operations between wavefronts of a work-group, but not between 4301 operations performed by the same wavefront. 4302* The vector memory operations are performed as wavefront wide operations and 4303 completion is reported to a wavefront in execution order. The exception is 4304 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 4305 vector memory order if they access LDS memory, and out of LDS operation order 4306 if they access global memory. 4307* The vector memory operations access a single vector L1 cache shared by all 4308 SIMDs a CU. Therefore, no special action is required for coherence between the 4309 lanes of a single wavefront, or for coherence between wavefronts in the same 4310 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 4311 wavefronts executing in different work-groups as they may be executing on 4312 different CUs. 4313* The scalar memory operations access a scalar L1 cache shared by all wavefronts 4314 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 4315 scalar operations are used in a restricted way so do not impact the memory 4316 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 4317* The vector and scalar memory operations use an L2 cache shared by all CUs on 4318 the same agent. 4319* The L2 cache has independent channels to service disjoint ranges of virtual 4320 addresses. 4321* Each CU has a separate request queue per channel. Therefore, the vector and 4322 scalar memory operations performed by wavefronts executing in different 4323 work-groups (which may be executing on different CUs) of an agent can be 4324 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 4325 ensure synchronization between vector memory operations of different CUs. It 4326 ensures a previous vector memory operation has completed before executing a 4327 subsequent vector memory or LDS operation and so can be used to meet the 4328 requirements of acquire and release. 4329* The L2 cache can be kept coherent with other agents on some targets, or ranges 4330 of virtual addresses can be set up to bypass it to ensure system coherence. 4331 4332Scalar memory operations are only used to access memory that is proven to not 4333change during the execution of the kernel dispatch. This includes constant 4334address space and global address space for program scope ``const`` variables. 4335Therefore, the kernel machine code does not have to maintain the scalar cache to 4336ensure it is coherent with the vector caches. The scalar and vector caches are 4337invalidated between kernel dispatches by CP since constant address space data 4338may change between kernel dispatch executions. See 4339:ref:`amdgpu-amdhsa-memory-spaces`. 4340 4341The one exception is if scalar writes are used to spill SGPR registers. In this 4342case the AMDGPU backend ensures the memory location used to spill is never 4343accessed by vector memory operations at the same time. If scalar writes are used 4344then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 4345return since the locations may be used for vector memory instructions by a 4346future wavefront that uses the same scratch area, or a function call that 4347creates a frame at the same address, respectively. There is no need for a 4348``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 4349 4350For kernarg backing memory: 4351 4352* CP invalidates the L1 cache at the start of each kernel dispatch. 4353* On dGPU the kernarg backing memory is allocated in host memory accessed as 4354 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 4355 causes it to be treated as non-volatile and so is not invalidated by 4356 ``*_vol``. 4357* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 4358 and so the L2 cache will be coherent with the CPU and other agents. 4359 4360Scratch backing memory (which is used for the private address space) is accessed 4361with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 4362only accessed by a single thread, and is always write-before-read, there is 4363never a need to invalidate these entries from the L1 cache. Hence all cache 4364invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 4365 4366The code sequences used to implement the memory model for GFX6-GFX9 are defined 4367in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 4368 4369 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 4370 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 4371 4372 ============ ============ ============== ========== ================================ 4373 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 4374 Ordering Sync Scope Address GFX6-9 4375 Space 4376 ============ ============ ============== ========== ================================ 4377 **Non-Atomic** 4378 ------------------------------------------------------------------------------------ 4379 load *none* *none* - global - !volatile & !nontemporal 4380 - generic 4381 - private 1. buffer/global/flat_load 4382 - constant 4383 - volatile & !nontemporal 4384 4385 1. buffer/global/flat_load 4386 glc=1 4387 4388 - nontemporal 4389 4390 1. buffer/global/flat_load 4391 glc=1 slc=1 4392 4393 load *none* *none* - local 1. ds_load 4394 store *none* *none* - global - !nontemporal 4395 - generic 4396 - private 1. buffer/global/flat_store 4397 - constant 4398 - nontemporal 4399 4400 1. buffer/global/flat_store 4401 glc=1 slc=1 4402 4403 store *none* *none* - local 1. ds_store 4404 **Unordered Atomic** 4405 ------------------------------------------------------------------------------------ 4406 load atomic unordered *any* *any* *Same as non-atomic*. 4407 store atomic unordered *any* *any* *Same as non-atomic*. 4408 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 4409 **Monotonic Atomic** 4410 ------------------------------------------------------------------------------------ 4411 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 4412 - wavefront - local 4413 - workgroup - generic 4414 load atomic monotonic - agent - global 1. buffer/global/flat_load 4415 - system - generic glc=1 4416 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 4417 - wavefront - generic 4418 - workgroup 4419 - agent 4420 - system 4421 store atomic monotonic - singlethread - local 1. ds_store 4422 - wavefront 4423 - workgroup 4424 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 4425 - wavefront - generic 4426 - workgroup 4427 - agent 4428 - system 4429 atomicrmw monotonic - singlethread - local 1. ds_atomic 4430 - wavefront 4431 - workgroup 4432 **Acquire Atomic** 4433 ------------------------------------------------------------------------------------ 4434 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 4435 - wavefront - local 4436 - generic 4437 load atomic acquire - workgroup - global 1. buffer/global_load 4438 load atomic acquire - workgroup - local 1. ds/flat_load 4439 - generic 2. s_waitcnt lgkmcnt(0) 4440 4441 - If OpenCL, omit. 4442 - Must happen before 4443 any following 4444 global/generic 4445 load/load 4446 atomic/store/store 4447 atomic/atomicrmw. 4448 - Ensures any 4449 following global 4450 data read is no 4451 older than a local load 4452 atomic value being 4453 acquired. 4454 4455 load atomic acquire - agent - global 1. buffer/global_load 4456 - system glc=1 4457 2. s_waitcnt vmcnt(0) 4458 4459 - Must happen before 4460 following 4461 buffer_wbinvl1_vol. 4462 - Ensures the load 4463 has completed 4464 before invalidating 4465 the cache. 4466 4467 3. buffer_wbinvl1_vol 4468 4469 - Must happen before 4470 any following 4471 global/generic 4472 load/load 4473 atomic/atomicrmw. 4474 - Ensures that 4475 following 4476 loads will not see 4477 stale global data. 4478 4479 load atomic acquire - agent - generic 1. flat_load glc=1 4480 - system 2. s_waitcnt vmcnt(0) & 4481 lgkmcnt(0) 4482 4483 - If OpenCL omit 4484 lgkmcnt(0). 4485 - Must happen before 4486 following 4487 buffer_wbinvl1_vol. 4488 - Ensures the flat_load 4489 has completed 4490 before invalidating 4491 the cache. 4492 4493 3. buffer_wbinvl1_vol 4494 4495 - Must happen before 4496 any following 4497 global/generic 4498 load/load 4499 atomic/atomicrmw. 4500 - Ensures that 4501 following loads 4502 will not see stale 4503 global data. 4504 4505 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 4506 - wavefront - local 4507 - generic 4508 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 4509 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 4510 - generic 2. s_waitcnt lgkmcnt(0) 4511 4512 - If OpenCL, omit. 4513 - Must happen before 4514 any following 4515 global/generic 4516 load/load 4517 atomic/store/store 4518 atomic/atomicrmw. 4519 - Ensures any 4520 following global 4521 data read is no 4522 older than a local 4523 atomicrmw value 4524 being acquired. 4525 4526 atomicrmw acquire - agent - global 1. buffer/global_atomic 4527 - system 2. s_waitcnt vmcnt(0) 4528 4529 - Must happen before 4530 following 4531 buffer_wbinvl1_vol. 4532 - Ensures the 4533 atomicrmw has 4534 completed before 4535 invalidating the 4536 cache. 4537 4538 3. buffer_wbinvl1_vol 4539 4540 - Must happen before 4541 any following 4542 global/generic 4543 load/load 4544 atomic/atomicrmw. 4545 - Ensures that 4546 following loads 4547 will not see stale 4548 global data. 4549 4550 atomicrmw acquire - agent - generic 1. flat_atomic 4551 - system 2. s_waitcnt vmcnt(0) & 4552 lgkmcnt(0) 4553 4554 - If OpenCL, omit 4555 lgkmcnt(0). 4556 - Must happen before 4557 following 4558 buffer_wbinvl1_vol. 4559 - Ensures the 4560 atomicrmw has 4561 completed before 4562 invalidating the 4563 cache. 4564 4565 3. buffer_wbinvl1_vol 4566 4567 - Must happen before 4568 any following 4569 global/generic 4570 load/load 4571 atomic/atomicrmw. 4572 - Ensures that 4573 following loads 4574 will not see stale 4575 global data. 4576 4577 fence acquire - singlethread *none* *none* 4578 - wavefront 4579 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 4580 4581 - If OpenCL and 4582 address space is 4583 not generic, omit. 4584 - However, since LLVM 4585 currently has no 4586 address space on 4587 the fence need to 4588 conservatively 4589 always generate. If 4590 fence had an 4591 address space then 4592 set to address 4593 space of OpenCL 4594 fence flag, or to 4595 generic if both 4596 local and global 4597 flags are 4598 specified. 4599 - Must happen after 4600 any preceding 4601 local/generic load 4602 atomic/atomicrmw 4603 with an equal or 4604 wider sync scope 4605 and memory ordering 4606 stronger than 4607 unordered (this is 4608 termed the 4609 fence-paired-atomic). 4610 - Must happen before 4611 any following 4612 global/generic 4613 load/load 4614 atomic/store/store 4615 atomic/atomicrmw. 4616 - Ensures any 4617 following global 4618 data read is no 4619 older than the 4620 value read by the 4621 fence-paired-atomic. 4622 4623 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 4624 - system vmcnt(0) 4625 4626 - If OpenCL and 4627 address space is 4628 not generic, omit 4629 lgkmcnt(0). 4630 - However, since LLVM 4631 currently has no 4632 address space on 4633 the fence need to 4634 conservatively 4635 always generate 4636 (see comment for 4637 previous fence). 4638 - Could be split into 4639 separate s_waitcnt 4640 vmcnt(0) and 4641 s_waitcnt 4642 lgkmcnt(0) to allow 4643 them to be 4644 independently moved 4645 according to the 4646 following rules. 4647 - s_waitcnt vmcnt(0) 4648 must happen after 4649 any preceding 4650 global/generic load 4651 atomic/atomicrmw 4652 with an equal or 4653 wider sync scope 4654 and memory ordering 4655 stronger than 4656 unordered (this is 4657 termed the 4658 fence-paired-atomic). 4659 - s_waitcnt lgkmcnt(0) 4660 must happen after 4661 any preceding 4662 local/generic load 4663 atomic/atomicrmw 4664 with an equal or 4665 wider sync scope 4666 and memory ordering 4667 stronger than 4668 unordered (this is 4669 termed the 4670 fence-paired-atomic). 4671 - Must happen before 4672 the following 4673 buffer_wbinvl1_vol. 4674 - Ensures that the 4675 fence-paired atomic 4676 has completed 4677 before invalidating 4678 the 4679 cache. Therefore 4680 any following 4681 locations read must 4682 be no older than 4683 the value read by 4684 the 4685 fence-paired-atomic. 4686 4687 2. buffer_wbinvl1_vol 4688 4689 - Must happen before any 4690 following global/generic 4691 load/load 4692 atomic/store/store 4693 atomic/atomicrmw. 4694 - Ensures that 4695 following loads 4696 will not see stale 4697 global data. 4698 4699 **Release Atomic** 4700 ------------------------------------------------------------------------------------ 4701 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 4702 - wavefront - local 4703 - generic 4704 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 4705 - generic 4706 - If OpenCL, omit. 4707 - Must happen after 4708 any preceding 4709 local/generic 4710 load/store/load 4711 atomic/store 4712 atomic/atomicrmw. 4713 - Must happen before 4714 the following 4715 store. 4716 - Ensures that all 4717 memory operations 4718 to local have 4719 completed before 4720 performing the 4721 store that is being 4722 released. 4723 4724 2. buffer/global/flat_store 4725 store atomic release - workgroup - local 1. ds_store 4726 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 4727 - system - generic vmcnt(0) 4728 4729 - If OpenCL and 4730 address space is 4731 not generic, omit 4732 lgkmcnt(0). 4733 - Could be split into 4734 separate s_waitcnt 4735 vmcnt(0) and 4736 s_waitcnt 4737 lgkmcnt(0) to allow 4738 them to be 4739 independently moved 4740 according to the 4741 following rules. 4742 - s_waitcnt vmcnt(0) 4743 must happen after 4744 any preceding 4745 global/generic 4746 load/store/load 4747 atomic/store 4748 atomic/atomicrmw. 4749 - s_waitcnt lgkmcnt(0) 4750 must happen after 4751 any preceding 4752 local/generic 4753 load/store/load 4754 atomic/store 4755 atomic/atomicrmw. 4756 - Must happen before 4757 the following 4758 store. 4759 - Ensures that all 4760 memory operations 4761 to memory have 4762 completed before 4763 performing the 4764 store that is being 4765 released. 4766 4767 2. buffer/global/flat_store 4768 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 4769 - wavefront - local 4770 - generic 4771 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 4772 - generic 4773 - If OpenCL, omit. 4774 - Must happen after 4775 any preceding 4776 local/generic 4777 load/store/load 4778 atomic/store 4779 atomic/atomicrmw. 4780 - Must happen before 4781 the following 4782 atomicrmw. 4783 - Ensures that all 4784 memory operations 4785 to local have 4786 completed before 4787 performing the 4788 atomicrmw that is 4789 being released. 4790 4791 2. buffer/global/flat_atomic 4792 atomicrmw release - workgroup - local 1. ds_atomic 4793 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 4794 - system - generic vmcnt(0) 4795 4796 - If OpenCL, omit 4797 lgkmcnt(0). 4798 - Could be split into 4799 separate s_waitcnt 4800 vmcnt(0) and 4801 s_waitcnt 4802 lgkmcnt(0) to allow 4803 them to be 4804 independently moved 4805 according to the 4806 following rules. 4807 - s_waitcnt vmcnt(0) 4808 must happen after 4809 any preceding 4810 global/generic 4811 load/store/load 4812 atomic/store 4813 atomic/atomicrmw. 4814 - s_waitcnt lgkmcnt(0) 4815 must happen after 4816 any preceding 4817 local/generic 4818 load/store/load 4819 atomic/store 4820 atomic/atomicrmw. 4821 - Must happen before 4822 the following 4823 atomicrmw. 4824 - Ensures that all 4825 memory operations 4826 to global and local 4827 have completed 4828 before performing 4829 the atomicrmw that 4830 is being released. 4831 4832 2. buffer/global/flat_atomic 4833 fence release - singlethread *none* *none* 4834 - wavefront 4835 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 4836 4837 - If OpenCL and 4838 address space is 4839 not generic, omit. 4840 - However, since LLVM 4841 currently has no 4842 address space on 4843 the fence need to 4844 conservatively 4845 always generate. If 4846 fence had an 4847 address space then 4848 set to address 4849 space of OpenCL 4850 fence flag, or to 4851 generic if both 4852 local and global 4853 flags are 4854 specified. 4855 - Must happen after 4856 any preceding 4857 local/generic 4858 load/load 4859 atomic/store/store 4860 atomic/atomicrmw. 4861 - Must happen before 4862 any following store 4863 atomic/atomicrmw 4864 with an equal or 4865 wider sync scope 4866 and memory ordering 4867 stronger than 4868 unordered (this is 4869 termed the 4870 fence-paired-atomic). 4871 - Ensures that all 4872 memory operations 4873 to local have 4874 completed before 4875 performing the 4876 following 4877 fence-paired-atomic. 4878 4879 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 4880 - system vmcnt(0) 4881 4882 - If OpenCL and 4883 address space is 4884 not generic, omit 4885 lgkmcnt(0). 4886 - If OpenCL and 4887 address space is 4888 local, omit 4889 vmcnt(0). 4890 - However, since LLVM 4891 currently has no 4892 address space on 4893 the fence need to 4894 conservatively 4895 always generate. If 4896 fence had an 4897 address space then 4898 set to address 4899 space of OpenCL 4900 fence flag, or to 4901 generic if both 4902 local and global 4903 flags are 4904 specified. 4905 - Could be split into 4906 separate s_waitcnt 4907 vmcnt(0) and 4908 s_waitcnt 4909 lgkmcnt(0) to allow 4910 them to be 4911 independently moved 4912 according to the 4913 following rules. 4914 - s_waitcnt vmcnt(0) 4915 must happen after 4916 any preceding 4917 global/generic 4918 load/store/load 4919 atomic/store 4920 atomic/atomicrmw. 4921 - s_waitcnt lgkmcnt(0) 4922 must happen after 4923 any preceding 4924 local/generic 4925 load/store/load 4926 atomic/store 4927 atomic/atomicrmw. 4928 - Must happen before 4929 any following store 4930 atomic/atomicrmw 4931 with an equal or 4932 wider sync scope 4933 and memory ordering 4934 stronger than 4935 unordered (this is 4936 termed the 4937 fence-paired-atomic). 4938 - Ensures that all 4939 memory operations 4940 have 4941 completed before 4942 performing the 4943 following 4944 fence-paired-atomic. 4945 4946 **Acquire-Release Atomic** 4947 ------------------------------------------------------------------------------------ 4948 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 4949 - wavefront - local 4950 - generic 4951 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 4952 4953 - If OpenCL, omit. 4954 - Must happen after 4955 any preceding 4956 local/generic 4957 load/store/load 4958 atomic/store 4959 atomic/atomicrmw. 4960 - Must happen before 4961 the following 4962 atomicrmw. 4963 - Ensures that all 4964 memory operations 4965 to local have 4966 completed before 4967 performing the 4968 atomicrmw that is 4969 being released. 4970 4971 2. buffer/global_atomic 4972 4973 atomicrmw acq_rel - workgroup - local 1. ds_atomic 4974 2. s_waitcnt lgkmcnt(0) 4975 4976 - If OpenCL, omit. 4977 - Must happen before 4978 any following 4979 global/generic 4980 load/load 4981 atomic/store/store 4982 atomic/atomicrmw. 4983 - Ensures any 4984 following global 4985 data read is no 4986 older than the local load 4987 atomic value being 4988 acquired. 4989 4990 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 4991 4992 - If OpenCL, omit. 4993 - Must happen after 4994 any preceding 4995 local/generic 4996 load/store/load 4997 atomic/store 4998 atomic/atomicrmw. 4999 - Must happen before 5000 the following 5001 atomicrmw. 5002 - Ensures that all 5003 memory operations 5004 to local have 5005 completed before 5006 performing the 5007 atomicrmw that is 5008 being released. 5009 5010 2. flat_atomic 5011 3. s_waitcnt lgkmcnt(0) 5012 5013 - If OpenCL, omit. 5014 - Must happen before 5015 any following 5016 global/generic 5017 load/load 5018 atomic/store/store 5019 atomic/atomicrmw. 5020 - Ensures any 5021 following global 5022 data read is no 5023 older than a local load 5024 atomic value being 5025 acquired. 5026 5027 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5028 - system vmcnt(0) 5029 5030 - If OpenCL, omit 5031 lgkmcnt(0). 5032 - Could be split into 5033 separate s_waitcnt 5034 vmcnt(0) and 5035 s_waitcnt 5036 lgkmcnt(0) to allow 5037 them to be 5038 independently moved 5039 according to the 5040 following rules. 5041 - s_waitcnt vmcnt(0) 5042 must happen after 5043 any preceding 5044 global/generic 5045 load/store/load 5046 atomic/store 5047 atomic/atomicrmw. 5048 - s_waitcnt lgkmcnt(0) 5049 must happen after 5050 any preceding 5051 local/generic 5052 load/store/load 5053 atomic/store 5054 atomic/atomicrmw. 5055 - Must happen before 5056 the following 5057 atomicrmw. 5058 - Ensures that all 5059 memory operations 5060 to global have 5061 completed before 5062 performing the 5063 atomicrmw that is 5064 being released. 5065 5066 2. buffer/global_atomic 5067 3. s_waitcnt vmcnt(0) 5068 5069 - Must happen before 5070 following 5071 buffer_wbinvl1_vol. 5072 - Ensures the 5073 atomicrmw has 5074 completed before 5075 invalidating the 5076 cache. 5077 5078 4. buffer_wbinvl1_vol 5079 5080 - Must happen before 5081 any following 5082 global/generic 5083 load/load 5084 atomic/atomicrmw. 5085 - Ensures that 5086 following loads 5087 will not see stale 5088 global data. 5089 5090 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5091 - system vmcnt(0) 5092 5093 - If OpenCL, omit 5094 lgkmcnt(0). 5095 - Could be split into 5096 separate s_waitcnt 5097 vmcnt(0) and 5098 s_waitcnt 5099 lgkmcnt(0) to allow 5100 them to be 5101 independently moved 5102 according to the 5103 following rules. 5104 - s_waitcnt vmcnt(0) 5105 must happen after 5106 any preceding 5107 global/generic 5108 load/store/load 5109 atomic/store 5110 atomic/atomicrmw. 5111 - s_waitcnt lgkmcnt(0) 5112 must happen after 5113 any preceding 5114 local/generic 5115 load/store/load 5116 atomic/store 5117 atomic/atomicrmw. 5118 - Must happen before 5119 the following 5120 atomicrmw. 5121 - Ensures that all 5122 memory operations 5123 to global have 5124 completed before 5125 performing the 5126 atomicrmw that is 5127 being released. 5128 5129 2. flat_atomic 5130 3. s_waitcnt vmcnt(0) & 5131 lgkmcnt(0) 5132 5133 - If OpenCL, omit 5134 lgkmcnt(0). 5135 - Must happen before 5136 following 5137 buffer_wbinvl1_vol. 5138 - Ensures the 5139 atomicrmw has 5140 completed before 5141 invalidating the 5142 cache. 5143 5144 4. buffer_wbinvl1_vol 5145 5146 - Must happen before 5147 any following 5148 global/generic 5149 load/load 5150 atomic/atomicrmw. 5151 - Ensures that 5152 following loads 5153 will not see stale 5154 global data. 5155 5156 fence acq_rel - singlethread *none* *none* 5157 - wavefront 5158 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5159 5160 - If OpenCL and 5161 address space is 5162 not generic, omit. 5163 - However, 5164 since LLVM 5165 currently has no 5166 address space on 5167 the fence need to 5168 conservatively 5169 always generate 5170 (see comment for 5171 previous fence). 5172 - Must happen after 5173 any preceding 5174 local/generic 5175 load/load 5176 atomic/store/store 5177 atomic/atomicrmw. 5178 - Must happen before 5179 any following 5180 global/generic 5181 load/load 5182 atomic/store/store 5183 atomic/atomicrmw. 5184 - Ensures that all 5185 memory operations 5186 to local have 5187 completed before 5188 performing any 5189 following global 5190 memory operations. 5191 - Ensures that the 5192 preceding 5193 local/generic load 5194 atomic/atomicrmw 5195 with an equal or 5196 wider sync scope 5197 and memory ordering 5198 stronger than 5199 unordered (this is 5200 termed the 5201 acquire-fence-paired-atomic) 5202 has completed 5203 before following 5204 global memory 5205 operations. This 5206 satisfies the 5207 requirements of 5208 acquire. 5209 - Ensures that all 5210 previous memory 5211 operations have 5212 completed before a 5213 following 5214 local/generic store 5215 atomic/atomicrmw 5216 with an equal or 5217 wider sync scope 5218 and memory ordering 5219 stronger than 5220 unordered (this is 5221 termed the 5222 release-fence-paired-atomic). 5223 This satisfies the 5224 requirements of 5225 release. 5226 5227 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 5228 - system vmcnt(0) 5229 5230 - If OpenCL and 5231 address space is 5232 not generic, omit 5233 lgkmcnt(0). 5234 - However, since LLVM 5235 currently has no 5236 address space on 5237 the fence need to 5238 conservatively 5239 always generate 5240 (see comment for 5241 previous fence). 5242 - Could be split into 5243 separate s_waitcnt 5244 vmcnt(0) and 5245 s_waitcnt 5246 lgkmcnt(0) to allow 5247 them to be 5248 independently moved 5249 according to the 5250 following rules. 5251 - s_waitcnt vmcnt(0) 5252 must happen after 5253 any preceding 5254 global/generic 5255 load/store/load 5256 atomic/store 5257 atomic/atomicrmw. 5258 - s_waitcnt lgkmcnt(0) 5259 must happen after 5260 any preceding 5261 local/generic 5262 load/store/load 5263 atomic/store 5264 atomic/atomicrmw. 5265 - Must happen before 5266 the following 5267 buffer_wbinvl1_vol. 5268 - Ensures that the 5269 preceding 5270 global/local/generic 5271 load 5272 atomic/atomicrmw 5273 with an equal or 5274 wider sync scope 5275 and memory ordering 5276 stronger than 5277 unordered (this is 5278 termed the 5279 acquire-fence-paired-atomic) 5280 has completed 5281 before invalidating 5282 the cache. This 5283 satisfies the 5284 requirements of 5285 acquire. 5286 - Ensures that all 5287 previous memory 5288 operations have 5289 completed before a 5290 following 5291 global/local/generic 5292 store 5293 atomic/atomicrmw 5294 with an equal or 5295 wider sync scope 5296 and memory ordering 5297 stronger than 5298 unordered (this is 5299 termed the 5300 release-fence-paired-atomic). 5301 This satisfies the 5302 requirements of 5303 release. 5304 5305 2. buffer_wbinvl1_vol 5306 5307 - Must happen before 5308 any following 5309 global/generic 5310 load/load 5311 atomic/store/store 5312 atomic/atomicrmw. 5313 - Ensures that 5314 following loads 5315 will not see stale 5316 global data. This 5317 satisfies the 5318 requirements of 5319 acquire. 5320 5321 **Sequential Consistent Atomic** 5322 ------------------------------------------------------------------------------------ 5323 load atomic seq_cst - singlethread - global *Same as corresponding 5324 - wavefront - local load atomic acquire, 5325 - generic except must generated 5326 all instructions even 5327 for OpenCL.* 5328 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 5329 - generic 5330 5331 - Must 5332 happen after 5333 preceding 5334 local/generic load 5335 atomic/store 5336 atomic/atomicrmw 5337 with memory 5338 ordering of seq_cst 5339 and with equal or 5340 wider sync scope. 5341 (Note that seq_cst 5342 fences have their 5343 own s_waitcnt 5344 lgkmcnt(0) and so do 5345 not need to be 5346 considered.) 5347 - Ensures any 5348 preceding 5349 sequential 5350 consistent local 5351 memory instructions 5352 have completed 5353 before executing 5354 this sequentially 5355 consistent 5356 instruction. This 5357 prevents reordering 5358 a seq_cst store 5359 followed by a 5360 seq_cst load. (Note 5361 that seq_cst is 5362 stronger than 5363 acquire/release as 5364 the reordering of 5365 load acquire 5366 followed by a store 5367 release is 5368 prevented by the 5369 s_waitcnt of 5370 the release, but 5371 there is nothing 5372 preventing a store 5373 release followed by 5374 load acquire from 5375 completing out of 5376 order. The s_waitcnt 5377 could be placed after 5378 seq_store or before 5379 the seq_load. We 5380 choose the load to 5381 make the s_waitcnt be 5382 as late as possible 5383 so that the store 5384 may have already 5385 completed.) 5386 5387 2. *Following 5388 instructions same as 5389 corresponding load 5390 atomic acquire, 5391 except must generated 5392 all instructions even 5393 for OpenCL.* 5394 load atomic seq_cst - workgroup - local *Same as corresponding 5395 load atomic acquire, 5396 except must generated 5397 all instructions even 5398 for OpenCL.* 5399 5400 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 5401 - system - generic vmcnt(0) 5402 5403 - Could be split into 5404 separate s_waitcnt 5405 vmcnt(0) 5406 and s_waitcnt 5407 lgkmcnt(0) to allow 5408 them to be 5409 independently moved 5410 according to the 5411 following rules. 5412 - s_waitcnt lgkmcnt(0) 5413 must happen after 5414 preceding 5415 global/generic load 5416 atomic/store 5417 atomic/atomicrmw 5418 with memory 5419 ordering of seq_cst 5420 and with equal or 5421 wider sync scope. 5422 (Note that seq_cst 5423 fences have their 5424 own s_waitcnt 5425 lgkmcnt(0) and so do 5426 not need to be 5427 considered.) 5428 - s_waitcnt vmcnt(0) 5429 must happen after 5430 preceding 5431 global/generic load 5432 atomic/store 5433 atomic/atomicrmw 5434 with memory 5435 ordering of seq_cst 5436 and with equal or 5437 wider sync scope. 5438 (Note that seq_cst 5439 fences have their 5440 own s_waitcnt 5441 vmcnt(0) and so do 5442 not need to be 5443 considered.) 5444 - Ensures any 5445 preceding 5446 sequential 5447 consistent global 5448 memory instructions 5449 have completed 5450 before executing 5451 this sequentially 5452 consistent 5453 instruction. This 5454 prevents reordering 5455 a seq_cst store 5456 followed by a 5457 seq_cst load. (Note 5458 that seq_cst is 5459 stronger than 5460 acquire/release as 5461 the reordering of 5462 load acquire 5463 followed by a store 5464 release is 5465 prevented by the 5466 s_waitcnt of 5467 the release, but 5468 there is nothing 5469 preventing a store 5470 release followed by 5471 load acquire from 5472 completing out of 5473 order. The s_waitcnt 5474 could be placed after 5475 seq_store or before 5476 the seq_load. We 5477 choose the load to 5478 make the s_waitcnt be 5479 as late as possible 5480 so that the store 5481 may have already 5482 completed.) 5483 5484 2. *Following 5485 instructions same as 5486 corresponding load 5487 atomic acquire, 5488 except must generated 5489 all instructions even 5490 for OpenCL.* 5491 store atomic seq_cst - singlethread - global *Same as corresponding 5492 - wavefront - local store atomic release, 5493 - workgroup - generic except must generated 5494 - agent all instructions even 5495 - system for OpenCL.* 5496 atomicrmw seq_cst - singlethread - global *Same as corresponding 5497 - wavefront - local atomicrmw acq_rel, 5498 - workgroup - generic except must generated 5499 - agent all instructions even 5500 - system for OpenCL.* 5501 fence seq_cst - singlethread *none* *Same as corresponding 5502 - wavefront fence acq_rel, 5503 - workgroup except must generated 5504 - agent all instructions even 5505 - system for OpenCL.* 5506 ============ ============ ============== ========== ================================ 5507 5508.. _amdgpu-amdhsa-memory-model-gfx10: 5509 5510Memory Model GFX10 5511++++++++++++++++++ 5512 5513For GFX10: 5514 5515* Each agent has multiple shader arrays (SA). 5516* Each SA has multiple work-group processors (WGP). 5517* Each WGP has multiple compute units (CU). 5518* Each CU has multiple SIMDs that execute wavefronts. 5519* The wavefronts for a single work-group are executed in the same 5520 WGP. In CU wavefront execution mode the wavefronts may be executed by 5521 different SIMDs in the same CU. In WGP wavefront execution mode the 5522 wavefronts may be executed by different SIMDs in different CUs in the same 5523 WGP. 5524* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 5525 executing on it. 5526* All LDS operations of a WGP are performed as wavefront wide operations in a 5527 global order and involve no caching. Completion is reported to a wavefront in 5528 execution order. 5529* The LDS memory has multiple request queues shared by the SIMDs of a 5530 WGP. Therefore, the LDS operations performed by different wavefronts of a 5531 work-group can be reordered relative to each other, which can result in 5532 reordering the visibility of vector memory operations with respect to LDS 5533 operations of other wavefronts in the same work-group. A ``s_waitcnt 5534 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5535 vector memory operations between wavefronts of a work-group, but not between 5536 operations performed by the same wavefront. 5537* The vector memory operations are performed as wavefront wide operations. 5538 Completion of load/store/sample operations are reported to a wavefront in 5539 execution order of other load/store/sample operations performed by that 5540 wavefront. 5541* The vector memory operations access a vector L0 cache. There is a single L0 5542 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 5543 special action is required for coherence between the lanes of a single 5544 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 5545 wavefronts executing in the same work-group as they may be executing on SIMDs 5546 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 5547 required for coherence between wavefronts executing in different work-groups 5548 as they may be executing on different WGPs. 5549* The scalar memory operations access a scalar L0 cache shared by all wavefronts 5550 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 5551 operations are used in a restricted way so do not impact the memory model. See 5552 :ref:`amdgpu-amdhsa-memory-spaces`. 5553* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 5554 the same SA. Therefore, no special action is required for coherence between 5555 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 5556 required for coherence between wavefronts executing in different work-groups 5557 as they may be executing on different SAs that access different L1s. 5558* The L1 caches have independent quadrants to service disjoint ranges of virtual 5559 addresses. 5560* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 5561 vector and scalar memory operations performed by different wavefronts, whether 5562 executing in the same or different work-groups (which may be executing on 5563 different CUs accessing different L0s), can be reordered relative to each 5564 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 5565 synchronization between vector memory operations of different wavefronts. It 5566 ensures a previous vector memory operation has completed before executing a 5567 subsequent vector memory or LDS operation and so can be used to meet the 5568 requirements of acquire, release and sequential consistency. 5569* The L1 caches use an L2 cache shared by all SAs on the same agent. 5570* The L2 cache has independent channels to service disjoint ranges of virtual 5571 addresses. 5572* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 5573 quadrant has a separate request queue per L2 channel. Therefore, the vector 5574 and scalar memory operations performed by wavefronts executing in different 5575 work-groups (which may be executing on different SAs) of an agent can be 5576 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 5577 required to ensure synchronization between vector memory operations of 5578 different SAs. It ensures a previous vector memory operation has completed 5579 before executing a subsequent vector memory and so can be used to meet the 5580 requirements of acquire, release and sequential consistency. 5581* The L2 cache can be kept coherent with other agents on some targets, or ranges 5582 of virtual addresses can be set up to bypass it to ensure system coherence. 5583 5584Scalar memory operations are only used to access memory that is proven to not 5585change during the execution of the kernel dispatch. This includes constant 5586address space and global address space for program scope ``const`` variables. 5587Therefore, the kernel machine code does not have to maintain the scalar cache to 5588ensure it is coherent with the vector caches. The scalar and vector caches are 5589invalidated between kernel dispatches by CP since constant address space data 5590may change between kernel dispatch executions. See 5591:ref:`amdgpu-amdhsa-memory-spaces`. 5592 5593The one exception is if scalar writes are used to spill SGPR registers. In this 5594case the AMDGPU backend ensures the memory location used to spill is never 5595accessed by vector memory operations at the same time. If scalar writes are used 5596then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5597return since the locations may be used for vector memory instructions by a 5598future wavefront that uses the same scratch area, or a function call that 5599creates a frame at the same address, respectively. There is no need for a 5600``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5601 5602For kernarg backing memory: 5603 5604* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 5605* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 5606 needing to invalidate the L2 cache. 5607* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 5608 so the L2 cache will be coherent with the CPU and other agents. 5609 5610Scratch backing memory (which is used for the private address space) is accessed 5611with MTYPE NC (non-coherent). Since the private address space is only accessed 5612by a single thread, and is always write-before-read, there is never a need to 5613invalidate these entries from the L0 or L1 caches. 5614 5615Wavefronts are executed in native mode with in-order reporting of loads and 5616sample instructions. In this mode vmcnt reports completion of load, atomic with 5617return and sample instructions in order, and the vscnt reports the completion of 5618store and atomic without return in order. See ``MEM_ORDERED`` field in 5619:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 5620 5621Wavefronts can be executed in WGP or CU wavefront execution mode: 5622 5623* In WGP wavefront execution mode the wavefronts of a work-group are executed 5624 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 5625 CU L0 caches is required for work-group synchronization. Also accesses to L1 5626 at work-group scope need to be explicitly ordered as the accesses from 5627 different CUs are not ordered. 5628* In CU wavefront execution mode the wavefronts of a work-group are executed on 5629 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 5630 the work-group access the same L0 which in turn ensures L1 accesses are 5631 ordered and so do not require explicit management of the caches for 5632 work-group synchronization. 5633 5634See ``WGP_MODE`` field in 5635:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 5636:ref:`amdgpu-target-features`. 5637 5638The code sequences used to implement the memory model for GFX10 are defined in 5639table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 5640 5641 .. table:: AMDHSA Memory Model Code Sequences GFX10 5642 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 5643 5644 ============ ============ ============== ========== ================================ 5645 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5646 Ordering Sync Scope Address GFX10 5647 Space 5648 ============ ============ ============== ========== ================================ 5649 **Non-Atomic** 5650 ------------------------------------------------------------------------------------ 5651 load *none* *none* - global - !volatile & !nontemporal 5652 - generic 5653 - private 1. buffer/global/flat_load 5654 - constant 5655 - volatile & !nontemporal 5656 5657 1. buffer/global/flat_load 5658 glc=1 dlc=1 5659 5660 - nontemporal 5661 5662 1. buffer/global/flat_load 5663 slc=1 5664 5665 load *none* *none* - local 1. ds_load 5666 store *none* *none* - global - !nontemporal 5667 - generic 5668 - private 1. buffer/global/flat_store 5669 - constant 5670 - nontemporal 5671 5672 1. buffer/global/flat_store 5673 slc=1 5674 5675 store *none* *none* - local 1. ds_store 5676 **Unordered Atomic** 5677 ------------------------------------------------------------------------------------ 5678 load atomic unordered *any* *any* *Same as non-atomic*. 5679 store atomic unordered *any* *any* *Same as non-atomic*. 5680 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5681 **Monotonic Atomic** 5682 ------------------------------------------------------------------------------------ 5683 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 5684 - wavefront - generic 5685 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 5686 - generic glc=1 5687 5688 - If CU wavefront execution 5689 mode, omit glc=1. 5690 5691 load atomic monotonic - singlethread - local 1. ds_load 5692 - wavefront 5693 - workgroup 5694 load atomic monotonic - agent - global 1. buffer/global/flat_load 5695 - system - generic glc=1 dlc=1 5696 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5697 - wavefront - generic 5698 - workgroup 5699 - agent 5700 - system 5701 store atomic monotonic - singlethread - local 1. ds_store 5702 - wavefront 5703 - workgroup 5704 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5705 - wavefront - generic 5706 - workgroup 5707 - agent 5708 - system 5709 atomicrmw monotonic - singlethread - local 1. ds_atomic 5710 - wavefront 5711 - workgroup 5712 **Acquire Atomic** 5713 ------------------------------------------------------------------------------------ 5714 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5715 - wavefront - local 5716 - generic 5717 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 5718 5719 - If CU wavefront execution 5720 mode, omit glc=1. 5721 5722 2. s_waitcnt vmcnt(0) 5723 5724 - If CU wavefront execution 5725 mode, omit. 5726 - Must happen before 5727 the following buffer_gl0_inv 5728 and before any following 5729 global/generic 5730 load/load 5731 atomic/store/store 5732 atomic/atomicrmw. 5733 5734 3. buffer_gl0_inv 5735 5736 - If CU wavefront execution 5737 mode, omit. 5738 - Ensures that 5739 following 5740 loads will not see 5741 stale data. 5742 5743 load atomic acquire - workgroup - local 1. ds_load 5744 2. s_waitcnt lgkmcnt(0) 5745 5746 - If OpenCL, omit. 5747 - Must happen before 5748 the following buffer_gl0_inv 5749 and before any following 5750 global/generic load/load 5751 atomic/store/store 5752 atomic/atomicrmw. 5753 - Ensures any 5754 following global 5755 data read is no 5756 older than the local load 5757 atomic value being 5758 acquired. 5759 5760 3. buffer_gl0_inv 5761 5762 - If CU wavefront execution 5763 mode, omit. 5764 - If OpenCL, omit. 5765 - Ensures that 5766 following 5767 loads will not see 5768 stale data. 5769 5770 load atomic acquire - workgroup - generic 1. flat_load glc=1 5771 5772 - If CU wavefront execution 5773 mode, omit glc=1. 5774 5775 2. s_waitcnt lgkmcnt(0) & 5776 vmcnt(0) 5777 5778 - If CU wavefront execution 5779 mode, omit vmcnt(0). 5780 - If OpenCL, omit 5781 lgkmcnt(0). 5782 - Must happen before 5783 the following 5784 buffer_gl0_inv and any 5785 following global/generic 5786 load/load 5787 atomic/store/store 5788 atomic/atomicrmw. 5789 - Ensures any 5790 following global 5791 data read is no 5792 older than a local load 5793 atomic value being 5794 acquired. 5795 5796 3. buffer_gl0_inv 5797 5798 - If CU wavefront execution 5799 mode, omit. 5800 - Ensures that 5801 following 5802 loads will not see 5803 stale data. 5804 5805 load atomic acquire - agent - global 1. buffer/global_load 5806 - system glc=1 dlc=1 5807 2. s_waitcnt vmcnt(0) 5808 5809 - Must happen before 5810 following 5811 buffer_gl*_inv. 5812 - Ensures the load 5813 has completed 5814 before invalidating 5815 the caches. 5816 5817 3. buffer_gl0_inv; 5818 buffer_gl1_inv 5819 5820 - Must happen before 5821 any following 5822 global/generic 5823 load/load 5824 atomic/atomicrmw. 5825 - Ensures that 5826 following 5827 loads will not see 5828 stale global data. 5829 5830 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 5831 - system 2. s_waitcnt vmcnt(0) & 5832 lgkmcnt(0) 5833 5834 - If OpenCL omit 5835 lgkmcnt(0). 5836 - Must happen before 5837 following 5838 buffer_gl*_invl. 5839 - Ensures the flat_load 5840 has completed 5841 before invalidating 5842 the caches. 5843 5844 3. buffer_gl0_inv; 5845 buffer_gl1_inv 5846 5847 - Must happen before 5848 any following 5849 global/generic 5850 load/load 5851 atomic/atomicrmw. 5852 - Ensures that 5853 following loads 5854 will not see stale 5855 global data. 5856 5857 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5858 - wavefront - local 5859 - generic 5860 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5861 2. s_waitcnt vm/vscnt(0) 5862 5863 - If CU wavefront execution 5864 mode, omit. 5865 - Use vmcnt(0) if atomic with 5866 return and vscnt(0) if 5867 atomic with no-return. 5868 - Must happen before 5869 the following buffer_gl0_inv 5870 and before any following 5871 global/generic 5872 load/load 5873 atomic/store/store 5874 atomic/atomicrmw. 5875 5876 3. buffer_gl0_inv 5877 5878 - If CU wavefront execution 5879 mode, omit. 5880 - Ensures that 5881 following 5882 loads will not see 5883 stale data. 5884 5885 atomicrmw acquire - workgroup - local 1. ds_atomic 5886 2. s_waitcnt lgkmcnt(0) 5887 5888 - If OpenCL, omit. 5889 - Must happen before 5890 the following 5891 buffer_gl0_inv. 5892 - Ensures any 5893 following global 5894 data read is no 5895 older than the local 5896 atomicrmw value 5897 being acquired. 5898 5899 3. buffer_gl0_inv 5900 5901 - If OpenCL omit. 5902 - Ensures that 5903 following 5904 loads will not see 5905 stale data. 5906 5907 atomicrmw acquire - workgroup - generic 1. flat_atomic 5908 2. s_waitcnt lgkmcnt(0) & 5909 vm/vscnt(0) 5910 5911 - If CU wavefront execution 5912 mode, omit vm/vscnt(0). 5913 - If OpenCL, omit lgkmcnt(0). 5914 - Use vmcnt(0) if atomic with 5915 return and vscnt(0) if 5916 atomic with no-return. 5917 - Must happen before 5918 the following 5919 buffer_gl0_inv. 5920 - Ensures any 5921 following global 5922 data read is no 5923 older than a local 5924 atomicrmw value 5925 being acquired. 5926 5927 3. buffer_gl0_inv 5928 5929 - If CU wavefront execution 5930 mode, omit. 5931 - Ensures that 5932 following 5933 loads will not see 5934 stale data. 5935 5936 atomicrmw acquire - agent - global 1. buffer/global_atomic 5937 - system 2. s_waitcnt vm/vscnt(0) 5938 5939 - Use vmcnt(0) if atomic with 5940 return and vscnt(0) if 5941 atomic with no-return. 5942 - Must happen before 5943 following 5944 buffer_gl*_inv. 5945 - Ensures the 5946 atomicrmw has 5947 completed before 5948 invalidating the 5949 caches. 5950 5951 3. buffer_gl0_inv; 5952 buffer_gl1_inv 5953 5954 - Must happen before 5955 any following 5956 global/generic 5957 load/load 5958 atomic/atomicrmw. 5959 - Ensures that 5960 following loads 5961 will not see stale 5962 global data. 5963 5964 atomicrmw acquire - agent - generic 1. flat_atomic 5965 - system 2. s_waitcnt vm/vscnt(0) & 5966 lgkmcnt(0) 5967 5968 - If OpenCL, omit 5969 lgkmcnt(0). 5970 - Use vmcnt(0) if atomic with 5971 return and vscnt(0) if 5972 atomic with no-return. 5973 - Must happen before 5974 following 5975 buffer_gl*_inv. 5976 - Ensures the 5977 atomicrmw has 5978 completed before 5979 invalidating the 5980 caches. 5981 5982 3. buffer_gl0_inv; 5983 buffer_gl1_inv 5984 5985 - Must happen before 5986 any following 5987 global/generic 5988 load/load 5989 atomic/atomicrmw. 5990 - Ensures that 5991 following loads 5992 will not see stale 5993 global data. 5994 5995 fence acquire - singlethread *none* *none* 5996 - wavefront 5997 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 5998 vmcnt(0) & vscnt(0) 5999 6000 - If CU wavefront execution 6001 mode, omit vmcnt(0) and 6002 vscnt(0). 6003 - If OpenCL and 6004 address space is 6005 not generic, omit 6006 lgkmcnt(0). 6007 - If OpenCL and 6008 address space is 6009 local, omit 6010 vmcnt(0) and vscnt(0). 6011 - However, since LLVM 6012 currently has no 6013 address space on 6014 the fence need to 6015 conservatively 6016 always generate. If 6017 fence had an 6018 address space then 6019 set to address 6020 space of OpenCL 6021 fence flag, or to 6022 generic if both 6023 local and global 6024 flags are 6025 specified. 6026 - Could be split into 6027 separate s_waitcnt 6028 vmcnt(0), s_waitcnt 6029 vscnt(0) and s_waitcnt 6030 lgkmcnt(0) to allow 6031 them to be 6032 independently moved 6033 according to the 6034 following rules. 6035 - s_waitcnt vmcnt(0) 6036 must happen after 6037 any preceding 6038 global/generic load 6039 atomic/ 6040 atomicrmw-with-return-value 6041 with an equal or 6042 wider sync scope 6043 and memory ordering 6044 stronger than 6045 unordered (this is 6046 termed the 6047 fence-paired-atomic). 6048 - s_waitcnt vscnt(0) 6049 must happen after 6050 any preceding 6051 global/generic 6052 atomicrmw-no-return-value 6053 with an equal or 6054 wider sync scope 6055 and memory ordering 6056 stronger than 6057 unordered (this is 6058 termed the 6059 fence-paired-atomic). 6060 - s_waitcnt lgkmcnt(0) 6061 must happen after 6062 any preceding 6063 local/generic load 6064 atomic/atomicrmw 6065 with an equal or 6066 wider sync scope 6067 and memory ordering 6068 stronger than 6069 unordered (this is 6070 termed the 6071 fence-paired-atomic). 6072 - Must happen before 6073 the following 6074 buffer_gl0_inv. 6075 - Ensures that the 6076 fence-paired atomic 6077 has completed 6078 before invalidating 6079 the 6080 cache. Therefore 6081 any following 6082 locations read must 6083 be no older than 6084 the value read by 6085 the 6086 fence-paired-atomic. 6087 6088 3. buffer_gl0_inv 6089 6090 - If CU wavefront execution 6091 mode, omit. 6092 - Ensures that 6093 following 6094 loads will not see 6095 stale data. 6096 6097 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 6098 - system vmcnt(0) & vscnt(0) 6099 6100 - If OpenCL and 6101 address space is 6102 not generic, omit 6103 lgkmcnt(0). 6104 - If OpenCL and 6105 address space is 6106 local, omit 6107 vmcnt(0) and vscnt(0). 6108 - However, since LLVM 6109 currently has no 6110 address space on 6111 the fence need to 6112 conservatively 6113 always generate 6114 (see comment for 6115 previous fence). 6116 - Could be split into 6117 separate s_waitcnt 6118 vmcnt(0), s_waitcnt 6119 vscnt(0) and s_waitcnt 6120 lgkmcnt(0) to allow 6121 them to be 6122 independently moved 6123 according to the 6124 following rules. 6125 - s_waitcnt vmcnt(0) 6126 must happen after 6127 any preceding 6128 global/generic load 6129 atomic/ 6130 atomicrmw-with-return-value 6131 with an equal or 6132 wider sync scope 6133 and memory ordering 6134 stronger than 6135 unordered (this is 6136 termed the 6137 fence-paired-atomic). 6138 - s_waitcnt vscnt(0) 6139 must happen after 6140 any preceding 6141 global/generic 6142 atomicrmw-no-return-value 6143 with an equal or 6144 wider sync scope 6145 and memory ordering 6146 stronger than 6147 unordered (this is 6148 termed the 6149 fence-paired-atomic). 6150 - s_waitcnt lgkmcnt(0) 6151 must happen after 6152 any preceding 6153 local/generic load 6154 atomic/atomicrmw 6155 with an equal or 6156 wider sync scope 6157 and memory ordering 6158 stronger than 6159 unordered (this is 6160 termed the 6161 fence-paired-atomic). 6162 - Must happen before 6163 the following 6164 buffer_gl*_inv. 6165 - Ensures that the 6166 fence-paired atomic 6167 has completed 6168 before invalidating 6169 the 6170 caches. Therefore 6171 any following 6172 locations read must 6173 be no older than 6174 the value read by 6175 the 6176 fence-paired-atomic. 6177 6178 2. buffer_gl0_inv; 6179 buffer_gl1_inv 6180 6181 - Must happen before any 6182 following global/generic 6183 load/load 6184 atomic/store/store 6185 atomic/atomicrmw. 6186 - Ensures that 6187 following loads 6188 will not see stale 6189 global data. 6190 6191 **Release Atomic** 6192 ------------------------------------------------------------------------------------ 6193 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 6194 - wavefront - local 6195 - generic 6196 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 6197 - generic vmcnt(0) & vscnt(0) 6198 6199 - If CU wavefront execution 6200 mode, omit vmcnt(0) and 6201 vscnt(0). 6202 - If OpenCL, omit 6203 lgkmcnt(0). 6204 - Could be split into 6205 separate s_waitcnt 6206 vmcnt(0), s_waitcnt 6207 vscnt(0) and s_waitcnt 6208 lgkmcnt(0) to allow 6209 them to be 6210 independently moved 6211 according to the 6212 following rules. 6213 - s_waitcnt vmcnt(0) 6214 must happen after 6215 any preceding 6216 global/generic load/load 6217 atomic/ 6218 atomicrmw-with-return-value. 6219 - s_waitcnt vscnt(0) 6220 must happen after 6221 any preceding 6222 global/generic 6223 store/store 6224 atomic/ 6225 atomicrmw-no-return-value. 6226 - s_waitcnt lgkmcnt(0) 6227 must happen after 6228 any preceding 6229 local/generic 6230 load/store/load 6231 atomic/store 6232 atomic/atomicrmw. 6233 - Must happen before 6234 the following 6235 store. 6236 - Ensures that all 6237 memory operations 6238 have 6239 completed before 6240 performing the 6241 store that is being 6242 released. 6243 6244 2. buffer/global/flat_store 6245 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 6246 6247 - If CU wavefront execution 6248 mode, omit. 6249 - If OpenCL, omit. 6250 - Could be split into 6251 separate s_waitcnt 6252 vmcnt(0) and s_waitcnt 6253 vscnt(0) to allow 6254 them to be 6255 independently moved 6256 according to the 6257 following rules. 6258 - s_waitcnt vmcnt(0) 6259 must happen after 6260 any preceding 6261 global/generic load/load 6262 atomic/ 6263 atomicrmw-with-return-value. 6264 - s_waitcnt vscnt(0) 6265 must happen after 6266 any preceding 6267 global/generic 6268 store/store atomic/ 6269 atomicrmw-no-return-value. 6270 - Must happen before 6271 the following 6272 store. 6273 - Ensures that all 6274 global memory 6275 operations have 6276 completed before 6277 performing the 6278 store that is being 6279 released. 6280 6281 2. ds_store 6282 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 6283 - system - generic vmcnt(0) & vscnt(0) 6284 6285 - If OpenCL and 6286 address space is 6287 not generic, omit 6288 lgkmcnt(0). 6289 - Could be split into 6290 separate s_waitcnt 6291 vmcnt(0), s_waitcnt vscnt(0) 6292 and s_waitcnt 6293 lgkmcnt(0) to allow 6294 them to be 6295 independently moved 6296 according to the 6297 following rules. 6298 - s_waitcnt vmcnt(0) 6299 must happen after 6300 any preceding 6301 global/generic 6302 load/load 6303 atomic/ 6304 atomicrmw-with-return-value. 6305 - s_waitcnt vscnt(0) 6306 must happen after 6307 any preceding 6308 global/generic 6309 store/store atomic/ 6310 atomicrmw-no-return-value. 6311 - s_waitcnt lgkmcnt(0) 6312 must happen after 6313 any preceding 6314 local/generic 6315 load/store/load 6316 atomic/store 6317 atomic/atomicrmw. 6318 - Must happen before 6319 the following 6320 store. 6321 - Ensures that all 6322 memory operations 6323 have 6324 completed before 6325 performing the 6326 store that is being 6327 released. 6328 6329 2. buffer/global/flat_store 6330 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 6331 - wavefront - local 6332 - generic 6333 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 6334 - generic vmcnt(0) & vscnt(0) 6335 6336 - If CU wavefront execution 6337 mode, omit vmcnt(0) and 6338 vscnt(0). 6339 - If OpenCL, omit lgkmcnt(0). 6340 - Could be split into 6341 separate s_waitcnt 6342 vmcnt(0), s_waitcnt 6343 vscnt(0) and s_waitcnt 6344 lgkmcnt(0) to allow 6345 them to be 6346 independently moved 6347 according to the 6348 following rules. 6349 - s_waitcnt vmcnt(0) 6350 must happen after 6351 any preceding 6352 global/generic load/load 6353 atomic/ 6354 atomicrmw-with-return-value. 6355 - s_waitcnt vscnt(0) 6356 must happen after 6357 any preceding 6358 global/generic 6359 store/store 6360 atomic/ 6361 atomicrmw-no-return-value. 6362 - s_waitcnt lgkmcnt(0) 6363 must happen after 6364 any preceding 6365 local/generic 6366 load/store/load 6367 atomic/store 6368 atomic/atomicrmw. 6369 - Must happen before 6370 the following 6371 atomicrmw. 6372 - Ensures that all 6373 memory operations 6374 have 6375 completed before 6376 performing the 6377 atomicrmw that is 6378 being released. 6379 6380 2. buffer/global/flat_atomic 6381 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 6382 6383 - If CU wavefront execution 6384 mode, omit. 6385 - If OpenCL, omit. 6386 - Could be split into 6387 separate s_waitcnt 6388 vmcnt(0) and s_waitcnt 6389 vscnt(0) to allow 6390 them to be 6391 independently moved 6392 according to the 6393 following rules. 6394 - s_waitcnt vmcnt(0) 6395 must happen after 6396 any preceding 6397 global/generic load/load 6398 atomic/ 6399 atomicrmw-with-return-value. 6400 - s_waitcnt vscnt(0) 6401 must happen after 6402 any preceding 6403 global/generic 6404 store/store atomic/ 6405 atomicrmw-no-return-value. 6406 - Must happen before 6407 the following 6408 store. 6409 - Ensures that all 6410 global memory 6411 operations have 6412 completed before 6413 performing the 6414 store that is being 6415 released. 6416 6417 2. ds_atomic 6418 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 6419 - system - generic vmcnt(0) & vscnt(0) 6420 6421 - If OpenCL, omit 6422 lgkmcnt(0). 6423 - Could be split into 6424 separate s_waitcnt 6425 vmcnt(0), s_waitcnt 6426 vscnt(0) and s_waitcnt 6427 lgkmcnt(0) to allow 6428 them to be 6429 independently moved 6430 according to the 6431 following rules. 6432 - s_waitcnt vmcnt(0) 6433 must happen after 6434 any preceding 6435 global/generic 6436 load/load atomic/ 6437 atomicrmw-with-return-value. 6438 - s_waitcnt vscnt(0) 6439 must happen after 6440 any preceding 6441 global/generic 6442 store/store atomic/ 6443 atomicrmw-no-return-value. 6444 - s_waitcnt lgkmcnt(0) 6445 must happen after 6446 any preceding 6447 local/generic 6448 load/store/load 6449 atomic/store 6450 atomic/atomicrmw. 6451 - Must happen before 6452 the following 6453 atomicrmw. 6454 - Ensures that all 6455 memory operations 6456 to global and local 6457 have completed 6458 before performing 6459 the atomicrmw that 6460 is being released. 6461 6462 2. buffer/global/flat_atomic 6463 fence release - singlethread *none* *none* 6464 - wavefront 6465 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 6466 vmcnt(0) & vscnt(0) 6467 6468 - If CU wavefront execution 6469 mode, omit vmcnt(0) and 6470 vscnt(0). 6471 - If OpenCL and 6472 address space is 6473 not generic, omit 6474 lgkmcnt(0). 6475 - If OpenCL and 6476 address space is 6477 local, omit 6478 vmcnt(0) and vscnt(0). 6479 - However, since LLVM 6480 currently has no 6481 address space on 6482 the fence need to 6483 conservatively 6484 always generate. If 6485 fence had an 6486 address space then 6487 set to address 6488 space of OpenCL 6489 fence flag, or to 6490 generic if both 6491 local and global 6492 flags are 6493 specified. 6494 - Could be split into 6495 separate s_waitcnt 6496 vmcnt(0), s_waitcnt 6497 vscnt(0) and s_waitcnt 6498 lgkmcnt(0) to allow 6499 them to be 6500 independently moved 6501 according to the 6502 following rules. 6503 - s_waitcnt vmcnt(0) 6504 must happen after 6505 any preceding 6506 global/generic 6507 load/load 6508 atomic/ 6509 atomicrmw-with-return-value. 6510 - s_waitcnt vscnt(0) 6511 must happen after 6512 any preceding 6513 global/generic 6514 store/store atomic/ 6515 atomicrmw-no-return-value. 6516 - s_waitcnt lgkmcnt(0) 6517 must happen after 6518 any preceding 6519 local/generic 6520 load/store/load 6521 atomic/store atomic/ 6522 atomicrmw. 6523 - Must happen before 6524 any following store 6525 atomic/atomicrmw 6526 with an equal or 6527 wider sync scope 6528 and memory ordering 6529 stronger than 6530 unordered (this is 6531 termed the 6532 fence-paired-atomic). 6533 - Ensures that all 6534 memory operations 6535 have 6536 completed before 6537 performing the 6538 following 6539 fence-paired-atomic. 6540 6541 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 6542 - system vmcnt(0) & vscnt(0) 6543 6544 - If OpenCL and 6545 address space is 6546 not generic, omit 6547 lgkmcnt(0). 6548 - If OpenCL and 6549 address space is 6550 local, omit 6551 vmcnt(0) and vscnt(0). 6552 - However, since LLVM 6553 currently has no 6554 address space on 6555 the fence need to 6556 conservatively 6557 always generate. If 6558 fence had an 6559 address space then 6560 set to address 6561 space of OpenCL 6562 fence flag, or to 6563 generic if both 6564 local and global 6565 flags are 6566 specified. 6567 - Could be split into 6568 separate s_waitcnt 6569 vmcnt(0), s_waitcnt 6570 vscnt(0) and s_waitcnt 6571 lgkmcnt(0) to allow 6572 them to be 6573 independently moved 6574 according to the 6575 following rules. 6576 - s_waitcnt vmcnt(0) 6577 must happen after 6578 any preceding 6579 global/generic 6580 load/load atomic/ 6581 atomicrmw-with-return-value. 6582 - s_waitcnt vscnt(0) 6583 must happen after 6584 any preceding 6585 global/generic 6586 store/store atomic/ 6587 atomicrmw-no-return-value. 6588 - s_waitcnt lgkmcnt(0) 6589 must happen after 6590 any preceding 6591 local/generic 6592 load/store/load 6593 atomic/store 6594 atomic/atomicrmw. 6595 - Must happen before 6596 any following store 6597 atomic/atomicrmw 6598 with an equal or 6599 wider sync scope 6600 and memory ordering 6601 stronger than 6602 unordered (this is 6603 termed the 6604 fence-paired-atomic). 6605 - Ensures that all 6606 memory operations 6607 have 6608 completed before 6609 performing the 6610 following 6611 fence-paired-atomic. 6612 6613 **Acquire-Release Atomic** 6614 ------------------------------------------------------------------------------------ 6615 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 6616 - wavefront - local 6617 - generic 6618 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 6619 vmcnt(0) & vscnt(0) 6620 6621 - If CU wavefront execution 6622 mode, omit vmcnt(0) and 6623 vscnt(0). 6624 - If OpenCL, omit 6625 lgkmcnt(0). 6626 - Must happen after 6627 any preceding 6628 local/generic 6629 load/store/load 6630 atomic/store 6631 atomic/atomicrmw. 6632 - Could be split into 6633 separate s_waitcnt 6634 vmcnt(0), s_waitcnt 6635 vscnt(0), and s_waitcnt 6636 lgkmcnt(0) to allow 6637 them to be 6638 independently moved 6639 according to the 6640 following rules. 6641 - s_waitcnt vmcnt(0) 6642 must happen after 6643 any preceding 6644 global/generic load/load 6645 atomic/ 6646 atomicrmw-with-return-value. 6647 - s_waitcnt vscnt(0) 6648 must happen after 6649 any preceding 6650 global/generic 6651 store/store 6652 atomic/ 6653 atomicrmw-no-return-value. 6654 - s_waitcnt lgkmcnt(0) 6655 must happen after 6656 any preceding 6657 local/generic 6658 load/store/load 6659 atomic/store 6660 atomic/atomicrmw. 6661 - Must happen before 6662 the following 6663 atomicrmw. 6664 - Ensures that all 6665 memory operations 6666 have 6667 completed before 6668 performing the 6669 atomicrmw that is 6670 being released. 6671 6672 2. buffer/global_atomic 6673 3. s_waitcnt vm/vscnt(0) 6674 6675 - If CU wavefront execution 6676 mode, omit. 6677 - Use vmcnt(0) if atomic with 6678 return and vscnt(0) if 6679 atomic with no-return. 6680 - Must happen before 6681 the following 6682 buffer_gl0_inv. 6683 - Ensures any 6684 following global 6685 data read is no 6686 older than the 6687 atomicrmw value 6688 being acquired. 6689 6690 4. buffer_gl0_inv 6691 6692 - If CU wavefront execution 6693 mode, omit. 6694 - Ensures that 6695 following 6696 loads will not see 6697 stale data. 6698 6699 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 6700 6701 - If CU wavefront execution 6702 mode, omit. 6703 - If OpenCL, omit. 6704 - Could be split into 6705 separate s_waitcnt 6706 vmcnt(0) and s_waitcnt 6707 vscnt(0) to allow 6708 them to be 6709 independently moved 6710 according to the 6711 following rules. 6712 - s_waitcnt vmcnt(0) 6713 must happen after 6714 any preceding 6715 global/generic load/load 6716 atomic/ 6717 atomicrmw-with-return-value. 6718 - s_waitcnt vscnt(0) 6719 must happen after 6720 any preceding 6721 global/generic 6722 store/store atomic/ 6723 atomicrmw-no-return-value. 6724 - Must happen before 6725 the following 6726 store. 6727 - Ensures that all 6728 global memory 6729 operations have 6730 completed before 6731 performing the 6732 store that is being 6733 released. 6734 6735 2. ds_atomic 6736 3. s_waitcnt lgkmcnt(0) 6737 6738 - If OpenCL, omit. 6739 - Must happen before 6740 the following 6741 buffer_gl0_inv. 6742 - Ensures any 6743 following global 6744 data read is no 6745 older than the local load 6746 atomic value being 6747 acquired. 6748 6749 4. buffer_gl0_inv 6750 6751 - If CU wavefront execution 6752 mode, omit. 6753 - If OpenCL omit. 6754 - Ensures that 6755 following 6756 loads will not see 6757 stale data. 6758 6759 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 6760 vmcnt(0) & vscnt(0) 6761 6762 - If CU wavefront execution 6763 mode, omit vmcnt(0) and 6764 vscnt(0). 6765 - If OpenCL, omit lgkmcnt(0). 6766 - Could be split into 6767 separate s_waitcnt 6768 vmcnt(0), s_waitcnt 6769 vscnt(0) and s_waitcnt 6770 lgkmcnt(0) to allow 6771 them to be 6772 independently moved 6773 according to the 6774 following rules. 6775 - s_waitcnt vmcnt(0) 6776 must happen after 6777 any preceding 6778 global/generic load/load 6779 atomic/ 6780 atomicrmw-with-return-value. 6781 - s_waitcnt vscnt(0) 6782 must happen after 6783 any preceding 6784 global/generic 6785 store/store 6786 atomic/ 6787 atomicrmw-no-return-value. 6788 - s_waitcnt lgkmcnt(0) 6789 must happen after 6790 any preceding 6791 local/generic 6792 load/store/load 6793 atomic/store 6794 atomic/atomicrmw. 6795 - Must happen before 6796 the following 6797 atomicrmw. 6798 - Ensures that all 6799 memory operations 6800 have 6801 completed before 6802 performing the 6803 atomicrmw that is 6804 being released. 6805 6806 2. flat_atomic 6807 3. s_waitcnt lgkmcnt(0) & 6808 vmcnt(0) & vscnt(0) 6809 6810 - If CU wavefront execution 6811 mode, omit vmcnt(0) and 6812 vscnt(0). 6813 - If OpenCL, omit lgkmcnt(0). 6814 - Must happen before 6815 the following 6816 buffer_gl0_inv. 6817 - Ensures any 6818 following global 6819 data read is no 6820 older than the load 6821 atomic value being 6822 acquired. 6823 6824 3. buffer_gl0_inv 6825 6826 - If CU wavefront execution 6827 mode, omit. 6828 - Ensures that 6829 following 6830 loads will not see 6831 stale data. 6832 6833 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 6834 - system vmcnt(0) & vscnt(0) 6835 6836 - If OpenCL, omit 6837 lgkmcnt(0). 6838 - Could be split into 6839 separate s_waitcnt 6840 vmcnt(0), s_waitcnt 6841 vscnt(0) and s_waitcnt 6842 lgkmcnt(0) to allow 6843 them to be 6844 independently moved 6845 according to the 6846 following rules. 6847 - s_waitcnt vmcnt(0) 6848 must happen after 6849 any preceding 6850 global/generic 6851 load/load atomic/ 6852 atomicrmw-with-return-value. 6853 - s_waitcnt vscnt(0) 6854 must happen after 6855 any preceding 6856 global/generic 6857 store/store atomic/ 6858 atomicrmw-no-return-value. 6859 - s_waitcnt lgkmcnt(0) 6860 must happen after 6861 any preceding 6862 local/generic 6863 load/store/load 6864 atomic/store 6865 atomic/atomicrmw. 6866 - Must happen before 6867 the following 6868 atomicrmw. 6869 - Ensures that all 6870 memory operations 6871 to global have 6872 completed before 6873 performing the 6874 atomicrmw that is 6875 being released. 6876 6877 2. buffer/global_atomic 6878 3. s_waitcnt vm/vscnt(0) 6879 6880 - Use vmcnt(0) if atomic with 6881 return and vscnt(0) if 6882 atomic with no-return. 6883 - Must happen before 6884 following 6885 buffer_gl*_inv. 6886 - Ensures the 6887 atomicrmw has 6888 completed before 6889 invalidating the 6890 caches. 6891 6892 4. buffer_gl0_inv; 6893 buffer_gl1_inv 6894 6895 - Must happen before 6896 any following 6897 global/generic 6898 load/load 6899 atomic/atomicrmw. 6900 - Ensures that 6901 following loads 6902 will not see stale 6903 global data. 6904 6905 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 6906 - system vmcnt(0) & vscnt(0) 6907 6908 - If OpenCL, omit 6909 lgkmcnt(0). 6910 - Could be split into 6911 separate s_waitcnt 6912 vmcnt(0), s_waitcnt 6913 vscnt(0), and s_waitcnt 6914 lgkmcnt(0) to allow 6915 them to be 6916 independently moved 6917 according to the 6918 following rules. 6919 - s_waitcnt vmcnt(0) 6920 must happen after 6921 any preceding 6922 global/generic 6923 load/load atomic 6924 atomicrmw-with-return-value. 6925 - s_waitcnt vscnt(0) 6926 must happen after 6927 any preceding 6928 global/generic 6929 store/store atomic/ 6930 atomicrmw-no-return-value. 6931 - s_waitcnt lgkmcnt(0) 6932 must happen after 6933 any preceding 6934 local/generic 6935 load/store/load 6936 atomic/store 6937 atomic/atomicrmw. 6938 - Must happen before 6939 the following 6940 atomicrmw. 6941 - Ensures that all 6942 memory operations 6943 have 6944 completed before 6945 performing the 6946 atomicrmw that is 6947 being released. 6948 6949 2. flat_atomic 6950 3. s_waitcnt vm/vscnt(0) & 6951 lgkmcnt(0) 6952 6953 - If OpenCL, omit 6954 lgkmcnt(0). 6955 - Use vmcnt(0) if atomic with 6956 return and vscnt(0) if 6957 atomic with no-return. 6958 - Must happen before 6959 following 6960 buffer_gl*_inv. 6961 - Ensures the 6962 atomicrmw has 6963 completed before 6964 invalidating the 6965 caches. 6966 6967 4. buffer_gl0_inv; 6968 buffer_gl1_inv 6969 6970 - Must happen before 6971 any following 6972 global/generic 6973 load/load 6974 atomic/atomicrmw. 6975 - Ensures that 6976 following loads 6977 will not see stale 6978 global data. 6979 6980 fence acq_rel - singlethread *none* *none* 6981 - wavefront 6982 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 6983 vmcnt(0) & vscnt(0) 6984 6985 - If CU wavefront execution 6986 mode, omit vmcnt(0) and 6987 vscnt(0). 6988 - If OpenCL and 6989 address space is 6990 not generic, omit 6991 lgkmcnt(0). 6992 - If OpenCL and 6993 address space is 6994 local, omit 6995 vmcnt(0) and vscnt(0). 6996 - However, 6997 since LLVM 6998 currently has no 6999 address space on 7000 the fence need to 7001 conservatively 7002 always generate 7003 (see comment for 7004 previous fence). 7005 - Could be split into 7006 separate s_waitcnt 7007 vmcnt(0), s_waitcnt 7008 vscnt(0) and s_waitcnt 7009 lgkmcnt(0) to allow 7010 them to be 7011 independently moved 7012 according to the 7013 following rules. 7014 - s_waitcnt vmcnt(0) 7015 must happen after 7016 any preceding 7017 global/generic 7018 load/load 7019 atomic/ 7020 atomicrmw-with-return-value. 7021 - s_waitcnt vscnt(0) 7022 must happen after 7023 any preceding 7024 global/generic 7025 store/store atomic/ 7026 atomicrmw-no-return-value. 7027 - s_waitcnt lgkmcnt(0) 7028 must happen after 7029 any preceding 7030 local/generic 7031 load/store/load 7032 atomic/store atomic/ 7033 atomicrmw. 7034 - Must happen before 7035 any following 7036 global/generic 7037 load/load 7038 atomic/store/store 7039 atomic/atomicrmw. 7040 - Ensures that all 7041 memory operations 7042 have 7043 completed before 7044 performing any 7045 following global 7046 memory operations. 7047 - Ensures that the 7048 preceding 7049 local/generic load 7050 atomic/atomicrmw 7051 with an equal or 7052 wider sync scope 7053 and memory ordering 7054 stronger than 7055 unordered (this is 7056 termed the 7057 acquire-fence-paired-atomic) 7058 has completed 7059 before following 7060 global memory 7061 operations. This 7062 satisfies the 7063 requirements of 7064 acquire. 7065 - Ensures that all 7066 previous memory 7067 operations have 7068 completed before a 7069 following 7070 local/generic store 7071 atomic/atomicrmw 7072 with an equal or 7073 wider sync scope 7074 and memory ordering 7075 stronger than 7076 unordered (this is 7077 termed the 7078 release-fence-paired-atomic). 7079 This satisfies the 7080 requirements of 7081 release. 7082 - Must happen before 7083 the following 7084 buffer_gl0_inv. 7085 - Ensures that the 7086 acquire-fence-paired 7087 atomic has completed 7088 before invalidating 7089 the 7090 cache. Therefore 7091 any following 7092 locations read must 7093 be no older than 7094 the value read by 7095 the 7096 acquire-fence-paired-atomic. 7097 7098 3. buffer_gl0_inv 7099 7100 - If CU wavefront execution 7101 mode, omit. 7102 - Ensures that 7103 following 7104 loads will not see 7105 stale data. 7106 7107 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 7108 - system vmcnt(0) & vscnt(0) 7109 7110 - If OpenCL and 7111 address space is 7112 not generic, omit 7113 lgkmcnt(0). 7114 - If OpenCL and 7115 address space is 7116 local, omit 7117 vmcnt(0) and vscnt(0). 7118 - However, since LLVM 7119 currently has no 7120 address space on 7121 the fence need to 7122 conservatively 7123 always generate 7124 (see comment for 7125 previous fence). 7126 - Could be split into 7127 separate s_waitcnt 7128 vmcnt(0), s_waitcnt 7129 vscnt(0) and s_waitcnt 7130 lgkmcnt(0) to allow 7131 them to be 7132 independently moved 7133 according to the 7134 following rules. 7135 - s_waitcnt vmcnt(0) 7136 must happen after 7137 any preceding 7138 global/generic 7139 load/load 7140 atomic/ 7141 atomicrmw-with-return-value. 7142 - s_waitcnt vscnt(0) 7143 must happen after 7144 any preceding 7145 global/generic 7146 store/store atomic/ 7147 atomicrmw-no-return-value. 7148 - s_waitcnt lgkmcnt(0) 7149 must happen after 7150 any preceding 7151 local/generic 7152 load/store/load 7153 atomic/store 7154 atomic/atomicrmw. 7155 - Must happen before 7156 the following 7157 buffer_gl*_inv. 7158 - Ensures that the 7159 preceding 7160 global/local/generic 7161 load 7162 atomic/atomicrmw 7163 with an equal or 7164 wider sync scope 7165 and memory ordering 7166 stronger than 7167 unordered (this is 7168 termed the 7169 acquire-fence-paired-atomic) 7170 has completed 7171 before invalidating 7172 the caches. This 7173 satisfies the 7174 requirements of 7175 acquire. 7176 - Ensures that all 7177 previous memory 7178 operations have 7179 completed before a 7180 following 7181 global/local/generic 7182 store 7183 atomic/atomicrmw 7184 with an equal or 7185 wider sync scope 7186 and memory ordering 7187 stronger than 7188 unordered (this is 7189 termed the 7190 release-fence-paired-atomic). 7191 This satisfies the 7192 requirements of 7193 release. 7194 7195 2. buffer_gl0_inv; 7196 buffer_gl1_inv 7197 7198 - Must happen before 7199 any following 7200 global/generic 7201 load/load 7202 atomic/store/store 7203 atomic/atomicrmw. 7204 - Ensures that 7205 following loads 7206 will not see stale 7207 global data. This 7208 satisfies the 7209 requirements of 7210 acquire. 7211 7212 **Sequential Consistent Atomic** 7213 ------------------------------------------------------------------------------------ 7214 load atomic seq_cst - singlethread - global *Same as corresponding 7215 - wavefront - local load atomic acquire, 7216 - generic except must generated 7217 all instructions even 7218 for OpenCL.* 7219 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 7220 - generic vmcnt(0) & vscnt(0) 7221 7222 - If CU wavefront execution 7223 mode, omit vmcnt(0) and 7224 vscnt(0). 7225 - Could be split into 7226 separate s_waitcnt 7227 vmcnt(0), s_waitcnt 7228 vscnt(0), and s_waitcnt 7229 lgkmcnt(0) to allow 7230 them to be 7231 independently moved 7232 according to the 7233 following rules. 7234 - s_waitcnt lgkmcnt(0) must 7235 happen after 7236 preceding 7237 local/generic load 7238 atomic/store 7239 atomic/atomicrmw 7240 with memory 7241 ordering of seq_cst 7242 and with equal or 7243 wider sync scope. 7244 (Note that seq_cst 7245 fences have their 7246 own s_waitcnt 7247 lgkmcnt(0) and so do 7248 not need to be 7249 considered.) 7250 - s_waitcnt vmcnt(0) 7251 must happen after 7252 preceding 7253 global/generic load 7254 atomic/ 7255 atomicrmw-with-return-value 7256 with memory 7257 ordering of seq_cst 7258 and with equal or 7259 wider sync scope. 7260 (Note that seq_cst 7261 fences have their 7262 own s_waitcnt 7263 vmcnt(0) and so do 7264 not need to be 7265 considered.) 7266 - s_waitcnt vscnt(0) 7267 Must happen after 7268 preceding 7269 global/generic store 7270 atomic/ 7271 atomicrmw-no-return-value 7272 with memory 7273 ordering of seq_cst 7274 and with equal or 7275 wider sync scope. 7276 (Note that seq_cst 7277 fences have their 7278 own s_waitcnt 7279 vscnt(0) and so do 7280 not need to be 7281 considered.) 7282 - Ensures any 7283 preceding 7284 sequential 7285 consistent global/local 7286 memory instructions 7287 have completed 7288 before executing 7289 this sequentially 7290 consistent 7291 instruction. This 7292 prevents reordering 7293 a seq_cst store 7294 followed by a 7295 seq_cst load. (Note 7296 that seq_cst is 7297 stronger than 7298 acquire/release as 7299 the reordering of 7300 load acquire 7301 followed by a store 7302 release is 7303 prevented by the 7304 s_waitcnt of 7305 the release, but 7306 there is nothing 7307 preventing a store 7308 release followed by 7309 load acquire from 7310 completing out of 7311 order. The s_waitcnt 7312 could be placed after 7313 seq_store or before 7314 the seq_load. We 7315 choose the load to 7316 make the s_waitcnt be 7317 as late as possible 7318 so that the store 7319 may have already 7320 completed.) 7321 7322 2. *Following 7323 instructions same as 7324 corresponding load 7325 atomic acquire, 7326 except must generated 7327 all instructions even 7328 for OpenCL.* 7329 load atomic seq_cst - workgroup - local 7330 7331 1. s_waitcnt vmcnt(0) & vscnt(0) 7332 7333 - If CU wavefront execution 7334 mode, omit. 7335 - Could be split into 7336 separate s_waitcnt 7337 vmcnt(0) and s_waitcnt 7338 vscnt(0) to allow 7339 them to be 7340 independently moved 7341 according to the 7342 following rules. 7343 - s_waitcnt vmcnt(0) 7344 Must happen after 7345 preceding 7346 global/generic load 7347 atomic/ 7348 atomicrmw-with-return-value 7349 with memory 7350 ordering of seq_cst 7351 and with equal or 7352 wider sync scope. 7353 (Note that seq_cst 7354 fences have their 7355 own s_waitcnt 7356 vmcnt(0) and so do 7357 not need to be 7358 considered.) 7359 - s_waitcnt vscnt(0) 7360 Must happen after 7361 preceding 7362 global/generic store 7363 atomic/ 7364 atomicrmw-no-return-value 7365 with memory 7366 ordering of seq_cst 7367 and with equal or 7368 wider sync scope. 7369 (Note that seq_cst 7370 fences have their 7371 own s_waitcnt 7372 vscnt(0) and so do 7373 not need to be 7374 considered.) 7375 - Ensures any 7376 preceding 7377 sequential 7378 consistent global 7379 memory instructions 7380 have completed 7381 before executing 7382 this sequentially 7383 consistent 7384 instruction. This 7385 prevents reordering 7386 a seq_cst store 7387 followed by a 7388 seq_cst load. (Note 7389 that seq_cst is 7390 stronger than 7391 acquire/release as 7392 the reordering of 7393 load acquire 7394 followed by a store 7395 release is 7396 prevented by the 7397 s_waitcnt of 7398 the release, but 7399 there is nothing 7400 preventing a store 7401 release followed by 7402 load acquire from 7403 completing out of 7404 order. The s_waitcnt 7405 could be placed after 7406 seq_store or before 7407 the seq_load. We 7408 choose the load to 7409 make the s_waitcnt be 7410 as late as possible 7411 so that the store 7412 may have already 7413 completed.) 7414 7415 2. *Following 7416 instructions same as 7417 corresponding load 7418 atomic acquire, 7419 except must generated 7420 all instructions even 7421 for OpenCL.* 7422 7423 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 7424 - system - generic vmcnt(0) & vscnt(0) 7425 7426 - Could be split into 7427 separate s_waitcnt 7428 vmcnt(0), s_waitcnt 7429 vscnt(0) and s_waitcnt 7430 lgkmcnt(0) to allow 7431 them to be 7432 independently moved 7433 according to the 7434 following rules. 7435 - s_waitcnt lgkmcnt(0) 7436 must happen after 7437 preceding 7438 local load 7439 atomic/store 7440 atomic/atomicrmw 7441 with memory 7442 ordering of seq_cst 7443 and with equal or 7444 wider sync scope. 7445 (Note that seq_cst 7446 fences have their 7447 own s_waitcnt 7448 lgkmcnt(0) and so do 7449 not need to be 7450 considered.) 7451 - s_waitcnt vmcnt(0) 7452 must happen after 7453 preceding 7454 global/generic load 7455 atomic/ 7456 atomicrmw-with-return-value 7457 with memory 7458 ordering of seq_cst 7459 and with equal or 7460 wider sync scope. 7461 (Note that seq_cst 7462 fences have their 7463 own s_waitcnt 7464 vmcnt(0) and so do 7465 not need to be 7466 considered.) 7467 - s_waitcnt vscnt(0) 7468 Must happen after 7469 preceding 7470 global/generic store 7471 atomic/ 7472 atomicrmw-no-return-value 7473 with memory 7474 ordering of seq_cst 7475 and with equal or 7476 wider sync scope. 7477 (Note that seq_cst 7478 fences have their 7479 own s_waitcnt 7480 vscnt(0) and so do 7481 not need to be 7482 considered.) 7483 - Ensures any 7484 preceding 7485 sequential 7486 consistent global 7487 memory instructions 7488 have completed 7489 before executing 7490 this sequentially 7491 consistent 7492 instruction. This 7493 prevents reordering 7494 a seq_cst store 7495 followed by a 7496 seq_cst load. (Note 7497 that seq_cst is 7498 stronger than 7499 acquire/release as 7500 the reordering of 7501 load acquire 7502 followed by a store 7503 release is 7504 prevented by the 7505 s_waitcnt of 7506 the release, but 7507 there is nothing 7508 preventing a store 7509 release followed by 7510 load acquire from 7511 completing out of 7512 order. The s_waitcnt 7513 could be placed after 7514 seq_store or before 7515 the seq_load. We 7516 choose the load to 7517 make the s_waitcnt be 7518 as late as possible 7519 so that the store 7520 may have already 7521 completed.) 7522 7523 2. *Following 7524 instructions same as 7525 corresponding load 7526 atomic acquire, 7527 except must generated 7528 all instructions even 7529 for OpenCL.* 7530 store atomic seq_cst - singlethread - global *Same as corresponding 7531 - wavefront - local store atomic release, 7532 - workgroup - generic except must generated 7533 - agent all instructions even 7534 - system for OpenCL.* 7535 atomicrmw seq_cst - singlethread - global *Same as corresponding 7536 - wavefront - local atomicrmw acq_rel, 7537 - workgroup - generic except must generated 7538 - agent all instructions even 7539 - system for OpenCL.* 7540 fence seq_cst - singlethread *none* *Same as corresponding 7541 - wavefront fence acq_rel, 7542 - workgroup except must generated 7543 - agent all instructions even 7544 - system for OpenCL.* 7545 ============ ============ ============== ========== ================================ 7546 7547Trap Handler ABI 7548~~~~~~~~~~~~~~~~ 7549 7550For code objects generated by AMDGPU backend for HSA [HSA]_ compatible runtimes 7551(such as ROCm [AMD-ROCm]_), the runtime installs a trap handler that supports 7552the ``s_trap`` instruction with the following usage: 7553 7554 .. table:: AMDGPU Trap Handler for AMDHSA OS 7555 :name: amdgpu-trap-handler-for-amdhsa-os-table 7556 7557 =================== =============== =============== ======================= 7558 Usage Code Sequence Trap Handler Description 7559 Inputs 7560 =================== =============== =============== ======================= 7561 reserved ``s_trap 0x00`` Reserved by hardware. 7562 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for HSA 7563 ``queue_ptr`` ``debugtrap`` 7564 ``VGPR0``: intrinsic (not 7565 ``arg`` implemented). 7566 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes dispatch to be 7567 ``queue_ptr`` terminated and its 7568 associated queue put 7569 into the error state. 7570 ``llvm.debugtrap`` ``s_trap 0x03`` - If debugger not 7571 installed then 7572 behaves as a 7573 no-operation. The 7574 trap handler is 7575 entered and 7576 immediately returns 7577 to continue 7578 execution of the 7579 wavefront. 7580 - If the debugger is 7581 installed, causes 7582 the debug trap to be 7583 reported by the 7584 debugger and the 7585 wavefront is put in 7586 the halt state until 7587 resumed by the 7588 debugger. 7589 reserved ``s_trap 0x04`` Reserved. 7590 reserved ``s_trap 0x05`` Reserved. 7591 reserved ``s_trap 0x06`` Reserved. 7592 debugger breakpoint ``s_trap 0x07`` Reserved for debugger 7593 breakpoints. 7594 reserved ``s_trap 0x08`` Reserved. 7595 reserved ``s_trap 0xfe`` Reserved. 7596 reserved ``s_trap 0xff`` Reserved. 7597 =================== =============== =============== ======================= 7598 7599.. _amdgpu-amdhsa-function-call-convention: 7600 7601Call Convention 7602~~~~~~~~~~~~~~~ 7603 7604.. note:: 7605 7606 This section is currently incomplete and has inaccuracies. It is WIP that will 7607 be updated as information is determined. 7608 7609See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 7610addresses. Unswizzled addresses are normal linear addresses. 7611 7612.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 7613 7614Kernel Functions 7615++++++++++++++++ 7616 7617This section describes the call convention ABI for the outer kernel function. 7618 7619See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 7620convention. 7621 7622The following is not part of the AMDGPU kernel calling convention but describes 7623how the AMDGPU implements function calls: 7624 76251. Clang decides the kernarg layout to match the *HSA Programmer's Language 7626 Reference* [HSA]_. 7627 7628 - All structs are passed directly. 7629 - Lambda values are passed *TBA*. 7630 7631 .. TODO:: 7632 7633 - Does this really follow HSA rules? Or are structs >16 bytes passed 7634 by-value struct? 7635 - What is ABI for lambda values? 7636 76374. The kernel performs certain setup in its prolog, as described in 7638 :ref:`amdgpu-amdhsa-kernel-prolog`. 7639 7640.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 7641 7642Non-Kernel Functions 7643++++++++++++++++++++ 7644 7645This section describes the call convention ABI for functions other than the 7646outer kernel function. 7647 7648If a kernel has function calls then scratch is always allocated and used for 7649the call stack which grows from low address to high address using the swizzled 7650scratch address space. 7651 7652On entry to a function: 7653 76541. SGPR0-3 contain a V# with the following properties (see 7655 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 7656 7657 * Base address pointing to the beginning of the wavefront scratch backing 7658 memory. 7659 * Swizzled with dword element size and stride of wavefront size elements. 7660 76612. The FLAT_SCRATCH register pair is setup. See 7662 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 76633. GFX6-8: M0 register set to the size of LDS in bytes. See 7664 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 76654. The EXEC register is set to the lanes active on entry to the function. 76665. MODE register: *TBD* 76676. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 7668 below. 76697. SGPR30-31 return address (RA). The code address that the function must 7670 return to when it completes. The value is undefined if the function is *no 7671 return*. 76728. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 7673 offset relative to the beginning of the wavefront scratch backing memory. 7674 7675 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 7676 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 7677 manner. 7678 7679 The unswizzled SP value can be converted into the swizzled SP value by: 7680 7681 | swizzled SP = unswizzled SP / wavefront size 7682 7683 This may be used to obtain the private address space address of stack 7684 objects and to convert this address to a flat address by adding the flat 7685 scratch aperture base address. 7686 7687 The swizzled SP value is always 4 bytes aligned for the ``r600`` 7688 architecture and 16 byte aligned for the ``amdgcn`` architecture. 7689 7690 .. note:: 7691 7692 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 7693 OpenCL language which has the largest base type defined as 16 bytes. 7694 7695 On entry, the swizzled SP value is the address of the first function 7696 argument passed on the stack. Other stack passed arguments are positive 7697 offsets from the entry swizzled SP value. 7698 7699 The function may use positive offsets beyond the last stack passed argument 7700 for stack allocated local variables and register spill slots. If necessary, 7701 the function may align these to greater alignment than 16 bytes. After these 7702 the function may dynamically allocate space for such things as runtime sized 7703 ``alloca`` local allocations. 7704 7705 If the function calls another function, it will place any stack allocated 7706 arguments after the last local allocation and adjust SGPR32 to the address 7707 after the last local allocation. 7708 77099. All other registers are unspecified. 771010. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 7711 to the function. 7712 7713On exit from a function: 7714 77151. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 7716 described below. Any registers used are considered clobbered registers. 77172. The following registers are preserved and have the same value as on entry: 7718 7719 * FLAT_SCRATCH 7720 * EXEC 7721 * GFX6-8: M0 7722 * All SGPR registers except the clobbered registers of SGPR4-31. 7723 * VGPR40-47 7724 VGPR56-63 7725 VGPR72-79 7726 VGPR88-95 7727 VGPR104-111 7728 VGPR120-127 7729 VGPR136-143 7730 VGPR152-159 7731 VGPR168-175 7732 VGPR184-191 7733 VGPR200-207 7734 VGPR216-223 7735 VGPR232-239 7736 VGPR248-255 7737 7738 *Except the argument registers, the VGPR clobbered and the preserved 7739 registers are intermixed at regular intervals in order to 7740 get a better occupancy.* 7741 7742 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 7743 optimization may mark some of clobbered SGPR and VGPR registers as 7744 preserved if it can be determined that the called function does not change 7745 their value. 7746 77472. The PC is set to the RA provided on entry. 77483. MODE register: *TBD*. 77494. All other registers are clobbered. 77505. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 7751 function is available to the caller. 7752 7753.. TODO:: 7754 7755 - On gfx908 are all ACC registers clobbered? 7756 7757 - How are function results returned? The address of structured types is passed 7758 by reference, but what about other types? 7759 7760The function input arguments are made up of the formal arguments explicitly 7761declared by the source language function plus the implicit input arguments used 7762by the implementation. 7763 7764The source language input arguments are: 7765 77661. Any source language implicit ``this`` or ``self`` argument comes first as a 7767 pointer type. 77682. Followed by the function formal arguments in left to right source order. 7769 7770The source language result arguments are: 7771 77721. The function result argument. 7773 7774The source language input or result struct type arguments that are less than or 7775equal to 16 bytes, are decomposed recursively into their base type fields, and 7776each field is passed as if a separate argument. For input arguments, if the 7777called function requires the struct to be in memory, for example because its 7778address is taken, then the function body is responsible for allocating a stack 7779location and copying the field arguments into it. Clang terms this *direct 7780struct*. 7781 7782The source language input struct type arguments that are greater than 16 bytes, 7783are passed by reference. The caller is responsible for allocating a stack 7784location to make a copy of the struct value and pass the address as the input 7785argument. The called function is responsible to perform the dereference when 7786accessing the input argument. Clang terms this *by-value struct*. 7787 7788A source language result struct type argument that is greater than 16 bytes, is 7789returned by reference. The caller is responsible for allocating a stack location 7790to hold the result value and passes the address as the last input argument 7791(before the implicit input arguments). In this case there are no result 7792arguments. The called function is responsible to perform the dereference when 7793storing the result value. Clang terms this *structured return (sret)*. 7794 7795*TODO: correct the ``sret`` definition.* 7796 7797.. TODO:: 7798 7799 Is this definition correct? Or is ``sret`` only used if passing in registers, and 7800 pass as non-decomposed struct as stack argument? Or something else? Is the 7801 memory location in the caller stack frame, or a stack memory argument and so 7802 no address is passed as the caller can directly write to the argument stack 7803 location? But then the stack location is still live after return. If an 7804 argument stack location is it the first stack argument or the last one? 7805 7806Lambda argument types are treated as struct types with an implementation defined 7807set of fields. 7808 7809.. TODO:: 7810 7811 Need to specify the ABI for lambda types for AMDGPU. 7812 7813For AMDGPU backend all source language arguments (including the decomposed 7814struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 7815they are passed in SGPRs. 7816 7817The AMDGPU backend walks the function call graph from the leaves to determine 7818which implicit input arguments are used, propagating to each caller of the 7819function. The used implicit arguments are appended to the function arguments 7820after the source language arguments in the following order: 7821 7822.. TODO:: 7823 7824 Is recursion or external functions supported? 7825 78261. Work-Item ID (1 VGPR) 7827 7828 The X, Y and Z work-item ID are packed into a single VGRP with the following 7829 layout. Only fields actually used by the function are set. The other bits 7830 are undefined. 7831 7832 The values come from the initial kernel execution state. See 7833 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-table`. 7834 7835 .. table:: Work-item implicit argument layout 7836 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 7837 7838 ======= ======= ============== 7839 Bits Size Field Name 7840 ======= ======= ============== 7841 9:0 10 bits X Work-Item ID 7842 19:10 10 bits Y Work-Item ID 7843 29:20 10 bits Z Work-Item ID 7844 31:30 2 bits Unused 7845 ======= ======= ============== 7846 78472. Dispatch Ptr (2 SGPRs) 7848 7849 The value comes from the initial kernel execution state. See 7850 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7851 78523. Queue Ptr (2 SGPRs) 7853 7854 The value comes from the initial kernel execution state. See 7855 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7856 78574. Kernarg Segment Ptr (2 SGPRs) 7858 7859 The value comes from the initial kernel execution state. See 7860 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7861 78625. Dispatch id (2 SGPRs) 7863 7864 The value comes from the initial kernel execution state. See 7865 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7866 78676. Work-Group ID X (1 SGPR) 7868 7869 The value comes from the initial kernel execution state. See 7870 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7871 78727. Work-Group ID Y (1 SGPR) 7873 7874 The value comes from the initial kernel execution state. See 7875 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7876 78778. Work-Group ID Z (1 SGPR) 7878 7879 The value comes from the initial kernel execution state. See 7880 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 7881 78829. Implicit Argument Ptr (2 SGPRs) 7883 7884 The value is computed by adding an offset to Kernarg Segment Ptr to get the 7885 global address space pointer to the first kernarg implicit argument. 7886 7887The input and result arguments are assigned in order in the following manner: 7888 7889.. note:: 7890 7891 There are likely some errors and omissions in the following description that 7892 need correction. 7893 7894 .. TODO:: 7895 7896 Check the clang source code to decipher how function arguments and return 7897 results are handled. Also see the AMDGPU specific values used. 7898 7899* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 7900 VGPR31. 7901 7902 If there are more arguments than will fit in these registers, the remaining 7903 arguments are allocated on the stack in order on naturally aligned 7904 addresses. 7905 7906 .. TODO:: 7907 7908 How are overly aligned structures allocated on the stack? 7909 7910* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 7911 SGPR29. 7912 7913 If there are more arguments than will fit in these registers, the remaining 7914 arguments are allocated on the stack in order on naturally aligned 7915 addresses. 7916 7917Note that decomposed struct type arguments may have some fields passed in 7918registers and some in memory. 7919 7920.. TODO:: 7921 7922 So, a struct which can pass some fields as decomposed register arguments, will 7923 pass the rest as decomposed stack elements? But an argument that will not start 7924 in registers will not be decomposed and will be passed as a non-decomposed 7925 stack value? 7926 7927The following is not part of the AMDGPU function calling convention but 7928describes how the AMDGPU implements function calls: 7929 79301. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 7931 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 7932 are used, or for the reasons defined in ``SIFrameLowering``. 79332. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 7934 to access the incoming stack arguments in the function. The BP is needed 7935 only when the function requires the runtime stack alignment. 7936 79373. Allocating SGPR arguments on the stack are not supported. 7938 79394. No CFI is currently generated. See 7940 :ref:`amdgpu-dwarf-call-frame-information`. 7941 7942 .. note:: 7943 7944 CFI will be generated that defines the CFA as the unswizzled address 7945 relative to the wave scratch base in the unswizzled private address space 7946 of the lowest address stack allocated local variable. 7947 7948 ``DW_AT_frame_base`` will be defined as the swizzled address in the 7949 swizzled private address space by dividing the CFA by the wavefront size 7950 (since CFA is always at least dword aligned which matches the scratch 7951 swizzle element size). 7952 7953 If no dynamic stack alignment was performed, the stack allocated arguments 7954 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 7955 local variables and register spill slots are accessed as positive offsets 7956 relative to ``DW_AT_frame_base``. 7957 79585. Function argument passing is implemented by copying the input physical 7959 registers to virtual registers on entry. The register allocator can spill if 7960 necessary. These are copied back to physical registers at call sites. The 7961 net effect is that each function call can have these values in entirely 7962 distinct locations. The IPRA can help avoid shuffling argument registers. 79636. Call sites are implemented by setting up the arguments at positive offsets 7964 from SP. Then SP is incremented to account for the known frame size before 7965 the call and decremented after the call. 7966 7967 .. note:: 7968 7969 The CFI will reflect the changed calculation needed to compute the CFA 7970 from SP. 7971 79727. 4 byte spill slots are used in the stack frame. One slot is allocated for an 7973 emergency spill slot. Buffer instructions are used for stack accesses and 7974 not the ``flat_scratch`` instruction. 7975 7976 .. TODO:: 7977 7978 Explain when the emergency spill slot is used. 7979 7980.. TODO:: 7981 7982 Possible broken issues: 7983 7984 - Stack arguments must be aligned to required alignment. 7985 - Stack is aligned to max(16, max formal argument alignment) 7986 - Direct argument < 64 bits should check register budget. 7987 - Register budget calculation should respect ``inreg`` for SGPR. 7988 - SGPR overflow is not handled. 7989 - struct with 1 member unpeeling is not checking size of member. 7990 - ``sret`` is after ``this`` pointer. 7991 - Caller is not implementing stack realignment: need an extra pointer. 7992 - Should say AMDGPU passes FP rather than SP. 7993 - Should CFI define CFA as address of locals or arguments. Difference is 7994 apparent when have implemented dynamic alignment. 7995 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 7996 highest address of stack frame and use negative offset for locals. Would 7997 allow SP to be the same as FP and could support signal-handler-like as now 7998 have a real SP for the top of the stack. 7999 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 8000 arguments? 8001 8002AMDPAL 8003------ 8004 8005This section provides code conventions used when the target triple OS is 8006``amdpal`` (see :ref:`amdgpu-target-triples`) for passing runtime parameters 8007from the application/runtime to each invocation of a hardware shader. These 8008parameters include both generic, application-controlled parameters called 8009*user data* as well as system-generated parameters that are a product of the 8010draw or dispatch execution. 8011 8012User Data 8013~~~~~~~~~ 8014 8015Each hardware stage has a set of 32-bit *user data registers* which can be 8016written from a command buffer and then loaded into SGPRs when waves are launched 8017via a subsequent dispatch or draw operation. This is the way most arguments are 8018passed from the application/runtime to a hardware shader. 8019 8020Compute User Data 8021~~~~~~~~~~~~~~~~~ 8022 8023Compute shader user data mappings are simpler than graphics shaders and have a 8024fixed mapping. 8025 8026Note that there are always 10 available *user data entries* in registers - 8027entries beyond that limit must be fetched from memory (via the spill table 8028pointer) by the shader. 8029 8030 .. table:: PAL Compute Shader User Data Registers 8031 :name: pal-compute-user-data-registers 8032 8033 ============= ================================ 8034 User Register Description 8035 ============= ================================ 8036 0 Global Internal Table (32-bit pointer) 8037 1 Per-Shader Internal Table (32-bit pointer) 8038 2 - 11 Application-Controlled User Data (10 32-bit values) 8039 12 Spill Table (32-bit pointer) 8040 13 - 14 Thread Group Count (64-bit pointer) 8041 15 GDS Range 8042 ============= ================================ 8043 8044Graphics User Data 8045~~~~~~~~~~~~~~~~~~ 8046 8047Graphics pipelines support a much more flexible user data mapping: 8048 8049 .. table:: PAL Graphics Shader User Data Registers 8050 :name: pal-graphics-user-data-registers 8051 8052 ============= ================================ 8053 User Register Description 8054 ============= ================================ 8055 0 Global Internal Table (32-bit pointer) 8056 + Per-Shader Internal Table (32-bit pointer) 8057 + 1-15 Application Controlled User Data 8058 (1-15 Contiguous 32-bit Values in Registers) 8059 + Spill Table (32-bit pointer) 8060 + Draw Index (First Stage Only) 8061 + Vertex Offset (First Stage Only) 8062 + Instance Offset (First Stage Only) 8063 ============= ================================ 8064 8065 The placement of the global internal table remains fixed in the first *user 8066 data SGPR register*. Otherwise all parameters are optional, and can be mapped 8067 to any desired *user data SGPR register*, with the following restrictions: 8068 8069 * Draw Index, Vertex Offset, and Instance Offset can only be used by the first 8070 active hardware stage in a graphics pipeline (i.e. where the API vertex 8071 shader runs). 8072 8073 * Application-controlled user data must be mapped into a contiguous range of 8074 user data registers. 8075 8076 * The application-controlled user data range supports compaction remapping, so 8077 only *entries* that are actually consumed by the shader must be assigned to 8078 corresponding *registers*. Note that in order to support an efficient runtime 8079 implementation, the remapping must pack *registers* in the same order as 8080 *entries*, with unused *entries* removed. 8081 8082.. _pal_global_internal_table: 8083 8084Global Internal Table 8085~~~~~~~~~~~~~~~~~~~~~ 8086 8087The global internal table is a table of *shader resource descriptors* (SRDs) 8088that define how certain engine-wide, runtime-managed resources should be 8089accessed from a shader. The majority of these resources have HW-defined formats, 8090and it is up to the compiler to write/read data as required by the target 8091hardware. 8092 8093The following table illustrates the required format: 8094 8095 .. table:: PAL Global Internal Table 8096 :name: pal-git-table 8097 8098 ============= ================================ 8099 Offset Description 8100 ============= ================================ 8101 0-3 Graphics Scratch SRD 8102 4-7 Compute Scratch SRD 8103 8-11 ES/GS Ring Output SRD 8104 12-15 ES/GS Ring Input SRD 8105 16-19 GS/VS Ring Output #0 8106 20-23 GS/VS Ring Output #1 8107 24-27 GS/VS Ring Output #2 8108 28-31 GS/VS Ring Output #3 8109 32-35 GS/VS Ring Input SRD 8110 36-39 Tessellation Factor Buffer SRD 8111 40-43 Off-Chip LDS Buffer SRD 8112 44-47 Off-Chip Param Cache Buffer SRD 8113 48-51 Sample Position Buffer SRD 8114 52 vaRange::ShadowDescriptorTable High Bits 8115 ============= ================================ 8116 8117 The pointer to the global internal table passed to the shader as user data 8118 is a 32-bit pointer. The top 32 bits should be assumed to be the same as 8119 the top 32 bits of the pipeline, so the shader may use the program 8120 counter's top 32 bits. 8121 8122.. _pal_call-convention: 8123 8124Call Convention 8125~~~~~~~~~~~~~~~ 8126 8127For graphics use cases, the calling convention is `amdgpu_gfx`. 8128 8129.. note:: 8130 8131 `amdgpu_gfx` Function calls are currently in development and are 8132 subject to major changes. 8133 8134This calling convention shares most properties with calling non-kernel 8135functions (see 8136:ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`). 8137Differences are: 8138 8139 - Currently there are none, differences will be listed here 8140 8141Unspecified OS 8142-------------- 8143 8144This section provides code conventions used when the target triple OS is 8145empty (see :ref:`amdgpu-target-triples`). 8146 8147Trap Handler ABI 8148~~~~~~~~~~~~~~~~ 8149 8150For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 8151not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 8152instructions are handled as follows: 8153 8154 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 8155 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 8156 8157 =============== =============== =========================================== 8158 Usage Code Sequence Description 8159 =============== =============== =========================================== 8160 llvm.trap s_endpgm Causes wavefront to be terminated. 8161 llvm.debugtrap *none* Compiler warning given that there is no 8162 trap handler installed. 8163 =============== =============== =========================================== 8164 8165Source Languages 8166================ 8167 8168.. _amdgpu-opencl: 8169 8170OpenCL 8171------ 8172 8173When the language is OpenCL the following differences occur: 8174 81751. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 81762. The AMDGPU backend appends additional arguments to the kernel's explicit 8177 arguments for the AMDHSA OS (see 8178 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 81793. Additional metadata is generated 8180 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 8181 8182 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 8183 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 8184 8185 ======== ==== ========= =========================================== 8186 Position Byte Byte Description 8187 Size Alignment 8188 ======== ==== ========= =========================================== 8189 1 8 8 OpenCL Global Offset X 8190 2 8 8 OpenCL Global Offset Y 8191 3 8 8 OpenCL Global Offset Z 8192 4 8 8 OpenCL address of printf buffer 8193 5 8 8 OpenCL address of virtual queue used by 8194 enqueue_kernel. 8195 6 8 8 OpenCL address of AqlWrap struct used by 8196 enqueue_kernel. 8197 7 8 8 Pointer argument used for Multi-gird 8198 synchronization. 8199 ======== ==== ========= =========================================== 8200 8201.. _amdgpu-hcc: 8202 8203HCC 8204--- 8205 8206When the language is HCC the following differences occur: 8207 82081. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 8209 8210.. _amdgpu-assembler: 8211 8212Assembler 8213--------- 8214 8215AMDGPU backend has LLVM-MC based assembler which is currently in development. 8216It supports AMDGCN GFX6-GFX10. 8217 8218This section describes general syntax for instructions and operands. 8219 8220Instructions 8221~~~~~~~~~~~~ 8222 8223An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 8224 8225 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 8226 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 8227 8228:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 8229:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 8230 8231The order of operands and modifiers is fixed. 8232Most modifiers are optional and may be omitted. 8233 8234Links to detailed instruction syntax description may be found in the following 8235table. Note that features under development are not included 8236in this description. 8237 8238 =================================== ======================================= 8239 Core ISA ISA Extensions 8240 =================================== ======================================= 8241 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 8242 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 8243 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 8244 8245 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 8246 8247 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 8248 8249 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 8250 8251 :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 8252 8253 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 8254 8255 :doc:`GFX10<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 8256 8257 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 8258 =================================== ======================================= 8259 8260For more information about instructions, their semantics and supported 8261combinations of operands, refer to one of instruction set architecture manuals 8262[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, [AMD-GCN-GFX9]_ and 8263[AMD-GCN-GFX10]_. 8264 8265Operands 8266~~~~~~~~ 8267 8268Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 8269 8270Modifiers 8271~~~~~~~~~ 8272 8273Detailed description of modifiers may be found 8274:doc:`here<AMDGPUModifierSyntax>`. 8275 8276Instruction Examples 8277~~~~~~~~~~~~~~~~~~~~ 8278 8279DS 8280++ 8281 8282.. code-block:: nasm 8283 8284 ds_add_u32 v2, v4 offset:16 8285 ds_write_src2_b64 v2 offset0:4 offset1:8 8286 ds_cmpst_f32 v2, v4, v6 8287 ds_min_rtn_f64 v[8:9], v2, v[4:5] 8288 8289For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 8290Manual. 8291 8292FLAT 8293++++ 8294 8295.. code-block:: nasm 8296 8297 flat_load_dword v1, v[3:4] 8298 flat_store_dwordx3 v[3:4], v[5:7] 8299 flat_atomic_swap v1, v[3:4], v5 glc 8300 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 8301 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 8302 8303For full list of supported instructions, refer to "FLAT instructions" in ISA 8304Manual. 8305 8306MUBUF 8307+++++ 8308 8309.. code-block:: nasm 8310 8311 buffer_load_dword v1, off, s[4:7], s1 8312 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 8313 buffer_store_format_xy v[1:2], off, s[4:7], s1 8314 buffer_wbinvl1 8315 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 8316 8317For full list of supported instructions, refer to "MUBUF Instructions" in ISA 8318Manual. 8319 8320SMRD/SMEM 8321+++++++++ 8322 8323.. code-block:: nasm 8324 8325 s_load_dword s1, s[2:3], 0xfc 8326 s_load_dwordx8 s[8:15], s[2:3], s4 8327 s_load_dwordx16 s[88:103], s[2:3], s4 8328 s_dcache_inv_vol 8329 s_memtime s[4:5] 8330 8331For full list of supported instructions, refer to "Scalar Memory Operations" in 8332ISA Manual. 8333 8334SOP1 8335++++ 8336 8337.. code-block:: nasm 8338 8339 s_mov_b32 s1, s2 8340 s_mov_b64 s[0:1], 0x80000000 8341 s_cmov_b32 s1, 200 8342 s_wqm_b64 s[2:3], s[4:5] 8343 s_bcnt0_i32_b64 s1, s[2:3] 8344 s_swappc_b64 s[2:3], s[4:5] 8345 s_cbranch_join s[4:5] 8346 8347For full list of supported instructions, refer to "SOP1 Instructions" in ISA 8348Manual. 8349 8350SOP2 8351++++ 8352 8353.. code-block:: nasm 8354 8355 s_add_u32 s1, s2, s3 8356 s_and_b64 s[2:3], s[4:5], s[6:7] 8357 s_cselect_b32 s1, s2, s3 8358 s_andn2_b32 s2, s4, s6 8359 s_lshr_b64 s[2:3], s[4:5], s6 8360 s_ashr_i32 s2, s4, s6 8361 s_bfm_b64 s[2:3], s4, s6 8362 s_bfe_i64 s[2:3], s[4:5], s6 8363 s_cbranch_g_fork s[4:5], s[6:7] 8364 8365For full list of supported instructions, refer to "SOP2 Instructions" in ISA 8366Manual. 8367 8368SOPC 8369++++ 8370 8371.. code-block:: nasm 8372 8373 s_cmp_eq_i32 s1, s2 8374 s_bitcmp1_b32 s1, s2 8375 s_bitcmp0_b64 s[2:3], s4 8376 s_setvskip s3, s5 8377 8378For full list of supported instructions, refer to "SOPC Instructions" in ISA 8379Manual. 8380 8381SOPP 8382++++ 8383 8384.. code-block:: nasm 8385 8386 s_barrier 8387 s_nop 2 8388 s_endpgm 8389 s_waitcnt 0 ; Wait for all counters to be 0 8390 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 8391 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 8392 s_sethalt 9 8393 s_sleep 10 8394 s_sendmsg 0x1 8395 s_sendmsg sendmsg(MSG_INTERRUPT) 8396 s_trap 1 8397 8398For full list of supported instructions, refer to "SOPP Instructions" in ISA 8399Manual. 8400 8401Unless otherwise mentioned, little verification is performed on the operands 8402of SOPP Instructions, so it is up to the programmer to be familiar with the 8403range or acceptable values. 8404 8405VALU 8406++++ 8407 8408For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 8409the assembler will automatically use optimal encoding based on its operands. To 8410force specific encoding, one can add a suffix to the opcode of the instruction: 8411 8412* _e32 for 32-bit VOP1/VOP2/VOPC 8413* _e64 for 64-bit VOP3 8414* _dpp for VOP_DPP 8415* _sdwa for VOP_SDWA 8416 8417VOP1/VOP2/VOP3/VOPC examples: 8418 8419.. code-block:: nasm 8420 8421 v_mov_b32 v1, v2 8422 v_mov_b32_e32 v1, v2 8423 v_nop 8424 v_cvt_f64_i32_e32 v[1:2], v2 8425 v_floor_f32_e32 v1, v2 8426 v_bfrev_b32_e32 v1, v2 8427 v_add_f32_e32 v1, v2, v3 8428 v_mul_i32_i24_e64 v1, v2, 3 8429 v_mul_i32_i24_e32 v1, -3, v3 8430 v_mul_i32_i24_e32 v1, -100, v3 8431 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 8432 v_max_f16_e32 v1, v2, v3 8433 8434VOP_DPP examples: 8435 8436.. code-block:: nasm 8437 8438 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 8439 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 8440 v_mov_b32 v0, v0 wave_shl:1 8441 v_mov_b32 v0, v0 row_mirror 8442 v_mov_b32 v0, v0 row_bcast:31 8443 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 8444 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 8445 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 8446 8447VOP_SDWA examples: 8448 8449.. code-block:: nasm 8450 8451 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 8452 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 8453 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 8454 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 8455 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 8456 8457For full list of supported instructions, refer to "Vector ALU instructions". 8458 8459.. TODO:: 8460 8461 Remove once we switch to code object v3 by default. 8462 8463.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 8464 8465Code Object V2 Predefined Symbols (--amdhsa-code-object-version=2) 8466~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8467 8468.. warning:: Code Object V2 is not the default code object version emitted by 8469 this version of LLVM. For a description of the predefined symbols available 8470 with the default configuration (Code Object V3) see 8471 :ref:`amdgpu-amdhsa-assembler-predefined-symbols-v3`. 8472 8473The AMDGPU assembler defines and updates some symbols automatically. These 8474symbols do not affect code generation. 8475 8476.option.machine_version_major 8477+++++++++++++++++++++++++++++ 8478 8479Set to the GFX major generation number of the target being assembled for. For 8480example, when assembling for a "GFX9" target this will be set to the integer 8481value "9". The possible GFX major generation numbers are presented in 8482:ref:`amdgpu-processors`. 8483 8484.option.machine_version_minor 8485+++++++++++++++++++++++++++++ 8486 8487Set to the GFX minor generation number of the target being assembled for. For 8488example, when assembling for a "GFX810" target this will be set to the integer 8489value "1". The possible GFX minor generation numbers are presented in 8490:ref:`amdgpu-processors`. 8491 8492.option.machine_version_stepping 8493++++++++++++++++++++++++++++++++ 8494 8495Set to the GFX stepping generation number of the target being assembled for. 8496For example, when assembling for a "GFX704" target this will be set to the 8497integer value "4". The possible GFX stepping generation numbers are presented 8498in :ref:`amdgpu-processors`. 8499 8500.kernel.vgpr_count 8501++++++++++++++++++ 8502 8503Set to zero each time a 8504:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 8505encountered. At each instruction, if the current value of this symbol is less 8506than or equal to the maximum VGPR number explicitly referenced within that 8507instruction then the symbol value is updated to equal that VGPR number plus 8508one. 8509 8510.kernel.sgpr_count 8511++++++++++++++++++ 8512 8513Set to zero each time a 8514:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 8515encountered. At each instruction, if the current value of this symbol is less 8516than or equal to the maximum VGPR number explicitly referenced within that 8517instruction then the symbol value is updated to equal that SGPR number plus 8518one. 8519 8520.. _amdgpu-amdhsa-assembler-directives-v2: 8521 8522Code Object V2 Directives (--amdhsa-code-object-version=2) 8523~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8524 8525.. warning:: Code Object V2 is not the default code object version emitted by 8526 this version of LLVM. For a description of the directives supported with 8527 the default configuration (Code Object V3) see 8528 :ref:`amdgpu-amdhsa-assembler-directives-v3`. 8529 8530AMDGPU ABI defines auxiliary data in output code object. In assembly source, 8531one can specify them with assembler directives. 8532 8533.hsa_code_object_version major, minor 8534+++++++++++++++++++++++++++++++++++++ 8535 8536*major* and *minor* are integers that specify the version of the HSA code 8537object that will be generated by the assembler. 8538 8539.hsa_code_object_isa [major, minor, stepping, vendor, arch] 8540+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 8541 8542 8543*major*, *minor*, and *stepping* are all integers that describe the instruction 8544set architecture (ISA) version of the assembly program. 8545 8546*vendor* and *arch* are quoted strings. *vendor* should always be equal to 8547"AMD" and *arch* should always be equal to "AMDGPU". 8548 8549By default, the assembler will derive the ISA version, *vendor*, and *arch* 8550from the value of the -mcpu option that is passed to the assembler. 8551 8552.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 8553 8554.amdgpu_hsa_kernel (name) 8555+++++++++++++++++++++++++ 8556 8557This directives specifies that the symbol with given name is a kernel entry 8558point (label) and the object should contain corresponding symbol of type 8559STT_AMDGPU_HSA_KERNEL. 8560 8561.amd_kernel_code_t 8562++++++++++++++++++ 8563 8564This directive marks the beginning of a list of key / value pairs that are used 8565to specify the amd_kernel_code_t object that will be emitted by the assembler. 8566The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 8567amd_kernel_code_t values that are unspecified a default value will be used. The 8568default value for all keys is 0, with the following exceptions: 8569 8570- *amd_code_version_major* defaults to 1. 8571- *amd_kernel_code_version_minor* defaults to 2. 8572- *amd_machine_kind* defaults to 1. 8573- *amd_machine_version_major*, *machine_version_minor*, and 8574 *amd_machine_version_stepping* are derived from the value of the -mcpu option 8575 that is passed to the assembler. 8576- *kernel_code_entry_byte_offset* defaults to 256. 8577- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 8578 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 8579 Note that wavefront size is specified as a power of two, so a value of **n** 8580 means a size of 2^ **n**. 8581- *call_convention* defaults to -1. 8582- *kernarg_segment_alignment*, *group_segment_alignment*, and 8583 *private_segment_alignment* default to 4. Note that alignments are specified 8584 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 8585- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 8586 GFX10 onwards. 8587- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 8588 8589The *.amd_kernel_code_t* directive must be placed immediately after the 8590function label and before any instructions. 8591 8592For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 8593comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 8594 8595.. _amdgpu-amdhsa-assembler-example-v2: 8596 8597Code Object V2 Example Source Code (--amdhsa-code-object-version=2) 8598~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8599 8600.. warning:: Code Object V2 is not the default code object version emitted by 8601 this version of LLVM. For a description of the directives supported with 8602 the default configuration (Code Object V3) see 8603 :ref:`amdgpu-amdhsa-assembler-example-v3`. 8604 8605Here is an example of a minimal assembly source file, defining one HSA kernel: 8606 8607.. code:: 8608 :number-lines: 8609 8610 .hsa_code_object_version 1,0 8611 .hsa_code_object_isa 8612 8613 .hsatext 8614 .globl hello_world 8615 .p2align 8 8616 .amdgpu_hsa_kernel hello_world 8617 8618 hello_world: 8619 8620 .amd_kernel_code_t 8621 enable_sgpr_kernarg_segment_ptr = 1 8622 is_ptr64 = 1 8623 compute_pgm_rsrc1_vgprs = 0 8624 compute_pgm_rsrc1_sgprs = 0 8625 compute_pgm_rsrc2_user_sgpr = 2 8626 compute_pgm_rsrc1_wgp_mode = 0 8627 compute_pgm_rsrc1_mem_ordered = 0 8628 compute_pgm_rsrc1_fwd_progress = 1 8629 .end_amd_kernel_code_t 8630 8631 s_load_dwordx2 s[0:1], s[0:1] 0x0 8632 v_mov_b32 v0, 3.14159 8633 s_waitcnt lgkmcnt(0) 8634 v_mov_b32 v1, s0 8635 v_mov_b32 v2, s1 8636 flat_store_dword v[1:2], v0 8637 s_endpgm 8638 .Lfunc_end0: 8639 .size hello_world, .Lfunc_end0-hello_world 8640 8641.. _amdgpu-amdhsa-assembler-predefined-symbols-v3: 8642 8643Code Object V3 Predefined Symbols (--amdhsa-code-object-version=3) 8644~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8645 8646The AMDGPU assembler defines and updates some symbols automatically. These 8647symbols do not affect code generation. 8648 8649.amdgcn.gfx_generation_number 8650+++++++++++++++++++++++++++++ 8651 8652Set to the GFX major generation number of the target being assembled for. For 8653example, when assembling for a "GFX9" target this will be set to the integer 8654value "9". The possible GFX major generation numbers are presented in 8655:ref:`amdgpu-processors`. 8656 8657.amdgcn.gfx_generation_minor 8658++++++++++++++++++++++++++++ 8659 8660Set to the GFX minor generation number of the target being assembled for. For 8661example, when assembling for a "GFX810" target this will be set to the integer 8662value "1". The possible GFX minor generation numbers are presented in 8663:ref:`amdgpu-processors`. 8664 8665.amdgcn.gfx_generation_stepping 8666+++++++++++++++++++++++++++++++ 8667 8668Set to the GFX stepping generation number of the target being assembled for. 8669For example, when assembling for a "GFX704" target this will be set to the 8670integer value "4". The possible GFX stepping generation numbers are presented 8671in :ref:`amdgpu-processors`. 8672 8673.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 8674 8675.amdgcn.next_free_vgpr 8676++++++++++++++++++++++ 8677 8678Set to zero before assembly begins. At each instruction, if the current value 8679of this symbol is less than or equal to the maximum VGPR number explicitly 8680referenced within that instruction then the symbol value is updated to equal 8681that VGPR number plus one. 8682 8683May be used to set the `.amdhsa_next_free_vgpr` directive in 8684:ref:`amdhsa-kernel-directives-table`. 8685 8686May be set at any time, e.g. manually set to zero at the start of each kernel. 8687 8688.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 8689 8690.amdgcn.next_free_sgpr 8691++++++++++++++++++++++ 8692 8693Set to zero before assembly begins. At each instruction, if the current value 8694of this symbol is less than or equal the maximum SGPR number explicitly 8695referenced within that instruction then the symbol value is updated to equal 8696that SGPR number plus one. 8697 8698May be used to set the `.amdhsa_next_free_spgr` directive in 8699:ref:`amdhsa-kernel-directives-table`. 8700 8701May be set at any time, e.g. manually set to zero at the start of each kernel. 8702 8703.. _amdgpu-amdhsa-assembler-directives-v3: 8704 8705Code Object V3 Directives (--amdhsa-code-object-version=3) 8706~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8707 8708Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 8709architecture processors, and are not OS-specific. Directives which begin with 8710``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 8711``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 8712:ref:`amdgpu-processors`. 8713 8714.amdgcn_target <target> 8715+++++++++++++++++++++++ 8716 8717Optional directive which declares the target supported by the containing 8718assembler source file. Valid values are described in 8719:ref:`amdgpu-amdhsa-code-object-target-identification`. Used by the assembler 8720to validate command-line options such as ``-triple``, ``-mcpu``, and those 8721which specify target features. 8722 8723.amdhsa_kernel <name> 8724+++++++++++++++++++++ 8725 8726Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 8727``<name>.kd``, in the current location of the current section. Only valid when 8728the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 8729instruction to execute, and does not need to be previously defined. 8730 8731Marks the beginning of a list of directives used to generate the bytes of a 8732kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 8733Directives which may appear in this list are described in 8734:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 8735be valid for the target being assembled for, and cannot be repeated. Directives 8736support the range of values specified by the field they reference in 8737:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 8738assumed to have its default value, unless it is marked as "Required", in which 8739case it is an error to omit the directive. This list of directives is 8740terminated by an ``.end_amdhsa_kernel`` directive. 8741 8742 .. table:: AMDHSA Kernel Assembler Directives 8743 :name: amdhsa-kernel-directives-table 8744 8745 ======================================================== =================== ============ =================== 8746 Directive Default Supported On Description 8747 ======================================================== =================== ============ =================== 8748 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 8749 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8750 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 8751 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8752 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 8753 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8754 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 8755 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8756 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 8757 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8758 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 8759 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8760 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 8761 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8762 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 8763 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8764 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 8765 :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8766 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 8767 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8768 Specific 8769 (-wavefrontsize64) 8770 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_WAVEFRONT_OFFSET in 8771 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8772 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 8773 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8774 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 8775 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8776 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 8777 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8778 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 8779 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8780 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 8781 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8782 Possible values are defined in 8783 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 8784 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 8785 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 8786 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8787 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 8788 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 8789 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8790 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 8791 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 8792 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8793 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 8794 scratch memory. Used to calculate 8795 GRANULATED_WAVEFRONT_SGPR_COUNT in 8796 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8797 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 8798 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 8799 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8800 (+xnack) 8801 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 8802 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8803 Possible values are defined in 8804 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 8805 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 8806 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8807 Possible values are defined in 8808 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 8809 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 8810 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8811 Possible values are defined in 8812 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 8813 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 8814 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8815 Possible values are defined in 8816 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 8817 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 8818 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8819 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 8820 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8821 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 8822 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8823 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 8824 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-gfx6-gfx10-table`. 8825 Specific 8826 (-cumode) 8827 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 8828 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8829 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 8830 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 8831 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 8832 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8833 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 8834 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8835 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 8836 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8837 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 8838 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8839 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 8840 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8841 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 8842 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8843 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 8844 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 8845 ======================================================== =================== ============ =================== 8846 8847.amdgpu_metadata 8848++++++++++++++++ 8849 8850Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 8851note record (see :ref:`amdgpu-elf-note-records-table-v3`). 8852 8853The contents must be in the [YAML]_ markup format, with the same structure and 8854semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`. 8855 8856This directive is terminated by an ``.end_amdgpu_metadata`` directive. 8857 8858.. _amdgpu-amdhsa-assembler-example-v3: 8859 8860Code Object V3 Example Source Code (--amdhsa-code-object-version=3) 8861~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 8862 8863Here is an example of a minimal assembly source file, defining one HSA kernel: 8864 8865.. code:: 8866 :number-lines: 8867 8868 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 8869 8870 .text 8871 .globl hello_world 8872 .p2align 8 8873 .type hello_world,@function 8874 hello_world: 8875 s_load_dwordx2 s[0:1], s[0:1] 0x0 8876 v_mov_b32 v0, 3.14159 8877 s_waitcnt lgkmcnt(0) 8878 v_mov_b32 v1, s0 8879 v_mov_b32 v2, s1 8880 flat_store_dword v[1:2], v0 8881 s_endpgm 8882 .Lfunc_end0: 8883 .size hello_world, .Lfunc_end0-hello_world 8884 8885 .rodata 8886 .p2align 6 8887 .amdhsa_kernel hello_world 8888 .amdhsa_user_sgpr_kernarg_segment_ptr 1 8889 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 8890 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 8891 .end_amdhsa_kernel 8892 8893 .amdgpu_metadata 8894 --- 8895 amdhsa.version: 8896 - 1 8897 - 0 8898 amdhsa.kernels: 8899 - .name: hello_world 8900 .symbol: hello_world.kd 8901 .kernarg_segment_size: 48 8902 .group_segment_fixed_size: 0 8903 .private_segment_fixed_size: 0 8904 .kernarg_segment_align: 4 8905 .wavefront_size: 64 8906 .sgpr_count: 2 8907 .vgpr_count: 3 8908 .max_flat_workgroup_size: 256 8909 ... 8910 .end_amdgpu_metadata 8911 8912If an assembly source file contains multiple kernels and/or functions, the 8913:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 8914:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 8915the ``.set <symbol>, <expression>`` directive. For example, in the case of two 8916kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 8917to group the function with the kernel that calls it and reset the symbols 8918between the two connected components: 8919 8920.. code:: 8921 :number-lines: 8922 8923 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 8924 8925 // gpr tracking symbols are implicitly set to zero 8926 8927 .text 8928 .globl kern0 8929 .p2align 8 8930 .type kern0,@function 8931 kern0: 8932 // ... 8933 s_endpgm 8934 .Lkern0_end: 8935 .size kern0, .Lkern0_end-kern0 8936 8937 .rodata 8938 .p2align 6 8939 .amdhsa_kernel kern0 8940 // ... 8941 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 8942 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 8943 .end_amdhsa_kernel 8944 8945 // reset symbols to begin tracking usage in func1 and kern1 8946 .set .amdgcn.next_free_vgpr, 0 8947 .set .amdgcn.next_free_sgpr, 0 8948 8949 .text 8950 .hidden func1 8951 .global func1 8952 .p2align 2 8953 .type func1,@function 8954 func1: 8955 // ... 8956 s_setpc_b64 s[30:31] 8957 .Lfunc1_end: 8958 .size func1, .Lfunc1_end-func1 8959 8960 .globl kern1 8961 .p2align 8 8962 .type kern1,@function 8963 kern1: 8964 // ... 8965 s_getpc_b64 s[4:5] 8966 s_add_u32 s4, s4, func1@rel32@lo+4 8967 s_addc_u32 s5, s5, func1@rel32@lo+4 8968 s_swappc_b64 s[30:31], s[4:5] 8969 // ... 8970 s_endpgm 8971 .Lkern1_end: 8972 .size kern1, .Lkern1_end-kern1 8973 8974 .rodata 8975 .p2align 6 8976 .amdhsa_kernel kern1 8977 // ... 8978 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 8979 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 8980 .end_amdhsa_kernel 8981 8982These symbols cannot identify connected components in order to automatically 8983track the usage for each kernel. However, in some cases careful organization of 8984the kernels and functions in the source file means there is minimal additional 8985effort required to accurately calculate GPR usage. 8986 8987Additional Documentation 8988======================== 8989 8990.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 8991.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 8992.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 8993.. [AMD-GCN-GFX9] `AMD "Vega" Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 8994.. [AMD-GCN-GFX10] `AMD "RDNA 1.0" Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 8995.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 8996.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 8997.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 8998.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 8999.. [AMD-ROCm] `AMD ROCm Platform <https://rocm-documentation.readthedocs.io>`__ 9000.. [AMD-ROCm-github] `ROCm github <http://github.com/RadeonOpenCompute>`__ 9001.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 9002.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 9003.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 9004.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 9005.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 9006.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 9007.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 9008.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 9009.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 9010