1============================= 2User Guide for AMDGPU Backend 3============================= 4 5.. contents:: 6 :local: 7 8.. toctree:: 9 :hidden: 10 11 AMDGPU/AMDGPUAsmGFX7 12 AMDGPU/AMDGPUAsmGFX8 13 AMDGPU/AMDGPUAsmGFX9 14 AMDGPU/AMDGPUAsmGFX900 15 AMDGPU/AMDGPUAsmGFX904 16 AMDGPU/AMDGPUAsmGFX906 17 AMDGPU/AMDGPUAsmGFX908 18 AMDGPU/AMDGPUAsmGFX90a 19 AMDGPU/AMDGPUAsmGFX940 20 AMDGPU/AMDGPUAsmGFX10 21 AMDGPU/AMDGPUAsmGFX1011 22 AMDGPU/AMDGPUAsmGFX1013 23 AMDGPU/AMDGPUAsmGFX1030 24 AMDGPUModifierSyntax 25 AMDGPUOperandSyntax 26 AMDGPUInstructionSyntax 27 AMDGPUInstructionNotation 28 AMDGPUDwarfExtensionsForHeterogeneousDebugging 29 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack 30 31Introduction 32============ 33 34The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the 35R600 family up until the current GCN families. It lives in the 36``llvm/lib/Target/AMDGPU`` directory. 37 38LLVM 39==== 40 41.. _amdgpu-target-triples: 42 43Target Triples 44-------------- 45 46Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>`` 47to specify the target triple: 48 49 .. table:: AMDGPU Architectures 50 :name: amdgpu-architecture-table 51 52 ============ ============================================================== 53 Architecture Description 54 ============ ============================================================== 55 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders. 56 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders. 57 ============ ============================================================== 58 59 .. table:: AMDGPU Vendors 60 :name: amdgpu-vendor-table 61 62 ============ ============================================================== 63 Vendor Description 64 ============ ============================================================== 65 ``amd`` Can be used for all AMD GPU usage. 66 ``mesa3d`` Can be used if the OS is ``mesa3d``. 67 ============ ============================================================== 68 69 .. table:: AMDGPU Operating Systems 70 :name: amdgpu-os 71 72 ============== ============================================================ 73 OS Description 74 ============== ============================================================ 75 *<empty>* Defaults to the *unknown* OS. 76 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes 77 such as: 78 79 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa* 80 loader on Linux. See *AMD ROCm Platform Release Notes* 81 [AMD-ROCm-Release-Notes]_ for supported hardware and 82 software. 83 - AMD's PAL runtime using the *pal-amdhsa* loader on 84 Windows. 85 86 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL 87 runtime using the *pal-amdpal* loader on Windows and Linux 88 Pro. 89 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa 90 3D runtime using the *mesa-mesa3d* loader on Linux. 91 ============== ============================================================ 92 93 .. table:: AMDGPU Environments 94 :name: amdgpu-environment-table 95 96 ============ ============================================================== 97 Environment Description 98 ============ ============================================================== 99 *<empty>* Default. 100 ============ ============================================================== 101 102.. _amdgpu-processors: 103 104Processors 105---------- 106 107Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to 108specify the AMDGPU processor together with optional target features. See 109:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target 110specific information. 111 112Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions: 113 114* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`). 115 116 117 .. table:: AMDGPU Processors 118 :name: amdgpu-processor-table 119 120 =========== =============== ============ ===== ================= =============== =============== ====================== 121 Processor Alternative Target dGPU/ Target Target OS Support Example 122 Processor Triple APU Features Properties *(see* Products 123 Architecture Supported `amdgpu-os`_ 124 *and 125 corresponding 126 runtime release 127 notes for 128 current 129 information and 130 level of 131 support)* 132 =========== =============== ============ ===== ================= =============== =============== ====================== 133 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_ 134 ----------------------------------------------------------------------------------------------------------------------- 135 ``r600`` ``r600`` dGPU - Does not 136 support 137 generic 138 address 139 space 140 ``r630`` ``r600`` dGPU - Does not 141 support 142 generic 143 address 144 space 145 ``rs880`` ``r600`` dGPU - Does not 146 support 147 generic 148 address 149 space 150 ``rv670`` ``r600`` dGPU - Does not 151 support 152 generic 153 address 154 space 155 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_ 156 ----------------------------------------------------------------------------------------------------------------------- 157 ``rv710`` ``r600`` dGPU - Does not 158 support 159 generic 160 address 161 space 162 ``rv730`` ``r600`` dGPU - Does not 163 support 164 generic 165 address 166 space 167 ``rv770`` ``r600`` dGPU - Does not 168 support 169 generic 170 address 171 space 172 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_ 173 ----------------------------------------------------------------------------------------------------------------------- 174 ``cedar`` ``r600`` dGPU - Does not 175 support 176 generic 177 address 178 space 179 ``cypress`` ``r600`` dGPU - Does not 180 support 181 generic 182 address 183 space 184 ``juniper`` ``r600`` dGPU - Does not 185 support 186 generic 187 address 188 space 189 ``redwood`` ``r600`` dGPU - Does not 190 support 191 generic 192 address 193 space 194 ``sumo`` ``r600`` dGPU - Does not 195 support 196 generic 197 address 198 space 199 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_ 200 ----------------------------------------------------------------------------------------------------------------------- 201 ``barts`` ``r600`` dGPU - Does not 202 support 203 generic 204 address 205 space 206 ``caicos`` ``r600`` dGPU - Does not 207 support 208 generic 209 address 210 space 211 ``cayman`` ``r600`` dGPU - Does not 212 support 213 generic 214 address 215 space 216 ``turks`` ``r600`` dGPU - Does not 217 support 218 generic 219 address 220 space 221 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_ 222 ----------------------------------------------------------------------------------------------------------------------- 223 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 224 support 225 generic 226 address 227 space 228 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 229 - ``verde`` support 230 generic 231 address 232 space 233 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal* 234 - ``oland`` support 235 generic 236 address 237 space 238 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_ 239 ----------------------------------------------------------------------------------------------------------------------- 240 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000 241 flat - *pal-amdhsa* - A6 Pro-7050B 242 scratch - *pal-amdpal* - A8-7100 243 - A8 Pro-7150B 244 - A10-7300 245 - A10 Pro-7350B 246 - FX-7500 247 - A8-7200P 248 - A10-7400P 249 - FX-7600P 250 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100 251 flat - *pal-amdhsa* - FirePro W9100 252 scratch - *pal-amdpal* - FirePro S9150 253 - FirePro S9170 254 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290 255 flat - *pal-amdhsa* - Radeon R9 290x 256 scratch - *pal-amdpal* - Radeon R390 257 - Radeon R390x 258 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100 259 - ``mullins`` flat - *pal-amdpal* - E1-2200 260 scratch - E1-2500 261 - E2-3000 262 - E2-3800 263 - A4-5000 264 - A4-5100 265 - A6-5200 266 - A4 Pro-3340B 267 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790 268 flat - *pal-amdpal* - Radeon HD 8770 269 scratch - R7 260 270 - R7 260X 271 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA* 272 flat - *pal-amdpal* 273 scratch .. TODO:: 274 275 Add product 276 names. 277 278 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_ 279 ----------------------------------------------------------------------------------------------------------------------- 280 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P 281 flat - *pal-amdhsa* - Pro A6-8500B 282 scratch - *pal-amdpal* - A8-8600P 283 - Pro A8-8600B 284 - FX-8800P 285 - Pro A12-8800B 286 - A10-8700P 287 - Pro A10-8700B 288 - A10-8780P 289 - A10-9600P 290 - A10-9630P 291 - A12-9700P 292 - A12-9730P 293 - FX-9800P 294 - FX-9830P 295 - E2-9010 296 - A6-9210 297 - A9-9410 298 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285 299 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380 300 scratch - *pal-amdpal* - Radeon R9 385 301 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano 302 - *pal-amdhsa* - Radeon R9 Fury 303 - *pal-amdpal* - Radeon R9 FuryX 304 - Radeon Pro Duo 305 - FirePro S9300x2 306 - Radeon Instinct MI8 307 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470 308 flat - *pal-amdhsa* - Radeon RX 480 309 scratch - *pal-amdpal* - Radeon Instinct MI6 310 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460 311 flat - *pal-amdhsa* 312 scratch - *pal-amdpal* 313 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150 314 flat - *pal-amdhsa* - FirePro S7100 315 scratch - *pal-amdpal* - FirePro W7100 316 - Mobile FirePro 317 M7170 318 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA* 319 flat - *pal-amdhsa* 320 scratch - *pal-amdpal* .. TODO:: 321 322 Add product 323 names. 324 325 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ 326 ----------------------------------------------------------------------------------------------------------------------- 327 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega 328 flat - *pal-amdhsa* Frontier Edition 329 scratch - *pal-amdpal* - Radeon RX Vega 56 330 - Radeon RX Vega 64 331 - Radeon RX Vega 64 332 Liquid 333 - Radeon Instinct MI25 334 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G 335 flat - *pal-amdhsa* - Ryzen 5 2400G 336 scratch - *pal-amdpal* 337 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA* 338 - *pal-amdhsa* 339 - *pal-amdpal* .. TODO:: 340 341 Add product 342 names. 343 344 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50 345 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60 346 scratch - *pal-amdpal* - Radeon VII 347 - Radeon Pro VII 348 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator 349 - xnack - Absolute 350 flat 351 scratch 352 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA* 353 flat 354 scratch .. TODO:: 355 356 Add product 357 names. 358 359 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* *TBA* 360 - tgsplit flat 361 - xnack scratch .. TODO:: 362 - Packed 363 work-item Add product 364 IDs names. 365 366 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G 367 flat - Ryzen 7 4700GE 368 scratch - Ryzen 5 4600G 369 - Ryzen 5 4600GE 370 - Ryzen 3 4300G 371 - Ryzen 3 4300GE 372 - Ryzen Pro 4000G 373 - Ryzen 7 Pro 4700G 374 - Ryzen 7 Pro 4750GE 375 - Ryzen 5 Pro 4650G 376 - Ryzen 5 Pro 4650GE 377 - Ryzen 3 Pro 4350G 378 - Ryzen 3 Pro 4350GE 379 380 ``gfx940`` ``amdgcn`` dGPU - sramecc - Architected *TBA* 381 - tgsplit flat 382 - xnack scratch .. TODO:: 383 - Packed 384 work-item Add product 385 IDs names. 386 387 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_ 388 ----------------------------------------------------------------------------------------------------------------------- 389 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5700 390 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5700 XT 391 - xnack scratch - *pal-amdpal* - Radeon Pro 5600 XT 392 - Radeon Pro 5600M 393 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520 394 - wavefrontsize64 - Absolute - *pal-amdhsa* 395 - xnack flat - *pal-amdpal* 396 scratch 397 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500 398 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT 399 - xnack scratch - *pal-amdpal* 400 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA* 401 - wavefrontsize64 flat - *pal-amdhsa* 402 - xnack scratch - *pal-amdpal* .. TODO:: 403 404 Add product 405 names. 406 407 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_ 408 ----------------------------------------------------------------------------------------------------------------------- 409 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800 410 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT 411 scratch - *pal-amdpal* - Radeon RX 6900 XT 412 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT 413 - wavefrontsize64 flat - *pal-amdhsa* 414 scratch - *pal-amdpal* 415 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA* 416 - wavefrontsize64 flat - *pal-amdhsa* 417 scratch - *pal-amdpal* .. TODO:: 418 419 Add product 420 names. 421 422 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 423 - wavefrontsize64 flat 424 scratch .. TODO:: 425 426 Add product 427 names. 428 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA* 429 - wavefrontsize64 flat 430 scratch .. TODO:: 431 432 Add product 433 names. 434 435 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 436 - wavefrontsize64 flat 437 scratch .. TODO:: 438 Add product 439 names. 440 441 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA* 442 - wavefrontsize64 flat 443 scratch .. TODO:: 444 445 Add product 446 names. 447 448 **GCN GFX11** 449 ----------------------------------------------------------------------------------------------------------------------- 450 ``gfx1100`` ``amdgcn`` dGPU - cumode - Architected - *pal-amdpal* *TBA* 451 - wavefrontsize64 flat 452 scratch .. TODO:: 453 - Packed 454 work-item Add product 455 IDs names. 456 457 ``gfx1101`` ``amdgcn`` dGPU - cumode - Architected *TBA* 458 - wavefrontsize64 flat 459 scratch .. TODO:: 460 - Packed 461 work-item Add product 462 IDs names. 463 464 ``gfx1102`` ``amdgcn`` dGPU - cumode - Architected *TBA* 465 - wavefrontsize64 flat 466 scratch .. TODO:: 467 - Packed 468 work-item Add product 469 IDs names. 470 471 ``gfx1103`` ``amdgcn`` APU - cumode - Architected *TBA* 472 - wavefrontsize64 flat 473 scratch .. TODO:: 474 - Packed 475 work-item Add product 476 IDs names. 477 478 =========== =============== ============ ===== ================= =============== =============== ====================== 479 480.. _amdgpu-target-features: 481 482Target Features 483--------------- 484 485Target features control how code is generated to support certain 486processor specific features. Not all target features are supported by 487all processors. The runtime must ensure that the features supported by 488the device used to execute the code match the features enabled when 489generating the code. A mismatch of features may result in incorrect 490execution, or a reduction in performance. 491 492The target features supported by each processor is listed in 493:ref:`amdgpu-processor-table`. 494 495Target features are controlled by exactly one of the following Clang 496options: 497 498``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` 499 500 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as 501 optional components of the target ID. If omitted, the target feature has the 502 ``any`` value. See :ref:`amdgpu-target-id`. 503 504``-m[no-]<target-feature>`` 505 506 Target features not specified by the target ID are specified using a 507 separate option. These target features can have an ``on`` or ``off`` 508 value. ``on`` is specified by omitting the ``no-`` prefix, and 509 ``off`` is specified by including the ``no-`` prefix. The default 510 if not specified is ``off``. 511 512For example: 513 514``-mcpu=gfx908:xnack+`` 515 Enable the ``xnack`` feature. 516``-mcpu=gfx908:xnack-`` 517 Disable the ``xnack`` feature. 518``-mcumode`` 519 Enable the ``cumode`` feature. 520``-mno-cumode`` 521 Disable the ``cumode`` feature. 522 523 .. table:: AMDGPU Target Features 524 :name: amdgpu-target-features-table 525 526 =============== ============================ ================================================== 527 Target Feature Clang Option to Control Description 528 Name 529 =============== ============================ ================================================== 530 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used 531 when generating code for kernels. When disabled 532 native WGP wavefront execution mode is used, 533 when enabled CU wavefront execution mode is used 534 (see :ref:`amdgpu-amdhsa-memory-model`). 535 536 sramecc - ``-mcpu`` If specified, generate code that can only be 537 - ``--offload-arch`` loaded and executed in a process that has a 538 matching setting for SRAMECC. 539 540 If not specified for code object V2 to V3, generate 541 code that can be loaded and executed in a process 542 with SRAMECC enabled. 543 544 If not specified for code object V4 or above, generate 545 code that can be loaded and executed in a process 546 with either setting of SRAMECC. 547 548 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes 549 work-groups are launched in threadgroup split mode. 550 When enabled the waves of a work-group may be 551 launched in different CUs. 552 553 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when 554 generating code for kernels. When disabled 555 native wavefront size 32 is used, when enabled 556 wavefront size 64 is used. 557 558 xnack - ``-mcpu`` If specified, generate code that can only be 559 - ``--offload-arch`` loaded and executed in a process that has a 560 matching setting for XNACK replay. 561 562 If not specified for code object V2 to V3, generate 563 code that can be loaded and executed in a process 564 with XNACK replay enabled. 565 566 If not specified for code object V4 or above, generate 567 code that can be loaded and executed in a process 568 with either setting of XNACK replay. 569 570 XNACK replay can be used for demand paging and 571 page migration. If enabled in the device, then if 572 a page fault occurs the code may execute 573 incorrectly unless generated with XNACK replay 574 enabled, or generated for code object V4 or above without 575 specifying XNACK replay. Executing code that was 576 generated with XNACK replay enabled, or generated 577 for code object V4 or above without specifying XNACK replay, 578 on a device that does not have XNACK replay 579 enabled will execute correctly but may be less 580 performant than code generated for XNACK replay 581 disabled. 582 =============== ============================ ================================================== 583 584.. _amdgpu-target-id: 585 586Target ID 587--------- 588 589AMDGPU supports target IDs. See `Clang Offload Bundler 590<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general 591description. The AMDGPU target specific information is: 592 593**processor** 594 Is an AMDGPU processor or alternative processor name specified in 595 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both 596 the primary processor and alternative processor names. The canonical form 597 target ID only allow the primary processor name. 598 599**target-feature** 600 Is a target feature name specified in :ref:`amdgpu-target-features-table` that 601 is supported by the processor. The target features supported by each processor 602 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in 603 a target ID are marked as being controlled by ``-mcpu`` and 604 ``--offload-arch``. Each target feature must appear at most once in a target 605 ID. The non-canonical form target ID allows the target features to be 606 specified in any order. The canonical form target ID requires the target 607 features to be specified in alphabetic order. 608 609.. _amdgpu-target-id-v2-v3: 610 611Code Object V2 to V3 Target ID 612~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 613 614The target ID syntax for code object V2 to V3 is the same as defined in `Clang 615Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except 616when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler 617directive and the bundle entry ID. In those cases it has the following BNF 618syntax: 619 620.. code:: 621 622 <target-id> ::== <processor> ( "+" <target-feature> )* 623 624Where a target feature is omitted if *Off* and present if *On* or *Any*. 625 626.. note:: 627 628 The code object V2 to V3 cannot represent *Any* and treats it the same as 629 *On*. 630 631.. _amdgpu-embedding-bundled-objects: 632 633Embedding Bundled Code Objects 634------------------------------ 635 636AMDGPU supports the HIP and OpenMP languages that perform code object embedding 637as described in `Clang Offload Bundler 638<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_. 639 640.. note:: 641 642 The target ID syntax used for code object V2 to V3 for a bundle entry ID 643 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 644 645.. _amdgpu-address-spaces: 646 647Address Spaces 648-------------- 649 650The AMDGPU architecture supports a number of memory address spaces. The address 651space names use the OpenCL standard names, with some additions. 652 653The AMDGPU address spaces correspond to target architecture specific LLVM 654address space numbers used in LLVM IR. 655 656The AMDGPU address spaces are described in 657:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are 658supported for the ``amdgcn`` target. 659 660 .. table:: AMDGPU Address Spaces 661 :name: amdgpu-address-spaces-table 662 663 ================================= =============== =========== ================ ======= ============================ 664 .. 64-Bit Process Address Space 665 --------------------------------- --------------- ----------- ---------------- ------------------------------------ 666 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value 667 Space Number Name Name Size 668 ================================= =============== =========== ================ ======= ============================ 669 Generic 0 flat flat 64 0x0000000000000000 670 Global 1 global global 64 0x0000000000000000 671 Region 2 N/A GDS 32 *not implemented for AMDHSA* 672 Local 3 group LDS 32 0xFFFFFFFF 673 Constant 4 constant *same as global* 64 0x0000000000000000 674 Private 5 private scratch 32 0xFFFFFFFF 675 Constant 32-bit 6 *TODO* 0x00000000 676 Buffer Fat Pointer (experimental) 7 *TODO* 677 ================================= =============== =========== ================ ======= ============================ 678 679**Generic** 680 The generic address space is supported unless the *Target Properties* column 681 of :ref:`amdgpu-processor-table` specifies *Does not support generic address 682 space*. 683 684 The generic address space uses the hardware flat address support for two fixed 685 ranges of virtual addresses (the private and local apertures), that are 686 outside the range of addressable global memory, to map from a flat address to 687 a private or local address. This uses FLAT instructions that can take a flat 688 address and access global, private (scratch), and group (LDS) memory depending 689 on if the address is within one of the aperture ranges. 690 691 Flat access to scratch requires hardware aperture setup and setup in the 692 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat 693 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register 694 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`). 695 696 To convert between a private or group address space address (termed a segment 697 address) and a flat address the base address of the corresponding aperture 698 can be used. For GFX7-GFX8 these are available in the 699 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 700 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 701 GFX9-GFX10 the aperture base addresses are directly available as inline 702 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. 703 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is 704 aligned to 2^32 which makes it easier to convert from flat to segment or 705 segment to flat. 706 707 A global address space address has the same value when used as a flat address 708 so no conversion is needed. 709 710**Global and Constant** 711 The global and constant address spaces both use global virtual addresses, 712 which are the same virtual address space used by the CPU. However, some 713 virtual addresses may only be accessible to the CPU, some only accessible 714 by the GPU, and some by both. 715 716 Using the constant address space indicates that the data will not change 717 during the execution of the kernel. This allows scalar read instructions to 718 be used. As the constant address space could only be modified on the host 719 side, a generic pointer loaded from the constant address space is safe to be 720 assumed as a global pointer since only the device global memory is visible 721 and managed on the host side. The vector and scalar L1 caches are invalidated 722 of volatile data before each kernel dispatch execution to allow constant 723 memory to change values between kernel dispatches. 724 725**Region** 726 The region address space uses the hardware Global Data Store (GDS). All 727 wavefronts executing on the same device will access the same memory for any 728 given region address. However, the same region address accessed by wavefronts 729 executing on different devices will access different memory. It is higher 730 performance than global memory. It is allocated by the runtime. The data 731 store (DS) instructions can be used to access it. 732 733**Local** 734 The local address space uses the hardware Local Data Store (LDS) which is 735 automatically allocated when the hardware creates the wavefronts of a 736 work-group, and freed when all the wavefronts of a work-group have 737 terminated. All wavefronts belonging to the same work-group will access the 738 same memory for any given local address. However, the same local address 739 accessed by wavefronts belonging to different work-groups will access 740 different memory. It is higher performance than global memory. The data store 741 (DS) instructions can be used to access it. 742 743**Private** 744 The private address space uses the hardware scratch memory support which 745 automatically allocates memory when it creates a wavefront and frees it when 746 a wavefronts terminates. The memory accessed by a lane of a wavefront for any 747 given private address will be different to the memory accessed by another lane 748 of the same or different wavefront for the same private address. 749 750 If a kernel dispatch uses scratch, then the hardware allocates memory from a 751 pool of backing memory allocated by the runtime for each wavefront. The lanes 752 of the wavefront access this using dword (4 byte) interleaving. The mapping 753 used from private address to backing memory address is: 754 755 ``wavefront-scratch-base + 756 ((private-address / 4) * wavefront-size * 4) + 757 (wavefront-lane-id * 4) + (private-address % 4)`` 758 759 If each lane of a wavefront accesses the same private address, the 760 interleaving results in adjacent dwords being accessed and hence requires 761 fewer cache lines to be fetched. 762 763 There are different ways that the wavefront scratch base address is 764 determined by a wavefront (see 765 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 766 767 Scratch memory can be accessed in an interleaved manner using buffer 768 instructions with the scratch buffer descriptor and per wavefront scratch 769 offset, by the scratch instructions, or by flat instructions. Multi-dword 770 access is not supported except by flat and scratch instructions in 771 GFX9-GFX10. 772 773**Constant 32-bit** 774 *TODO* 775 776**Buffer Fat Pointer** 777 The buffer fat pointer is an experimental address space that is currently 778 unsupported in the backend. It exposes a non-integral pointer that is in 779 the future intended to support the modelling of 128-bit buffer descriptors 780 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit 781 *pointer*), allowing normal LLVM load/store/atomic operations to be used to 782 model the buffer descriptors used heavily in graphics workloads targeting 783 the backend. 784 785.. _amdgpu-memory-scopes: 786 787Memory Scopes 788------------- 789 790This section provides LLVM memory synchronization scopes supported by the AMDGPU 791backend memory model when the target triple OS is ``amdhsa`` (see 792:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`). 793 794The memory model supported is based on the HSA memory model [HSA]_ which is 795based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before 796relation is transitive over the synchronizes-with relation independent of scope 797and synchronizes-with allows the memory scope instances to be inclusive (see 798table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`). 799 800This is different to the OpenCL [OpenCL]_ memory model which does not have scope 801inclusion and requires the memory scopes to exactly match. However, this 802is conservatively correct for OpenCL. 803 804 .. table:: AMDHSA LLVM Sync Scopes 805 :name: amdgpu-amdhsa-llvm-sync-scopes-table 806 807 ======================= =================================================== 808 LLVM Sync Scope Description 809 ======================= =================================================== 810 *none* The default: ``system``. 811 812 Synchronizes with, and participates in modification 813 and seq_cst total orderings with, other operations 814 (except image operations) for all address spaces 815 (except private, or generic that accesses private) 816 provided the other operation's sync scope is: 817 818 - ``system``. 819 - ``agent`` and executed by a thread on the same 820 agent. 821 - ``workgroup`` and executed by a thread in the 822 same work-group. 823 - ``wavefront`` and executed by a thread in the 824 same wavefront. 825 826 ``agent`` Synchronizes with, and participates in modification 827 and seq_cst total orderings with, other operations 828 (except image operations) for all address spaces 829 (except private, or generic that accesses private) 830 provided the other operation's sync scope is: 831 832 - ``system`` or ``agent`` and executed by a thread 833 on the same agent. 834 - ``workgroup`` and executed by a thread in the 835 same work-group. 836 - ``wavefront`` and executed by a thread in the 837 same wavefront. 838 839 ``workgroup`` Synchronizes with, and participates in modification 840 and seq_cst total orderings with, other operations 841 (except image operations) for all address spaces 842 (except private, or generic that accesses private) 843 provided the other operation's sync scope is: 844 845 - ``system``, ``agent`` or ``workgroup`` and 846 executed by a thread in the same work-group. 847 - ``wavefront`` and executed by a thread in the 848 same wavefront. 849 850 ``wavefront`` Synchronizes with, and participates in modification 851 and seq_cst total orderings with, other operations 852 (except image operations) for all address spaces 853 (except private, or generic that accesses private) 854 provided the other operation's sync scope is: 855 856 - ``system``, ``agent``, ``workgroup`` or 857 ``wavefront`` and executed by a thread in the 858 same wavefront. 859 860 ``singlethread`` Only synchronizes with and participates in 861 modification and seq_cst total orderings with, 862 other operations (except image operations) running 863 in the same thread for all address spaces (for 864 example, in signal handlers). 865 866 ``one-as`` Same as ``system`` but only synchronizes with other 867 operations within the same address space. 868 869 ``agent-one-as`` Same as ``agent`` but only synchronizes with other 870 operations within the same address space. 871 872 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with 873 other operations within the same address space. 874 875 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with 876 other operations within the same address space. 877 878 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with 879 other operations within the same address space. 880 ======================= =================================================== 881 882LLVM IR Intrinsics 883------------------ 884 885The AMDGPU backend implements the following LLVM IR intrinsics. 886 887*This section is WIP.* 888 889.. TODO:: 890 891 List AMDGPU intrinsics. 892 893LLVM IR Attributes 894------------------ 895 896The AMDGPU backend supports the following LLVM IR attributes. 897 898 .. table:: AMDGPU LLVM IR Attributes 899 :name: amdgpu-llvm-ir-attributes-table 900 901 ======================================= ========================================================== 902 LLVM Attribute Description 903 ======================================= ========================================================== 904 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that 905 will be specified when the kernel is dispatched. Generated 906 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_. 907 The implied default value is 1,1024. 908 909 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel 910 argument block size for the implicit arguments. This 911 varies by OS and language (for OpenCL see 912 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 913 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by 914 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_. 915 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the 916 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_. 917 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per 918 execution unit. Generated by the ``amdgpu_waves_per_eu`` 919 CLANG attribute [CLANG-ATTR]_. This is an optimization hint, 920 and the backend may not be able to satisfy the request. If 921 the specified range is incompatible with the function's 922 "amdgpu-flat-work-group-size" value, the implied occupancy 923 bounds by the workgroup size takes precedence. 924 925 "amdgpu-ieee" true/false. Specify whether the function expects the IEEE field of the 926 mode register to be set on entry. Overrides the default for 927 the calling convention. 928 "amdgpu-dx10-clamp" true/false. Specify whether the function expects the DX10_CLAMP field of 929 the mode register to be set on entry. Overrides the default 930 for the calling convention. 931 932 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the 933 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this 934 attribute, or reached through a call site marked with this attribute, 935 the value returned by the intrinsic is undefined. The backend can 936 generally infer this during code generation, so typically there is no 937 benefit to frontends marking functions with this. 938 939 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the 940 llvm.amdgcn.workitem.id.y intrinsic. 941 942 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the 943 llvm.amdgcn.workitem.id.z intrinsic. 944 945 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the 946 llvm.amdgcn.workgroup.id.x intrinsic. 947 948 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the 949 llvm.amdgcn.workgroup.id.y intrinsic. 950 951 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the 952 llvm.amdgcn.workgroup.id.z intrinsic. 953 954 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the 955 llvm.amdgcn.dispatch.ptr intrinsic. 956 957 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the 958 llvm.amdgcn.implicitarg.ptr intrinsic. 959 960 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the 961 llvm.amdgcn.dispatch.id intrinsic. 962 963 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the 964 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint 965 attributes, the queue pointer may be required in situations where the 966 intrinsic call does not directly appear in the program. Some subtargets 967 require the queue pointer for to handle some addrspacecasts, as well 968 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and 969 llvm.debug intrinsics. 970 971 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 972 kernel argument that holds the pointer to the hostcall buffer. If this 973 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 974 975 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 976 kernel argument that holds the pointer to an initialized memory buffer 977 that conforms to the requirements of the malloc/free device library V1 978 version implementation. If this attribute is absent, then the 979 amdgpu-no-implicitarg-ptr is also removed. 980 981 "amdgpu-no-multigrid-sync-arg" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit 982 kernel argument that holds the multigrid synchronization pointer. If this 983 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed. 984 ======================================= ========================================================== 985 986.. _amdgpu-elf-code-object: 987 988ELF Code Object 989=============== 990 991The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that 992can be linked by ``lld`` to produce a standard ELF shared code object which can 993be loaded and executed on an AMDGPU target. 994 995.. _amdgpu-elf-header: 996 997Header 998------ 999 1000The AMDGPU backend uses the following ELF header: 1001 1002 .. table:: AMDGPU ELF Header 1003 :name: amdgpu-elf-header-table 1004 1005 ========================== =============================== 1006 Field Value 1007 ========================== =============================== 1008 ``e_ident[EI_CLASS]`` ``ELFCLASS64`` 1009 ``e_ident[EI_DATA]`` ``ELFDATA2LSB`` 1010 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE`` 1011 - ``ELFOSABI_AMDGPU_HSA`` 1012 - ``ELFOSABI_AMDGPU_PAL`` 1013 - ``ELFOSABI_AMDGPU_MESA3D`` 1014 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2`` 1015 - ``ELFABIVERSION_AMDGPU_HSA_V3`` 1016 - ``ELFABIVERSION_AMDGPU_HSA_V4`` 1017 - ``ELFABIVERSION_AMDGPU_HSA_V5`` 1018 - ``ELFABIVERSION_AMDGPU_PAL`` 1019 - ``ELFABIVERSION_AMDGPU_MESA3D`` 1020 ``e_type`` - ``ET_REL`` 1021 - ``ET_DYN`` 1022 ``e_machine`` ``EM_AMDGPU`` 1023 ``e_entry`` 0 1024 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`, 1025 :ref:`amdgpu-elf-header-e_flags-table-v3`, 1026 and :ref:`amdgpu-elf-header-e_flags-table-v4-onwards` 1027 ========================== =============================== 1028 1029.. 1030 1031 .. table:: AMDGPU ELF Header Enumeration Values 1032 :name: amdgpu-elf-header-enumeration-values-table 1033 1034 =============================== ===== 1035 Name Value 1036 =============================== ===== 1037 ``EM_AMDGPU`` 224 1038 ``ELFOSABI_NONE`` 0 1039 ``ELFOSABI_AMDGPU_HSA`` 64 1040 ``ELFOSABI_AMDGPU_PAL`` 65 1041 ``ELFOSABI_AMDGPU_MESA3D`` 66 1042 ``ELFABIVERSION_AMDGPU_HSA_V2`` 0 1043 ``ELFABIVERSION_AMDGPU_HSA_V3`` 1 1044 ``ELFABIVERSION_AMDGPU_HSA_V4`` 2 1045 ``ELFABIVERSION_AMDGPU_HSA_V5`` 3 1046 ``ELFABIVERSION_AMDGPU_PAL`` 0 1047 ``ELFABIVERSION_AMDGPU_MESA3D`` 0 1048 =============================== ===== 1049 1050``e_ident[EI_CLASS]`` 1051 The ELF class is: 1052 1053 * ``ELFCLASS32`` for ``r600`` architecture. 1054 1055 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit 1056 process address space applications. 1057 1058``e_ident[EI_DATA]`` 1059 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering. 1060 1061``e_ident[EI_OSABI]`` 1062 One of the following AMDGPU target architecture specific OS ABIs 1063 (see :ref:`amdgpu-os`): 1064 1065 * ``ELFOSABI_NONE`` for *unknown* OS. 1066 1067 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS. 1068 1069 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS. 1070 1071 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS. 1072 1073``e_ident[EI_ABIVERSION]`` 1074 The ABI version of the AMDGPU target architecture specific OS ABI to which the code 1075 object conforms: 1076 1077 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA 1078 runtime ABI for code object V2. Specify using the Clang option 1079 ``-mcode-object-version=2``. 1080 1081 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA 1082 runtime ABI for code object V3. Specify using the Clang option 1083 ``-mcode-object-version=3``. 1084 1085 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA 1086 runtime ABI for code object V4. Specify using the Clang option 1087 ``-mcode-object-version=4``. This is the default code object 1088 version if not specified. 1089 1090 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA 1091 runtime ABI for code object V5. Specify using the Clang option 1092 ``-mcode-object-version=5``. 1093 1094 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL 1095 runtime ABI. 1096 1097 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA 1098 3D runtime ABI. 1099 1100``e_type`` 1101 Can be one of the following values: 1102 1103 1104 ``ET_REL`` 1105 The type produced by the AMDGPU backend compiler as it is relocatable code 1106 object. 1107 1108 ``ET_DYN`` 1109 The type produced by the linker as it is a shared code object. 1110 1111 The AMD HSA runtime loader requires a ``ET_DYN`` code object. 1112 1113``e_machine`` 1114 The value ``EM_AMDGPU`` is used for the machine for all processors supported 1115 by the ``r600`` and ``amdgcn`` architectures (see 1116 :ref:`amdgpu-processor-table`). The specific processor is specified in the 1117 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see 1118 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the 1119 ``e_flags`` for code object V3 and above (see 1120 :ref:`amdgpu-elf-header-e_flags-table-v3` and 1121 :ref:`amdgpu-elf-header-e_flags-table-v4-onwards`). 1122 1123``e_entry`` 1124 The entry point is 0 as the entry points for individual kernels must be 1125 selected in order to invoke them through AQL packets. 1126 1127``e_flags`` 1128 The AMDGPU backend uses the following ELF header flags: 1129 1130 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V2 1131 :name: amdgpu-elf-header-e_flags-v2-table 1132 1133 ===================================== ===== ============================= 1134 Name Value Description 1135 ===================================== ===== ============================= 1136 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack`` 1137 target feature is 1138 enabled for all code 1139 contained in the code object. 1140 If the processor 1141 does not support the 1142 ``xnack`` target 1143 feature then must 1144 be 0. 1145 See 1146 :ref:`amdgpu-target-features`. 1147 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap 1148 handler is enabled for all 1149 code contained in the code 1150 object. If the processor 1151 does not support a trap 1152 handler then must be 0. 1153 See 1154 :ref:`amdgpu-target-features`. 1155 ===================================== ===== ============================= 1156 1157 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V3 1158 :name: amdgpu-elf-header-e_flags-table-v3 1159 1160 ================================= ===== ============================= 1161 Name Value Description 1162 ================================= ===== ============================= 1163 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1164 mask for 1165 ``EF_AMDGPU_MACH_xxx`` values 1166 defined in 1167 :ref:`amdgpu-ef-amdgpu-mach-table`. 1168 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack`` 1169 target feature is 1170 enabled for all code 1171 contained in the code object. 1172 If the processor 1173 does not support the 1174 ``xnack`` target 1175 feature then must 1176 be 0. 1177 See 1178 :ref:`amdgpu-target-features`. 1179 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc`` 1180 target feature is 1181 enabled for all code 1182 contained in the code object. 1183 If the processor 1184 does not support the 1185 ``sramecc`` target 1186 feature then must 1187 be 0. 1188 See 1189 :ref:`amdgpu-target-features`. 1190 ================================= ===== ============================= 1191 1192 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and After 1193 :name: amdgpu-elf-header-e_flags-table-v4-onwards 1194 1195 ============================================ ===== =================================== 1196 Name Value Description 1197 ============================================ ===== =================================== 1198 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection 1199 mask for 1200 ``EF_AMDGPU_MACH_xxx`` values 1201 defined in 1202 :ref:`amdgpu-ef-amdgpu-mach-table`. 1203 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for 1204 ``EF_AMDGPU_FEATURE_XNACK_*_V4`` 1205 values. 1206 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsuppored. 1207 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value. 1208 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled. 1209 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled. 1210 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for 1211 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4`` 1212 values. 1213 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsuppored. 1214 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value. 1215 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled, 1216 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled. 1217 ============================================ ===== =================================== 1218 1219 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values 1220 :name: amdgpu-ef-amdgpu-mach-table 1221 1222 ==================================== ========== ============================= 1223 Name Value Description (see 1224 :ref:`amdgpu-processor-table`) 1225 ==================================== ========== ============================= 1226 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified* 1227 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600`` 1228 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630`` 1229 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880`` 1230 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670`` 1231 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710`` 1232 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730`` 1233 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770`` 1234 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar`` 1235 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress`` 1236 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper`` 1237 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood`` 1238 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo`` 1239 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts`` 1240 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos`` 1241 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman`` 1242 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks`` 1243 *reserved* 0x011 - Reserved for ``r600`` 1244 0x01f architecture processors. 1245 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600`` 1246 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601`` 1247 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700`` 1248 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701`` 1249 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702`` 1250 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703`` 1251 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704`` 1252 *reserved* 0x027 Reserved. 1253 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801`` 1254 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802`` 1255 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803`` 1256 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810`` 1257 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900`` 1258 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902`` 1259 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904`` 1260 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906`` 1261 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908`` 1262 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909`` 1263 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c`` 1264 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010`` 1265 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011`` 1266 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012`` 1267 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030`` 1268 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031`` 1269 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032`` 1270 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033`` 1271 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602`` 1272 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705`` 1273 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805`` 1274 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035`` 1275 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034`` 1276 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a`` 1277 ``EF_AMDGPU_MACH_AMDGCN_GFX940`` 0x040 ``gfx940`` 1278 ``EF_AMDGPU_MACH_AMDGCN_GFX1100`` 0x041 ``gfx1100`` 1279 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013`` 1280 *reserved* 0x043 Reserved. 1281 ``EF_AMDGPU_MACH_AMDGCN_GFX1103`` 0x044 ``gfx1103`` 1282 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036`` 1283 ``EF_AMDGPU_MACH_AMDGCN_GFX1101`` 0x046 ``gfx1101`` 1284 ``EF_AMDGPU_MACH_AMDGCN_GFX1102`` 0x047 ``gfx1102`` 1285 ==================================== ========== ============================= 1286 1287Sections 1288-------- 1289 1290An AMDGPU target ELF code object has the standard ELF sections which include: 1291 1292 .. table:: AMDGPU ELF Sections 1293 :name: amdgpu-elf-sections-table 1294 1295 ================== ================ ================================= 1296 Name Type Attributes 1297 ================== ================ ================================= 1298 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1299 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1300 ``.debug_``\ *\** ``SHT_PROGBITS`` *none* 1301 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC`` 1302 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1303 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1304 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE`` 1305 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC`` 1306 ``.note`` ``SHT_NOTE`` *none* 1307 ``.rela``\ *name* ``SHT_RELA`` *none* 1308 ``.rela.dyn`` ``SHT_RELA`` *none* 1309 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC`` 1310 ``.shstrtab`` ``SHT_STRTAB`` *none* 1311 ``.strtab`` ``SHT_STRTAB`` *none* 1312 ``.symtab`` ``SHT_SYMTAB`` *none* 1313 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR`` 1314 ================== ================ ================================= 1315 1316These sections have their standard meanings (see [ELF]_) and are only generated 1317if needed. 1318 1319``.debug``\ *\** 1320 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for 1321 information on the DWARF produced by the AMDGPU backend. 1322 1323``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash`` 1324 The standard sections used by a dynamic loader. 1325 1326``.note`` 1327 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU 1328 backend. 1329 1330``.rela``\ *name*, ``.rela.dyn`` 1331 For relocatable code objects, *name* is the name of the section that the 1332 relocation records apply. For example, ``.rela.text`` is the section name for 1333 relocation records associated with the ``.text`` section. 1334 1335 For linked shared code objects, ``.rela.dyn`` contains all the relocation 1336 records from each of the relocatable code object's ``.rela``\ *name* sections. 1337 1338 See :ref:`amdgpu-relocation-records` for the relocation records supported by 1339 the AMDGPU backend. 1340 1341``.text`` 1342 The executable machine code for the kernels and functions they call. Generated 1343 as position independent code. See :ref:`amdgpu-code-conventions` for 1344 information on conventions used in the isa generation. 1345 1346.. _amdgpu-note-records: 1347 1348Note Records 1349------------ 1350 1351The AMDGPU backend code object contains ELF note records in the ``.note`` 1352section. The set of generated notes and their semantics depend on the code 1353object version; see :ref:`amdgpu-note-records-v2` and 1354:ref:`amdgpu-note-records-v3-onwards`. 1355 1356As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding 1357must be generated after the ``name`` field to ensure the ``desc`` field is 4 1358byte aligned. In addition, minimal zero-byte padding must be generated to 1359ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign`` 1360field of the ``.note`` section must be at least 4 to indicate at least 8 byte 1361alignment. 1362 1363.. _amdgpu-note-records-v2: 1364 1365Code Object V2 Note Records 1366~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1367 1368.. warning:: 1369 Code object V2 is not the default code object version emitted by 1370 this version of LLVM. 1371 1372The AMDGPU backend code object uses the following ELF note record in the 1373``.note`` section when compiling for code object V2. 1374 1375The note record vendor field is "AMD". 1376 1377Additional note records may be present, but any which are not documented here 1378are deprecated and should not be used. 1379 1380 .. table:: AMDGPU Code Object V2 ELF Note Records 1381 :name: amdgpu-elf-note-records-v2-table 1382 1383 ===== ===================================== ====================================== 1384 Name Type Description 1385 ===== ===================================== ====================================== 1386 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version. 1387 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL 1388 Finalizer and not the LLVM compiler. 1389 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version. 1390 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in 1391 YAML [YAML]_ textual format. 1392 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name. 1393 ===== ===================================== ====================================== 1394 1395.. 1396 1397 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values 1398 :name: amdgpu-elf-note-record-enumeration-values-v2-table 1399 1400 ===================================== ===== 1401 Name Value 1402 ===================================== ===== 1403 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1 1404 ``NT_AMD_HSA_HSAIL`` 2 1405 ``NT_AMD_HSA_ISA_VERSION`` 3 1406 *reserved* 4-9 1407 ``NT_AMD_HSA_METADATA`` 10 1408 ``NT_AMD_HSA_ISA_NAME`` 11 1409 ===================================== ===== 1410 1411``NT_AMD_HSA_CODE_OBJECT_VERSION`` 1412 Specifies the code object version number. The description field has the 1413 following layout: 1414 1415 .. code:: c 1416 1417 struct amdgpu_hsa_note_code_object_version_s { 1418 uint32_t major_version; 1419 uint32_t minor_version; 1420 }; 1421 1422 The ``major_version`` has a value less than or equal to 2. 1423 1424``NT_AMD_HSA_HSAIL`` 1425 Specifies the HSAIL properties used by the HSAIL Finalizer. The description 1426 field has the following layout: 1427 1428 .. code:: c 1429 1430 struct amdgpu_hsa_note_hsail_s { 1431 uint32_t hsail_major_version; 1432 uint32_t hsail_minor_version; 1433 uint8_t profile; 1434 uint8_t machine_model; 1435 uint8_t default_float_round; 1436 }; 1437 1438``NT_AMD_HSA_ISA_VERSION`` 1439 Specifies the target ISA version. The description field has the following layout: 1440 1441 .. code:: c 1442 1443 struct amdgpu_hsa_note_isa_s { 1444 uint16_t vendor_name_size; 1445 uint16_t architecture_name_size; 1446 uint32_t major; 1447 uint32_t minor; 1448 uint32_t stepping; 1449 char vendor_and_architecture_name[1]; 1450 }; 1451 1452 ``vendor_name_size`` and ``architecture_name_size`` are the length of the 1453 vendor and architecture names respectively, including the NUL character. 1454 1455 ``vendor_and_architecture_name`` contains the NUL terminates string for the 1456 vendor, immediately followed by the NUL terminated string for the 1457 architecture. 1458 1459 This note record is used by the HSA runtime loader. 1460 1461 Code object V2 only supports a limited number of processors and has fixed 1462 settings for target features. See 1463 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of 1464 processors and the corresponding target ID. In the table the note record ISA 1465 name is a concatenation of the vendor name, architecture name, major, minor, 1466 and stepping separated by a ":". 1467 1468 The target ID column shows the processor name and fixed target features used 1469 by the LLVM compiler. The LLVM compiler does not generate a 1470 ``NT_AMD_HSA_HSAIL`` note record. 1471 1472 A code object generated by the Finalizer also uses code object V2 and always 1473 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and 1474 ``sramecc`` target feature is as shown in 1475 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack`` 1476 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` 1477 bit. 1478 1479``NT_AMD_HSA_ISA_NAME`` 1480 Specifies the target ISA name as a non-NUL terminated string. 1481 1482 This note record is not used by the HSA runtime loader. 1483 1484 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object 1485 V2's limited support of processors and fixed settings for target features. 1486 1487 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping 1488 from the string to the corresponding target ID. If the ``xnack`` target 1489 feature is supported and enabled, the string produced by the LLVM compiler 1490 will may have a ``+xnack`` appended. The Finlizer did not do the appending and 1491 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit. 1492 1493``NT_AMD_HSA_METADATA`` 1494 Specifies extensible metadata associated with the code objects executed on HSA 1495 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the 1496 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See 1497 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object 1498 metadata string. 1499 1500 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings 1501 :name: amdgpu-elf-note-record-supported_processors-v2-table 1502 1503 ===================== ========================== 1504 Note Record ISA Name Target ID 1505 ===================== ========================== 1506 ``AMD:AMDGPU:6:0:0`` ``gfx600`` 1507 ``AMD:AMDGPU:6:0:1`` ``gfx601`` 1508 ``AMD:AMDGPU:6:0:2`` ``gfx602`` 1509 ``AMD:AMDGPU:7:0:0`` ``gfx700`` 1510 ``AMD:AMDGPU:7:0:1`` ``gfx701`` 1511 ``AMD:AMDGPU:7:0:2`` ``gfx702`` 1512 ``AMD:AMDGPU:7:0:3`` ``gfx703`` 1513 ``AMD:AMDGPU:7:0:4`` ``gfx704`` 1514 ``AMD:AMDGPU:7:0:5`` ``gfx705`` 1515 ``AMD:AMDGPU:8:0:0`` ``gfx802`` 1516 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+`` 1517 ``AMD:AMDGPU:8:0:2`` ``gfx802`` 1518 ``AMD:AMDGPU:8:0:3`` ``gfx803`` 1519 ``AMD:AMDGPU:8:0:4`` ``gfx803`` 1520 ``AMD:AMDGPU:8:0:5`` ``gfx805`` 1521 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+`` 1522 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-`` 1523 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+`` 1524 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-`` 1525 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+`` 1526 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-`` 1527 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+`` 1528 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-`` 1529 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+`` 1530 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-`` 1531 ===================== ========================== 1532 1533.. _amdgpu-note-records-v3-onwards: 1534 1535Code Object V3 and Above Note Records 1536~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 1537 1538The AMDGPU backend code object uses the following ELF note record in the 1539``.note`` section when compiling for code object V3 and above. 1540 1541The note record vendor field is "AMDGPU". 1542 1543Additional note records may be present, but any which are not documented here 1544are deprecated and should not be used. 1545 1546 .. table:: AMDGPU Code Object V3 and Above ELF Note Records 1547 :name: amdgpu-elf-note-records-table-v3-onwards 1548 1549 ======== ============================== ====================================== 1550 Name Type Description 1551 ======== ============================== ====================================== 1552 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_ 1553 binary format. 1554 ======== ============================== ====================================== 1555 1556.. 1557 1558 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values 1559 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards 1560 1561 ============================== ===== 1562 Name Value 1563 ============================== ===== 1564 *reserved* 0-31 1565 ``NT_AMDGPU_METADATA`` 32 1566 ============================== ===== 1567 1568``NT_AMDGPU_METADATA`` 1569 Specifies extensible metadata associated with an AMDGPU code object. It is 1570 encoded as a map in the Message Pack [MsgPack]_ binary data format. See 1571 :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 1572 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and 1573 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the 1574 ``amdhsa`` OS. 1575 1576.. _amdgpu-symbols: 1577 1578Symbols 1579------- 1580 1581Symbols include the following: 1582 1583 .. table:: AMDGPU ELF Symbols 1584 :name: amdgpu-elf-symbols-table 1585 1586 ===================== ================== ================ ================== 1587 Name Type Section Description 1588 ===================== ================== ================ ================== 1589 *link-name* ``STT_OBJECT`` - ``.data`` Global variable 1590 - ``.rodata`` 1591 - ``.bss`` 1592 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor 1593 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point 1594 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS 1595 ===================== ================== ================ ================== 1596 1597Global variable 1598 Global variables both used and defined by the compilation unit. 1599 1600 If the symbol is defined in the compilation unit then it is allocated in the 1601 appropriate section according to if it has initialized data or is readonly. 1602 1603 If the symbol is external then its section is ``STN_UNDEF`` and the loader 1604 will resolve relocations using the definition provided by another code object 1605 or explicitly defined by the runtime. 1606 1607 If the symbol resides in local/group memory (LDS) then its section is the 1608 special processor specific section name ``SHN_AMDGPU_LDS``, and the 1609 ``st_value`` field describes alignment requirements as it does for common 1610 symbols. 1611 1612 .. TODO:: 1613 1614 Add description of linked shared object symbols. Seems undefined symbols 1615 are marked as STT_NOTYPE. 1616 1617Kernel descriptor 1618 Every HSA kernel has an associated kernel descriptor. It is the address of the 1619 kernel descriptor that is used in the AQL dispatch packet used to invoke the 1620 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is 1621 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`. 1622 1623Kernel entry point 1624 Every HSA kernel also has a symbol for its machine code entry point. 1625 1626.. _amdgpu-relocation-records: 1627 1628Relocation Records 1629------------------ 1630 1631AMDGPU backend generates ``Elf64_Rela`` relocation records. Supported 1632relocatable fields are: 1633 1634``word32`` 1635 This specifies a 32-bit field occupying 4 bytes with arbitrary byte 1636 alignment. These values use the same byte order as other word values in the 1637 AMDGPU architecture. 1638 1639``word64`` 1640 This specifies a 64-bit field occupying 8 bytes with arbitrary byte 1641 alignment. These values use the same byte order as other word values in the 1642 AMDGPU architecture. 1643 1644Following notations are used for specifying relocation calculations: 1645 1646**A** 1647 Represents the addend used to compute the value of the relocatable field. 1648 1649**G** 1650 Represents the offset into the global offset table at which the relocation 1651 entry's symbol will reside during execution. 1652 1653**GOT** 1654 Represents the address of the global offset table. 1655 1656**P** 1657 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``) 1658 of the storage unit being relocated (computed using ``r_offset``). 1659 1660**S** 1661 Represents the value of the symbol whose index resides in the relocation 1662 entry. Relocations not using this must specify a symbol index of 1663 ``STN_UNDEF``. 1664 1665**B** 1666 Represents the base address of a loaded executable or shared object which is 1667 the difference between the ELF address and the actual load address. 1668 Relocations using this are only valid in executable or shared objects. 1669 1670The following relocation types are supported: 1671 1672 .. table:: AMDGPU ELF Relocation Records 1673 :name: amdgpu-elf-relocation-records-table 1674 1675 ========================== ======= ===== ========== ============================== 1676 Relocation Type Kind Value Field Calculation 1677 ========================== ======= ===== ========== ============================== 1678 ``R_AMDGPU_NONE`` 0 *none* *none* 1679 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF 1680 Dynamic 1681 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 32 1682 Dynamic 1683 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A 1684 Dynamic 1685 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P 1686 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P 1687 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A 1688 Dynamic 1689 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P 1690 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF 1691 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 32 1692 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF 1693 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 32 1694 *reserved* 12 1695 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A 1696 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 4 1697 ========================== ======= ===== ========== ============================== 1698 1699``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by 1700the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``. 1701 1702There is no current OS loader support for 32-bit programs and so 1703``R_AMDGPU_ABS32`` is not used. 1704 1705.. _amdgpu-loaded-code-object-path-uniform-resource-identifier: 1706 1707Loaded Code Object Path Uniform Resource Identifier (URI) 1708--------------------------------------------------------- 1709 1710The AMD GPU code object loader represents the path of the ELF shared object from 1711which the code object was loaded as a textual Uniform Resource Identifier (URI). 1712Note that the code object is the in memory loaded relocated form of the ELF 1713shared object. Multiple code objects may be loaded at different memory 1714addresses in the same process from the same ELF shared object. 1715 1716The loaded code object path URI syntax is defined by the following BNF syntax: 1717 1718.. code:: 1719 1720 code_object_uri ::== file_uri | memory_uri 1721 file_uri ::== "file://" file_path [ range_specifier ] 1722 memory_uri ::== "memory://" process_id range_specifier 1723 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number 1724 file_path ::== URI_ENCODED_OS_FILE_PATH 1725 process_id ::== DECIMAL_NUMBER 1726 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER 1727 1728**number** 1729 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X", 1730 and octal values by "0". 1731 1732**file_path** 1733 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding, 1734 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is 1735 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in 1736 the path are separated by "/". 1737 1738**offset** 1739 Is a 0-based byte offset to the start of the code object. For a file URI, it 1740 is from the start of the file specified by the ``file_path``, and if omitted 1741 defaults to 0. For a memory URI, it is the memory address and is required. 1742 1743**size** 1744 Is the number of bytes in the code object. For a file URI, if omitted it 1745 defaults to the size of the file. It is required for a memory URI. 1746 1747**process_id** 1748 Is the identity of the process owning the memory. For Linux it is the C 1749 unsigned integral decimal literal for the process ID (PID). 1750 1751For example: 1752 1753.. code:: 1754 1755 file:///dir1/dir2/file1 1756 file:///dir3/dir4/file2#offset=0x2000&size=3000 1757 memory://1234#offset=0x20000&size=3000 1758 1759.. _amdgpu-dwarf-debug-information: 1760 1761DWARF Debug Information 1762======================= 1763 1764.. warning:: 1765 1766 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that 1767 is not currently fully implemented and is subject to change. 1768 1769AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see 1770:ref:`amdgpu-elf-code-object`) which contain information that maps the code 1771object executable code and data to the source language constructs. It can be 1772used by tools such as debuggers and profilers. It uses features defined in 1773:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in 1774DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension. 1775 1776This section defines the AMDGPU target architecture specific DWARF mappings. 1777 1778.. _amdgpu-dwarf-register-identifier: 1779 1780Register Identifier 1781------------------- 1782 1783This section defines the AMDGPU target architecture register numbers used in 1784DWARF operation expressions (see DWARF Version 5 section 2.5 and 1785:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information 1786instructions (see DWARF Version 5 section 6.4 and 1787:ref:`amdgpu-dwarf-call-frame-information`). 1788 1789A single code object can contain code for kernels that have different wavefront 1790sizes. The vector registers and some scalar registers are based on the wavefront 1791size. AMDGPU defines distinct DWARF registers for each wavefront size. This 1792simplifies the consumer of the DWARF so that each register has a fixed size, 1793rather than being dynamic according to the wavefront size mode. Similarly, 1794distinct DWARF registers are defined for those registers that vary in size 1795according to the process address size. This allows a consumer to treat a 1796specific AMDGPU processor as a single architecture regardless of how it is 1797configured at run time. The compiler explicitly specifies the DWARF registers 1798that match the mode in which the code it is generating will be executed. 1799 1800DWARF registers are encoded as numbers, which are mapped to architecture 1801registers. The mapping for AMDGPU is defined in 1802:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same 1803mapping. 1804 1805.. table:: AMDGPU DWARF Register Mapping 1806 :name: amdgpu-dwarf-register-mapping-table 1807 1808 ============== ================= ======== ================================== 1809 DWARF Register AMDGPU Register Bit Size Description 1810 ============== ================= ======== ================================== 1811 0 PC_32 32 Program Counter (PC) when 1812 executing in a 32-bit process 1813 address space. Used in the CFI to 1814 describe the PC of the calling 1815 frame. 1816 1 EXEC_MASK_32 32 Execution Mask Register when 1817 executing in wavefront 32 mode. 1818 2-15 *Reserved* *Reserved for highly accessed 1819 registers using DWARF shortcut.* 1820 16 PC_64 64 Program Counter (PC) when 1821 executing in a 64-bit process 1822 address space. Used in the CFI to 1823 describe the PC of the calling 1824 frame. 1825 17 EXEC_MASK_64 64 Execution Mask Register when 1826 executing in wavefront 64 mode. 1827 18-31 *Reserved* *Reserved for highly accessed 1828 registers using DWARF shortcut.* 1829 32-95 SGPR0-SGPR63 32 Scalar General Purpose 1830 Registers. 1831 96-127 *Reserved* *Reserved for frequently accessed 1832 registers using DWARF 1-byte ULEB.* 1833 128 STATUS 32 Status Register. 1834 129-511 *Reserved* *Reserved for future Scalar 1835 Architectural Registers.* 1836 512 VCC_32 32 Vector Condition Code Register 1837 when executing in wavefront 32 1838 mode. 1839 513-767 *Reserved* *Reserved for future Vector 1840 Architectural Registers when 1841 executing in wavefront 32 mode.* 1842 768 VCC_64 64 Vector Condition Code Register 1843 when executing in wavefront 64 1844 mode. 1845 769-1023 *Reserved* *Reserved for future Vector 1846 Architectural Registers when 1847 executing in wavefront 64 mode.* 1848 1024-1087 *Reserved* *Reserved for padding.* 1849 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers. 1850 1130-1535 *Reserved* *Reserved for future Scalar 1851 General Purpose Registers.* 1852 1536-1791 VGPR0-VGPR255 32*32 Vector General Purpose Registers 1853 when executing in wavefront 32 1854 mode. 1855 1792-2047 *Reserved* *Reserved for future Vector 1856 General Purpose Registers when 1857 executing in wavefront 32 mode.* 1858 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers 1859 when executing in wavefront 32 1860 mode. 1861 2304-2559 *Reserved* *Reserved for future Vector 1862 Accumulation Registers when 1863 executing in wavefront 32 mode.* 1864 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers 1865 when executing in wavefront 64 1866 mode. 1867 2816-3071 *Reserved* *Reserved for future Vector 1868 General Purpose Registers when 1869 executing in wavefront 64 mode.* 1870 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers 1871 when executing in wavefront 64 1872 mode. 1873 3328-3583 *Reserved* *Reserved for future Vector 1874 Accumulation Registers when 1875 executing in wavefront 64 mode.* 1876 ============== ================= ======== ================================== 1877 1878The vector registers are represented as the full size for the wavefront. They 1879are organized as consecutive dwords (32-bits), one per lane, with the dword at 1880the least significant bit position corresponding to lane 0 and so forth. DWARF 1881location expressions involving the ``DW_OP_LLVM_offset`` and 1882``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector 1883register corresponding to the lane that is executing the current thread of 1884execution in languages that are implemented using a SIMD or SIMT execution 1885model. 1886 1887If the wavefront size is 32 lanes then the wavefront 32 mode register 1888definitions are used. If the wavefront size is 64 lanes then the wavefront 64 1889mode register definitions are used. Some AMDGPU targets support executing in 1890both wavefront 32 and wavefront 64 mode. The register definitions corresponding 1891to the wavefront mode of the generated code will be used. 1892 1893If code is generated to execute in a 32-bit process address space, then the 189432-bit process address space register definitions are used. If code is generated 1895to execute in a 64-bit process address space, then the 64-bit process address 1896space register definitions are used. The ``amdgcn`` target only supports the 189764-bit process address space. 1898 1899.. _amdgpu-dwarf-address-class-identifier: 1900 1901Address Class Identifier 1902------------------------ 1903 1904The DWARF address class represents the source language memory space. See DWARF 1905Version 5 section 2.12 which is updated by the *DWARF Extensions For 1906Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1907 1908The DWARF address class mapping used for AMDGPU is defined in 1909:ref:`amdgpu-dwarf-address-class-mapping-table`. 1910 1911.. table:: AMDGPU DWARF Address Class Mapping 1912 :name: amdgpu-dwarf-address-class-mapping-table 1913 1914 ========================= ====== ================= 1915 DWARF AMDGPU 1916 -------------------------------- ----------------- 1917 Address Class Name Value Address Space 1918 ========================= ====== ================= 1919 ``DW_ADDR_none`` 0x0000 Generic (Flat) 1920 ``DW_ADDR_LLVM_global`` 0x0001 Global 1921 ``DW_ADDR_LLVM_constant`` 0x0002 Global 1922 ``DW_ADDR_LLVM_group`` 0x0003 Local (group/LDS) 1923 ``DW_ADDR_LLVM_private`` 0x0004 Private (Scratch) 1924 ``DW_ADDR_AMDGPU_region`` 0x8000 Region (GDS) 1925 ========================= ====== ================= 1926 1927The DWARF address class values defined in the *DWARF Extensions For 1928Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses` are used. 1929 1930In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is 1931available for use for the AMD extension for access to the hardware GDS memory 1932which is scratchpad memory allocated per device. 1933 1934For AMDGPU if no ``DW_AT_address_class`` attribute is present, then the default 1935address class of ``DW_ADDR_none`` is used. 1936 1937See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU 1938mapping of DWARF address classes to DWARF address spaces, including address size 1939and NULL value. 1940 1941.. _amdgpu-dwarf-address-space-identifier: 1942 1943Address Space Identifier 1944------------------------ 1945 1946DWARF address spaces correspond to target architecture specific linear 1947addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions 1948For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-segment_addresses`. 1949 1950The DWARF address space mapping used for AMDGPU is defined in 1951:ref:`amdgpu-dwarf-address-space-mapping-table`. 1952 1953.. table:: AMDGPU DWARF Address Space Mapping 1954 :name: amdgpu-dwarf-address-space-mapping-table 1955 1956 ======================================= ===== ======= ======== ================= ======================= 1957 DWARF AMDGPU Notes 1958 --------------------------------------- ----- ---------------- ----------------- ----------------------- 1959 Address Space Name Value Address Bit Size Address Space 1960 --------------------------------------- ----- ------- -------- ----------------- ----------------------- 1961 .. 64-bit 32-bit 1962 process process 1963 address address 1964 space space 1965 ======================================= ===== ======= ======== ================= ======================= 1966 ``DW_ASPACE_none`` 0x00 64 32 Global *default address space* 1967 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat) 1968 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS) 1969 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS) 1970 *Reserved* 0x04 1971 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane* 1972 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront* 1973 ======================================= ===== ======= ======== ================= ======================= 1974 1975See :ref:`amdgpu-address-spaces` for information on the AMDGPU address spaces 1976including address size and NULL value. 1977 1978The ``DW_ASPACE_none`` address space is the default target architecture address 1979space used in DWARF operations that do not specify an address space. It 1980therefore has to map to the global address space so that the ``DW_OP_addr*`` and 1981related operations can refer to addresses in the program code. 1982 1983The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to 1984specify the flat address space. If the address corresponds to an address in the 1985local address space, then it corresponds to the wavefront that is executing the 1986focused thread of execution. If the address corresponds to an address in the 1987private address space, then it corresponds to the lane that is executing the 1988focused thread of execution for languages that are implemented using a SIMD or 1989SIMT execution model. 1990 1991.. note:: 1992 1993 CUDA-like languages such as HIP that do not have address spaces in the 1994 language type system, but do allow variables to be allocated in different 1995 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic`` 1996 address space in the DWARF expression operations as the default address space 1997 is the global address space. 1998 1999The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to 2000specify the local address space corresponding to the wavefront that is executing 2001the focused thread of execution. 2002 2003The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions 2004to specify the private address space corresponding to the lane that is executing 2005the focused thread of execution for languages that are implemented using a SIMD 2006or SIMT execution model. 2007 2008The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions 2009to specify the unswizzled private address space corresponding to the wavefront 2010that is executing the focused thread of execution. The wavefront view of private 2011memory is the per wavefront unswizzled backing memory layout defined in 2012:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first 2013location for the backing memory of the wavefront (namely the address is not 2014offset by ``wavefront-scratch-base``). The following formula can be used to 2015convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a 2016``DW_ASPACE_AMDGPU_private_wave`` address: 2017 2018:: 2019 2020 private-address-wavefront = 2021 ((private-address-lane / 4) * wavefront-size * 4) + 2022 (wavefront-lane-id * 4) + (private-address-lane % 4) 2023 2024If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start 2025of the dwords for each lane starting with lane 0 is required, then this 2026simplifies to: 2027 2028:: 2029 2030 private-address-wavefront = 2031 private-address-lane * wavefront-size 2032 2033A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a 2034complete spilled vector register back into a complete vector register in the 2035CFI. The frame pointer can be a private lane address which is dword aligned, 2036which can be shifted to multiply by the wavefront size, and then used to form a 2037private wavefront address that gives a location for a contiguous set of dwords, 2038one per lane, where the vector register dwords are spilled. The compiler knows 2039the wavefront size since it generates the code. Note that the type of the 2040address may have to be converted as the size of a 2041``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a 2042``DW_ASPACE_AMDGPU_private_wave`` address. 2043 2044.. _amdgpu-dwarf-lane-identifier: 2045 2046Lane identifier 2047--------------- 2048 2049DWARF lane identifies specify a target architecture lane position for hardware 2050that executes in a SIMD or SIMT manner, and on which a source language maps its 2051threads of execution onto those lanes. The DWARF lane identifier is pushed by 2052the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 5 2053section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging* 2054section :ref:`amdgpu-dwarf-operation-expressions`. 2055 2056For AMDGPU, the lane identifier corresponds to the hardware lane ID of a 2057wavefront. It is numbered from 0 to the wavefront size minus 1. 2058 2059Operation Expressions 2060--------------------- 2061 2062DWARF expressions are used to compute program values and the locations of 2063program objects. See DWARF Version 5 section 2.5 and 2064:ref:`amdgpu-dwarf-operation-expressions`. 2065 2066DWARF location descriptions describe how to access storage which includes memory 2067and registers. When accessing storage on AMDGPU, bytes are ordered with least 2068significant bytes first, and bits are ordered within bytes with least 2069significant bits first. 2070 2071For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe 2072unwinding vector registers that are spilled under the execution mask to memory: 2073the zero-single location description is the vector register, and the one-single 2074location description is the spilled memory location description. The 2075``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the 2076memory location description. 2077 2078In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the 2079``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is 2080controlled by the execution mask. An undefined location description together 2081with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry 2082to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example. 2083 2084Debugger Information Entry Attributes 2085------------------------------------- 2086 2087This section describes how certain debugger information entry attributes are 2088used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.1 2089which are updated by *DWARF Extensions For Heterogeneous Debugging* section 2090:ref:`amdgpu-dwarf-low-level-information` and 2091:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`. 2092 2093.. _amdgpu-dwarf-dw-at-llvm-lane-pc: 2094 2095``DW_AT_LLVM_lane_pc`` 2096~~~~~~~~~~~~~~~~~~~~~~ 2097 2098For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program 2099location of the separate lanes of a SIMT thread. 2100 2101If the lane is an active lane then this will be the same as the current program 2102location. 2103 2104If the lane is inactive, but was active on entry to the subprogram, then this is 2105the program location in the subprogram at which execution of the lane is 2106conceptual positioned. 2107 2108If the lane was not active on entry to the subprogram, then this will be the 2109undefined location. A client debugger can check if the lane is part of a valid 2110work-group by checking that the lane is in the range of the associated 2111work-group within the grid, accounting for partial work-groups. If it is not, 2112then the debugger can omit any information for the lane. Otherwise, the debugger 2113may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the 2114calling subprogram until it finds a non-undefined location. Conceptually the 2115lane only has the call frames that it has a non-undefined 2116``DW_AT_LLVM_lane_pc``. 2117 2118The following example illustrates how the AMDGPU backend can generate a DWARF 2119location list expression for the nested ``IF/THEN/ELSE`` structures of the 2120following subprogram pseudo code for a target with 64 lanes per wavefront. 2121 2122.. code:: 2123 :number-lines: 2124 2125 SUBPROGRAM X 2126 BEGIN 2127 a; 2128 IF (c1) THEN 2129 b; 2130 IF (c2) THEN 2131 c; 2132 ELSE 2133 d; 2134 ENDIF 2135 e; 2136 ELSE 2137 f; 2138 ENDIF 2139 g; 2140 END 2141 2142The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the 2143execution mask (``EXEC``) to linearize the control flow. The condition is 2144evaluated to make a mask of the lanes for which the condition evaluates to true. 2145First the ``THEN`` region is executed by setting the ``EXEC`` mask to the 2146logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the 2147``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of 2148the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE`` 2149region the ``EXEC`` mask is restored to the value it had at the beginning of the 2150region. This is shown below. Other approaches are possible, but the basic 2151concept is the same. 2152 2153.. code:: 2154 :number-lines: 2155 2156 $lex_start: 2157 a; 2158 %1 = EXEC 2159 %2 = c1 2160 $lex_1_start: 2161 EXEC = %1 & %2 2162 $if_1_then: 2163 b; 2164 %3 = EXEC 2165 %4 = c2 2166 $lex_1_1_start: 2167 EXEC = %3 & %4 2168 $lex_1_1_then: 2169 c; 2170 EXEC = ~EXEC & %3 2171 $lex_1_1_else: 2172 d; 2173 EXEC = %3 2174 $lex_1_1_end: 2175 e; 2176 EXEC = ~EXEC & %1 2177 $lex_1_else: 2178 f; 2179 EXEC = %1 2180 $lex_1_end: 2181 g; 2182 $lex_end: 2183 2184To create the DWARF location list expression that defines the location 2185description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE`` 2186pseudo instruction can be used to annotate the linearized control flow. This can 2187be done by defining an artificial variable for the lane PC. The DWARF location 2188list expression created for it is used as the value of the 2189``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry. 2190 2191A DWARF procedure is defined for each well nested structured control flow region 2192which provides the conceptual lane program location for a lane if it is not 2193active (namely it is divergent). The DWARF operation expression for each region 2194conceptually inherits the value of the immediately enclosing region and modifies 2195it according to the semantics of the region. 2196 2197For an ``IF/THEN/ELSE`` region the divergent program location is at the start of 2198the region for the ``THEN`` region since it is executed first. For the ``ELSE`` 2199region the divergent program location is at the end of the ``IF/THEN/ELSE`` 2200region since the ``THEN`` region has completed. 2201 2202The lane PC artificial variable is assigned at each region transition. It uses 2203the immediately enclosing region's DWARF procedure to compute the program 2204location for each lane assuming they are divergent, and then modifies the result 2205by inserting the current program location for each lane that the ``EXEC`` mask 2206indicates is active. 2207 2208By having separate DWARF procedures for each region, they can be reused to 2209define the value for any nested region. This reduces the total size of the DWARF 2210operation expressions. 2211 2212The following provides an example using pseudo LLVM MIR. 2213 2214.. code:: 2215 :number-lines: 2216 2217 $lex_start: 2218 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[ 2219 DW_AT_name = "__uint64"; 2220 DW_AT_byte_size = 8; 2221 DW_AT_encoding = DW_ATE_unsigned; 2222 ]; 2223 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[ 2224 DW_AT_name = "__active_lane_pc"; 2225 DW_AT_location = [ 2226 DW_OP_regx PC; 2227 DW_OP_LLVM_extend 64, 64; 2228 DW_OP_regval_type EXEC, %uint_64; 2229 DW_OP_LLVM_select_bit_piece 64, 64; 2230 ]; 2231 ]; 2232 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[ 2233 DW_AT_name = "__divergent_lane_pc"; 2234 DW_AT_location = [ 2235 DW_OP_LLVM_undefined; 2236 DW_OP_LLVM_extend 64, 64; 2237 ]; 2238 ]; 2239 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2240 DW_OP_call_ref %__divergent_lane_pc; 2241 DW_OP_call_ref %__active_lane_pc; 2242 ]; 2243 a; 2244 %1 = EXEC; 2245 DBG_VALUE %1, $noreg, %__lex_1_save_exec; 2246 %2 = c1; 2247 $lex_1_start: 2248 EXEC = %1 & %2; 2249 $lex_1_then: 2250 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[ 2251 DW_AT_name = "__divergent_lane_pc_1_then"; 2252 DW_AT_location = DIExpression[ 2253 DW_OP_call_ref %__divergent_lane_pc; 2254 DW_OP_addrx &lex_1_start; 2255 DW_OP_stack_value; 2256 DW_OP_LLVM_extend 64, 64; 2257 DW_OP_call_ref %__lex_1_save_exec; 2258 DW_OP_deref_type 64, %__uint_64; 2259 DW_OP_LLVM_select_bit_piece 64, 64; 2260 ]; 2261 ]; 2262 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2263 DW_OP_call_ref %__divergent_lane_pc_1_then; 2264 DW_OP_call_ref %__active_lane_pc; 2265 ]; 2266 b; 2267 %3 = EXEC; 2268 DBG_VALUE %3, %__lex_1_1_save_exec; 2269 %4 = c2; 2270 $lex_1_1_start: 2271 EXEC = %3 & %4; 2272 $lex_1_1_then: 2273 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[ 2274 DW_AT_name = "__divergent_lane_pc_1_1_then"; 2275 DW_AT_location = DIExpression[ 2276 DW_OP_call_ref %__divergent_lane_pc_1_then; 2277 DW_OP_addrx &lex_1_1_start; 2278 DW_OP_stack_value; 2279 DW_OP_LLVM_extend 64, 64; 2280 DW_OP_call_ref %__lex_1_1_save_exec; 2281 DW_OP_deref_type 64, %__uint_64; 2282 DW_OP_LLVM_select_bit_piece 64, 64; 2283 ]; 2284 ]; 2285 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2286 DW_OP_call_ref %__divergent_lane_pc_1_1_then; 2287 DW_OP_call_ref %__active_lane_pc; 2288 ]; 2289 c; 2290 EXEC = ~EXEC & %3; 2291 $lex_1_1_else: 2292 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[ 2293 DW_AT_name = "__divergent_lane_pc_1_1_else"; 2294 DW_AT_location = DIExpression[ 2295 DW_OP_call_ref %__divergent_lane_pc_1_then; 2296 DW_OP_addrx &lex_1_1_end; 2297 DW_OP_stack_value; 2298 DW_OP_LLVM_extend 64, 64; 2299 DW_OP_call_ref %__lex_1_1_save_exec; 2300 DW_OP_deref_type 64, %__uint_64; 2301 DW_OP_LLVM_select_bit_piece 64, 64; 2302 ]; 2303 ]; 2304 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2305 DW_OP_call_ref %__divergent_lane_pc_1_1_else; 2306 DW_OP_call_ref %__active_lane_pc; 2307 ]; 2308 d; 2309 EXEC = %3; 2310 $lex_1_1_end: 2311 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2312 DW_OP_call_ref %__divergent_lane_pc; 2313 DW_OP_call_ref %__active_lane_pc; 2314 ]; 2315 e; 2316 EXEC = ~EXEC & %1; 2317 $lex_1_else: 2318 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[ 2319 DW_AT_name = "__divergent_lane_pc_1_else"; 2320 DW_AT_location = DIExpression[ 2321 DW_OP_call_ref %__divergent_lane_pc; 2322 DW_OP_addrx &lex_1_end; 2323 DW_OP_stack_value; 2324 DW_OP_LLVM_extend 64, 64; 2325 DW_OP_call_ref %__lex_1_save_exec; 2326 DW_OP_deref_type 64, %__uint_64; 2327 DW_OP_LLVM_select_bit_piece 64, 64; 2328 ]; 2329 ]; 2330 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[ 2331 DW_OP_call_ref %__divergent_lane_pc_1_else; 2332 DW_OP_call_ref %__active_lane_pc; 2333 ]; 2334 f; 2335 EXEC = %1; 2336 $lex_1_end: 2337 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[ 2338 DW_OP_call_ref %__divergent_lane_pc; 2339 DW_OP_call_ref %__active_lane_pc; 2340 ]; 2341 g; 2342 $lex_end: 2343 2344The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements 2345that are active, with the current program location. 2346 2347Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for 2348the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo 2349instruction, location list entries will be created that describe where the 2350artificial variables are allocated at any given program location. The compiler 2351may allocate them to registers or spill them to memory. 2352 2353The DWARF procedures for each region use the values of the saved execution mask 2354artificial variables to only update the lanes that are active on entry to the 2355region. All other lanes retain the value of the enclosing region where they were 2356last active. If they were not active on entry to the subprogram, then will have 2357the undefined location description. 2358 2359Other structured control flow regions can be handled similarly. For example, 2360loops would set the divergent program location for the region at the end of the 2361loop. Any lanes active will be in the loop, and any lanes not active must have 2362exited the loop. 2363 2364An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of 2365``IF/THEN/ELSE`` regions. 2366 2367The DWARF procedures can use the active lane artificial variable described in 2368:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual 2369``EXEC`` mask in order to support whole or quad wavefront mode. 2370 2371.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane: 2372 2373``DW_AT_LLVM_active_lane`` 2374~~~~~~~~~~~~~~~~~~~~~~~~~~ 2375 2376The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information 2377entry is used to specify the lanes that are conceptually active for a SIMT 2378thread. 2379 2380The execution mask may be modified to implement whole or quad wavefront mode 2381operations. For example, all lanes may need to temporarily be made active to 2382execute a whole wavefront operation. Such regions would save the ``EXEC`` mask, 2383update it to enable the necessary lanes, perform the operations, and then 2384restore the ``EXEC`` mask from the saved value. While executing the whole 2385wavefront region, the conceptual execution mask is the saved value, not the 2386``EXEC`` value. 2387 2388This is handled by defining an artificial variable for the active lane mask. The 2389active lane mask artificial variable would be the actual ``EXEC`` mask for 2390normal regions, and the saved execution mask for regions where the mask is 2391temporarily updated. The location list expression created for this artificial 2392variable is used to define the value of the ``DW_AT_LLVM_active_lane`` 2393attribute. 2394 2395``DW_AT_LLVM_augmentation`` 2396~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2397 2398For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit 2399debugger information entry has the following value for the augmentation string: 2400 2401:: 2402 2403 [amdgpu:v0.0] 2404 2405The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2406extensions used in the DWARF of the compilation unit. The version number 2407conforms to [SEMVER]_. 2408 2409Call Frame Information 2410---------------------- 2411 2412DWARF Call Frame Information (CFI) describes how a consumer can virtually 2413*unwind* call frames in a running process or core dump. See DWARF Version 5 2414section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`. 2415 2416For AMDGPU, the Common Information Entry (CIE) fields have the following values: 2417 24181. ``augmentation`` string contains the following null-terminated UTF-8 string: 2419 2420 :: 2421 2422 [amd:v0.0] 2423 2424 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU 2425 extensions used in this CIE or to the FDEs that use it. The version number 2426 conforms to [SEMVER]_. 2427 24282. ``address_size`` for the ``Global`` address space is defined in 2429 :ref:`amdgpu-dwarf-address-space-identifier`. 2430 24313. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector. 2432 24334. ``code_alignment_factor`` is 4 bytes. 2434 2435 .. TODO:: 2436 2437 Add to :ref:`amdgpu-processor-table` table. 2438 24395. ``data_alignment_factor`` is 4 bytes. 2440 2441 .. TODO:: 2442 2443 Add to :ref:`amdgpu-processor-table` table. 2444 24456. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64`` 2446 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`. 2447 24487. ``initial_instructions`` Since a subprogram X with fewer registers can be 2449 called from subprogram Y that has more allocated, X will not change any of 2450 the extra registers as it cannot access them. Therefore, the default rule 2451 for all columns is ``same value``. 2452 2453For AMDGPU the register number follows the numbering defined in 2454:ref:`amdgpu-dwarf-register-identifier`. 2455 2456For AMDGPU the instructions are variable size. A consumer can subtract 1 from 2457the return address to get the address of a byte within the call site 2458instructions. See DWARF Version 5 section 6.4.4. 2459 2460Accelerated Access 2461------------------ 2462 2463See DWARF Version 5 section 6.1. 2464 2465Lookup By Name Section Header 2466~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2467 2468See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`. 2469 2470For AMDGPU the lookup by name section header table: 2471 2472``augmentation_string_size`` (uword) 2473 2474 Set to the length of the ``augmentation_string`` value which is always a 2475 multiple of 4. 2476 2477``augmentation_string`` (sequence of UTF-8 characters) 2478 2479 Contains the following UTF-8 string null padded to a multiple of 4 bytes: 2480 2481 :: 2482 2483 [amdgpu:v0.0] 2484 2485 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU 2486 extensions used in the DWARF of this index. The version number conforms to 2487 [SEMVER]_. 2488 2489 .. note:: 2490 2491 This is different to the DWARF Version 5 definition that requires the first 2492 4 characters to be the vendor ID. But this is consistent with the other 2493 augmentation strings and does allow multiple vendor contributions. However, 2494 backwards compatibility may be more desirable. 2495 2496Lookup By Address Section Header 2497~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2498 2499See DWARF Version 5 section 6.1.2. 2500 2501For AMDGPU the lookup by address section header table: 2502 2503``address_size`` (ubyte) 2504 2505 Match the address size for the ``Global`` address space defined in 2506 :ref:`amdgpu-dwarf-address-space-identifier`. 2507 2508``segment_selector_size`` (ubyte) 2509 2510 AMDGPU does not use a segment selector so this is 0. The entries in the 2511 ``.debug_aranges`` do not have a segment selector. 2512 2513Line Number Information 2514----------------------- 2515 2516See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`. 2517 2518AMDGPU does not use the ``isa`` state machine registers and always sets it to 0. 2519The instruction set must be obtained from the ELF file header ``e_flags`` field 2520in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header 2521<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2. 2522 2523.. TODO:: 2524 2525 Should the ``isa`` state machine register be used to indicate if the code is 2526 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA? 2527 2528For AMDGPU the line number program header fields have the following values (see 2529DWARF Version 5 section 6.2.4): 2530 2531``address_size`` (ubyte) 2532 Matches the address size for the ``Global`` address space defined in 2533 :ref:`amdgpu-dwarf-address-space-identifier`. 2534 2535``segment_selector_size`` (ubyte) 2536 AMDGPU does not use a segment selector so this is 0. 2537 2538``minimum_instruction_length`` (ubyte) 2539 For GFX9-GFX10 this is 4. 2540 2541``maximum_operations_per_instruction`` (ubyte) 2542 For GFX9-GFX10 this is 1. 2543 2544Source text for online-compiled programs (for example, those compiled by the 2545OpenCL language runtime) may be embedded into the DWARF Version 5 line table. 2546See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For 2547Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source 2548<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`. 2549 2550The Clang option used to control source embedding in AMDGPU is defined in 2551:ref:`amdgpu-clang-debug-options-table`. 2552 2553 .. table:: AMDGPU Clang Debug Options 2554 :name: amdgpu-clang-debug-options-table 2555 2556 ==================== ================================================== 2557 Debug Flag Description 2558 ==================== ================================================== 2559 -g[no-]embed-source Enable/disable embedding source text in DWARF 2560 debug sections. Useful for environments where 2561 source cannot be written to disk, such as 2562 when performing online compilation. 2563 ==================== ================================================== 2564 2565For example: 2566 2567``-gembed-source`` 2568 Enable the embedded source. 2569 2570``-gno-embed-source`` 2571 Disable the embedded source. 2572 257332-Bit and 64-Bit DWARF Formats 2574------------------------------- 2575 2576See DWARF Version 5 section 7.4 and 2577:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`. 2578 2579For AMDGPU: 2580 2581* For the ``amdgcn`` target architecture only the 64-bit process address space 2582 is supported. 2583 2584* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates 2585 the 32-bit DWARF format. 2586 2587Unit Headers 2588------------ 2589 2590For AMDGPU the following values apply for each of the unit headers described in 2591DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3: 2592 2593``address_size`` (ubyte) 2594 Matches the address size for the ``Global`` address space defined in 2595 :ref:`amdgpu-dwarf-address-space-identifier`. 2596 2597.. _amdgpu-code-conventions: 2598 2599Code Conventions 2600================ 2601 2602This section provides code conventions used for each supported target triple OS 2603(see :ref:`amdgpu-target-triples`). 2604 2605AMDHSA 2606------ 2607 2608This section provides code conventions used when the target triple OS is 2609``amdhsa`` (see :ref:`amdgpu-target-triples`). 2610 2611.. _amdgpu-amdhsa-code-object-metadata: 2612 2613Code Object Metadata 2614~~~~~~~~~~~~~~~~~~~~ 2615 2616The code object metadata specifies extensible metadata associated with the code 2617objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The 2618encoding and semantics of this metadata depends on the code object version; see 2619:ref:`amdgpu-amdhsa-code-object-metadata-v2`, 2620:ref:`amdgpu-amdhsa-code-object-metadata-v3`, 2621:ref:`amdgpu-amdhsa-code-object-metadata-v4` and 2622:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 2623 2624Code object metadata is specified in a note record (see 2625:ref:`amdgpu-note-records`) and is required when the target triple OS is 2626``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum 2627information necessary to support the HSA compatible runtime kernel queries. For 2628example, the segment sizes needed in a dispatch packet. In addition, a 2629high-level language runtime may require other information to be included. For 2630example, the AMD OpenCL runtime records kernel argument information. 2631 2632.. _amdgpu-amdhsa-code-object-metadata-v2: 2633 2634Code Object V2 Metadata 2635+++++++++++++++++++++++ 2636 2637.. warning:: 2638 Code object V2 is not the default code object version emitted by this version 2639 of LLVM. 2640 2641Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record 2642(see :ref:`amdgpu-note-records-v2`). 2643 2644The metadata is specified as a YAML formatted string (see [YAML]_ and 2645:doc:`YamlIO`). 2646 2647.. TODO:: 2648 2649 Is the string null terminated? It probably should not if YAML allows it to 2650 contain null characters, otherwise it should be. 2651 2652The metadata is represented as a single YAML document comprised of the mapping 2653defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and 2654referenced tables. 2655 2656For boolean values, the string values of ``false`` and ``true`` are used for 2657false and true respectively. 2658 2659Additional information can be added to the mappings. To avoid conflicts, any 2660non-AMD key names should be prefixed by "*vendor-name*.". 2661 2662 .. table:: AMDHSA Code Object V2 Metadata Map 2663 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table 2664 2665 ========== ============== ========= ======================================= 2666 String Key Value Type Required? Description 2667 ========== ============== ========= ======================================= 2668 "Version" sequence of Required - The first integer is the major 2669 2 integers version. Currently 1. 2670 - The second integer is the minor 2671 version. Currently 0. 2672 "Printf" sequence of Each string is encoded information 2673 strings about a printf function call. The 2674 encoded information is organized as 2675 fields separated by colon (':'): 2676 2677 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 2678 2679 where: 2680 2681 ``ID`` 2682 A 32-bit integer as a unique id for 2683 each printf function call 2684 2685 ``N`` 2686 A 32-bit integer equal to the number 2687 of arguments of printf function call 2688 minus 1 2689 2690 ``S[i]`` (where i = 0, 1, ... , N-1) 2691 32-bit integers for the size in bytes 2692 of the i-th FormatString argument of 2693 the printf function call 2694 2695 FormatString 2696 The format string passed to the 2697 printf function call. 2698 "Kernels" sequence of Required Sequence of the mappings for each 2699 mapping kernel in the code object. See 2700 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table` 2701 for the definition of the mapping. 2702 ========== ============== ========= ======================================= 2703 2704.. 2705 2706 .. table:: AMDHSA Code Object V2 Kernel Metadata Map 2707 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table 2708 2709 ================= ============== ========= ================================ 2710 String Key Value Type Required? Description 2711 ================= ============== ========= ================================ 2712 "Name" string Required Source name of the kernel. 2713 "SymbolName" string Required Name of the kernel 2714 descriptor ELF symbol. 2715 "Language" string Source language of the kernel. 2716 Values include: 2717 2718 - "OpenCL C" 2719 - "OpenCL C++" 2720 - "HCC" 2721 - "OpenMP" 2722 2723 "LanguageVersion" sequence of - The first integer is the major 2724 2 integers version. 2725 - The second integer is the 2726 minor version. 2727 "Attrs" mapping Mapping of kernel attributes. 2728 See 2729 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table` 2730 for the mapping definition. 2731 "Args" sequence of Sequence of mappings of the 2732 mapping kernel arguments. See 2733 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table` 2734 for the definition of the mapping. 2735 "CodeProps" mapping Mapping of properties related to 2736 the kernel code. See 2737 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table` 2738 for the mapping definition. 2739 ================= ============== ========= ================================ 2740 2741.. 2742 2743 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map 2744 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table 2745 2746 =================== ============== ========= ============================== 2747 String Key Value Type Required? Description 2748 =================== ============== ========= ============================== 2749 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values 2750 3 integers must be >=1 and the dispatch 2751 work-group size X, Y, Z must 2752 correspond to the specified 2753 values. Defaults to 0, 0, 0. 2754 2755 Corresponds to the OpenCL 2756 ``reqd_work_group_size`` 2757 attribute. 2758 "WorkGroupSizeHint" sequence of The dispatch work-group size 2759 3 integers X, Y, Z is likely to be the 2760 specified values. 2761 2762 Corresponds to the OpenCL 2763 ``work_group_size_hint`` 2764 attribute. 2765 "VecTypeHint" string The name of a scalar or vector 2766 type. 2767 2768 Corresponds to the OpenCL 2769 ``vec_type_hint`` attribute. 2770 2771 "RuntimeHandle" string The external symbol name 2772 associated with a kernel. 2773 OpenCL runtime allocates a 2774 global buffer for the symbol 2775 and saves the kernel's address 2776 to it, which is used for 2777 device side enqueueing. Only 2778 available for device side 2779 enqueued kernels. 2780 =================== ============== ========= ============================== 2781 2782.. 2783 2784 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map 2785 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table 2786 2787 ================= ============== ========= ================================ 2788 String Key Value Type Required? Description 2789 ================= ============== ========= ================================ 2790 "Name" string Kernel argument name. 2791 "TypeName" string Kernel argument type name. 2792 "Size" integer Required Kernel argument size in bytes. 2793 "Align" integer Required Kernel argument alignment in 2794 bytes. Must be a power of two. 2795 "ValueKind" string Required Kernel argument kind that 2796 specifies how to set up the 2797 corresponding argument. 2798 Values include: 2799 2800 "ByValue" 2801 The argument is copied 2802 directly into the kernarg. 2803 2804 "GlobalBuffer" 2805 A global address space pointer 2806 to the buffer data is passed 2807 in the kernarg. 2808 2809 "DynamicSharedPointer" 2810 A group address space pointer 2811 to dynamically allocated LDS 2812 is passed in the kernarg. 2813 2814 "Sampler" 2815 A global address space 2816 pointer to a S# is passed in 2817 the kernarg. 2818 2819 "Image" 2820 A global address space 2821 pointer to a T# is passed in 2822 the kernarg. 2823 2824 "Pipe" 2825 A global address space pointer 2826 to an OpenCL pipe is passed in 2827 the kernarg. 2828 2829 "Queue" 2830 A global address space pointer 2831 to an OpenCL device enqueue 2832 queue is passed in the 2833 kernarg. 2834 2835 "HiddenGlobalOffsetX" 2836 The OpenCL grid dispatch 2837 global offset for the X 2838 dimension is passed in the 2839 kernarg. 2840 2841 "HiddenGlobalOffsetY" 2842 The OpenCL grid dispatch 2843 global offset for the Y 2844 dimension is passed in the 2845 kernarg. 2846 2847 "HiddenGlobalOffsetZ" 2848 The OpenCL grid dispatch 2849 global offset for the Z 2850 dimension is passed in the 2851 kernarg. 2852 2853 "HiddenNone" 2854 An argument that is not used 2855 by the kernel. Space needs to 2856 be left for it, but it does 2857 not need to be set up. 2858 2859 "HiddenPrintfBuffer" 2860 A global address space pointer 2861 to the runtime printf buffer 2862 is passed in kernarg. Mutually 2863 exclusive with 2864 "HiddenHostcallBuffer". 2865 2866 "HiddenHostcallBuffer" 2867 A global address space pointer 2868 to the runtime hostcall buffer 2869 is passed in kernarg. Mutually 2870 exclusive with 2871 "HiddenPrintfBuffer". 2872 2873 "HiddenDefaultQueue" 2874 A global address space pointer 2875 to the OpenCL device enqueue 2876 queue that should be used by 2877 the kernel by default is 2878 passed in the kernarg. 2879 2880 "HiddenCompletionAction" 2881 A global address space pointer 2882 to help link enqueued kernels into 2883 the ancestor tree for determining 2884 when the parent kernel has finished. 2885 2886 "HiddenMultiGridSyncArg" 2887 A global address space pointer for 2888 multi-grid synchronization is 2889 passed in the kernarg. 2890 2891 "ValueType" string Unused and deprecated. This should no longer 2892 be emitted, but is accepted for compatibility. 2893 2894 2895 "PointeeAlign" integer Alignment in bytes of pointee 2896 type for pointer type kernel 2897 argument. Must be a power 2898 of 2. Only present if 2899 "ValueKind" is 2900 "DynamicSharedPointer". 2901 "AddrSpaceQual" string Kernel argument address space 2902 qualifier. Only present if 2903 "ValueKind" is "GlobalBuffer" or 2904 "DynamicSharedPointer". Values 2905 are: 2906 2907 - "Private" 2908 - "Global" 2909 - "Constant" 2910 - "Local" 2911 - "Generic" 2912 - "Region" 2913 2914 .. TODO:: 2915 2916 Is GlobalBuffer only Global 2917 or Constant? Is 2918 DynamicSharedPointer always 2919 Local? Can HCC allow Generic? 2920 How can Private or Region 2921 ever happen? 2922 2923 "AccQual" string Kernel argument access 2924 qualifier. Only present if 2925 "ValueKind" is "Image" or 2926 "Pipe". Values 2927 are: 2928 2929 - "ReadOnly" 2930 - "WriteOnly" 2931 - "ReadWrite" 2932 2933 .. TODO:: 2934 2935 Does this apply to 2936 GlobalBuffer? 2937 2938 "ActualAccQual" string The actual memory accesses 2939 performed by the kernel on the 2940 kernel argument. Only present if 2941 "ValueKind" is "GlobalBuffer", 2942 "Image", or "Pipe". This may be 2943 more restrictive than indicated 2944 by "AccQual" to reflect what the 2945 kernel actual does. If not 2946 present then the runtime must 2947 assume what is implied by 2948 "AccQual" and "IsConst". Values 2949 are: 2950 2951 - "ReadOnly" 2952 - "WriteOnly" 2953 - "ReadWrite" 2954 2955 "IsConst" boolean Indicates if the kernel argument 2956 is const qualified. Only present 2957 if "ValueKind" is 2958 "GlobalBuffer". 2959 2960 "IsRestrict" boolean Indicates if the kernel argument 2961 is restrict qualified. Only 2962 present if "ValueKind" is 2963 "GlobalBuffer". 2964 2965 "IsVolatile" boolean Indicates if the kernel argument 2966 is volatile qualified. Only 2967 present if "ValueKind" is 2968 "GlobalBuffer". 2969 2970 "IsPipe" boolean Indicates if the kernel argument 2971 is pipe qualified. Only present 2972 if "ValueKind" is "Pipe". 2973 2974 .. TODO:: 2975 2976 Can GlobalBuffer be pipe 2977 qualified? 2978 2979 ================= ============== ========= ================================ 2980 2981.. 2982 2983 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map 2984 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table 2985 2986 ============================ ============== ========= ===================== 2987 String Key Value Type Required? Description 2988 ============================ ============== ========= ===================== 2989 "KernargSegmentSize" integer Required The size in bytes of 2990 the kernarg segment 2991 that holds the values 2992 of the arguments to 2993 the kernel. 2994 "GroupSegmentFixedSize" integer Required The amount of group 2995 segment memory 2996 required by a 2997 work-group in 2998 bytes. This does not 2999 include any 3000 dynamically allocated 3001 group segment memory 3002 that may be added 3003 when the kernel is 3004 dispatched. 3005 "PrivateSegmentFixedSize" integer Required The amount of fixed 3006 private address space 3007 memory required for a 3008 work-item in 3009 bytes. If the kernel 3010 uses a dynamic call 3011 stack then additional 3012 space must be added 3013 to this value for the 3014 call stack. 3015 "KernargSegmentAlign" integer Required The maximum byte 3016 alignment of 3017 arguments in the 3018 kernarg segment. Must 3019 be a power of 2. 3020 "WavefrontSize" integer Required Wavefront size. Must 3021 be a power of 2. 3022 "NumSGPRs" integer Required Number of scalar 3023 registers used by a 3024 wavefront for 3025 GFX6-GFX10. This 3026 includes the special 3027 SGPRs for VCC, Flat 3028 Scratch (GFX7-GFX10) 3029 and XNACK (for 3030 GFX8-GFX10). It does 3031 not include the 16 3032 SGPR added if a trap 3033 handler is 3034 enabled. It is not 3035 rounded up to the 3036 allocation 3037 granularity. 3038 "NumVGPRs" integer Required Number of vector 3039 registers used by 3040 each work-item for 3041 GFX6-GFX10 3042 "MaxFlatWorkGroupSize" integer Required Maximum flat 3043 work-group size 3044 supported by the 3045 kernel in work-items. 3046 Must be >=1 and 3047 consistent with 3048 ReqdWorkGroupSize if 3049 not 0, 0, 0. 3050 "NumSpilledSGPRs" integer Number of stores from 3051 a scalar register to 3052 a register allocator 3053 created spill 3054 location. 3055 "NumSpilledVGPRs" integer Number of stores from 3056 a vector register to 3057 a register allocator 3058 created spill 3059 location. 3060 ============================ ============== ========= ===================== 3061 3062.. _amdgpu-amdhsa-code-object-metadata-v3: 3063 3064Code Object V3 Metadata 3065+++++++++++++++++++++++ 3066 3067.. warning:: 3068 Code object V3 is not the default code object version emitted by this version 3069 of LLVM. 3070 3071Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note 3072record (see :ref:`amdgpu-note-records-v3-onwards`). 3073 3074The metadata is represented as Message Pack formatted binary data (see 3075[MsgPack]_). The top level is a Message Pack map that includes the 3076keys defined in table 3077:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced 3078tables. 3079 3080Additional information can be added to the maps. To avoid conflicts, 3081any key names should be prefixed by "*vendor-name*." where 3082``vendor-name`` can be the name of the vendor and specific vendor 3083tool that generates the information. The prefix is abbreviated to 3084simply "." when it appears within a map that has been added by the 3085same *vendor-name*. 3086 3087 .. table:: AMDHSA Code Object V3 Metadata Map 3088 :name: amdgpu-amdhsa-code-object-metadata-map-table-v3 3089 3090 ================= ============== ========= ======================================= 3091 String Key Value Type Required? Description 3092 ================= ============== ========= ======================================= 3093 "amdhsa.version" sequence of Required - The first integer is the major 3094 2 integers version. Currently 1. 3095 - The second integer is the minor 3096 version. Currently 0. 3097 "amdhsa.printf" sequence of Each string is encoded information 3098 strings about a printf function call. The 3099 encoded information is organized as 3100 fields separated by colon (':'): 3101 3102 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString`` 3103 3104 where: 3105 3106 ``ID`` 3107 A 32-bit integer as a unique id for 3108 each printf function call 3109 3110 ``N`` 3111 A 32-bit integer equal to the number 3112 of arguments of printf function call 3113 minus 1 3114 3115 ``S[i]`` (where i = 0, 1, ... , N-1) 3116 32-bit integers for the size in bytes 3117 of the i-th FormatString argument of 3118 the printf function call 3119 3120 FormatString 3121 The format string passed to the 3122 printf function call. 3123 "amdhsa.kernels" sequence of Required Sequence of the maps for each 3124 map kernel in the code object. See 3125 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3` 3126 for the definition of the keys included 3127 in that map. 3128 ================= ============== ========= ======================================= 3129 3130.. 3131 3132 .. table:: AMDHSA Code Object V3 Kernel Metadata Map 3133 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3 3134 3135 =================================== ============== ========= ================================ 3136 String Key Value Type Required? Description 3137 =================================== ============== ========= ================================ 3138 ".name" string Required Source name of the kernel. 3139 ".symbol" string Required Name of the kernel 3140 descriptor ELF symbol. 3141 ".language" string Source language of the kernel. 3142 Values include: 3143 3144 - "OpenCL C" 3145 - "OpenCL C++" 3146 - "HCC" 3147 - "HIP" 3148 - "OpenMP" 3149 - "Assembler" 3150 3151 ".language_version" sequence of - The first integer is the major 3152 2 integers version. 3153 - The second integer is the 3154 minor version. 3155 ".args" sequence of Sequence of maps of the 3156 map kernel arguments. See 3157 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3` 3158 for the definition of the keys 3159 included in that map. 3160 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values 3161 3 integers must be >=1 and the dispatch 3162 work-group size X, Y, Z must 3163 correspond to the specified 3164 values. Defaults to 0, 0, 0. 3165 3166 Corresponds to the OpenCL 3167 ``reqd_work_group_size`` 3168 attribute. 3169 ".workgroup_size_hint" sequence of The dispatch work-group size 3170 3 integers X, Y, Z is likely to be the 3171 specified values. 3172 3173 Corresponds to the OpenCL 3174 ``work_group_size_hint`` 3175 attribute. 3176 ".vec_type_hint" string The name of a scalar or vector 3177 type. 3178 3179 Corresponds to the OpenCL 3180 ``vec_type_hint`` attribute. 3181 3182 ".device_enqueue_symbol" string The external symbol name 3183 associated with a kernel. 3184 OpenCL runtime allocates a 3185 global buffer for the symbol 3186 and saves the kernel's address 3187 to it, which is used for 3188 device side enqueueing. Only 3189 available for device side 3190 enqueued kernels. 3191 ".kernarg_segment_size" integer Required The size in bytes of 3192 the kernarg segment 3193 that holds the values 3194 of the arguments to 3195 the kernel. 3196 ".group_segment_fixed_size" integer Required The amount of group 3197 segment memory 3198 required by a 3199 work-group in 3200 bytes. This does not 3201 include any 3202 dynamically allocated 3203 group segment memory 3204 that may be added 3205 when the kernel is 3206 dispatched. 3207 ".private_segment_fixed_size" integer Required The amount of fixed 3208 private address space 3209 memory required for a 3210 work-item in 3211 bytes. If the kernel 3212 uses a dynamic call 3213 stack then additional 3214 space must be added 3215 to this value for the 3216 call stack. 3217 ".kernarg_segment_align" integer Required The maximum byte 3218 alignment of 3219 arguments in the 3220 kernarg segment. Must 3221 be a power of 2. 3222 ".wavefront_size" integer Required Wavefront size. Must 3223 be a power of 2. 3224 ".sgpr_count" integer Required Number of scalar 3225 registers required by a 3226 wavefront for 3227 GFX6-GFX9. A register 3228 is required if it is 3229 used explicitly, or 3230 if a higher numbered 3231 register is used 3232 explicitly. This 3233 includes the special 3234 SGPRs for VCC, Flat 3235 Scratch (GFX7-GFX9) 3236 and XNACK (for 3237 GFX8-GFX9). It does 3238 not include the 16 3239 SGPR added if a trap 3240 handler is 3241 enabled. It is not 3242 rounded up to the 3243 allocation 3244 granularity. 3245 ".vgpr_count" integer Required Number of vector 3246 registers required by 3247 each work-item for 3248 GFX6-GFX9. A register 3249 is required if it is 3250 used explicitly, or 3251 if a higher numbered 3252 register is used 3253 explicitly. 3254 ".agpr_count" integer Required Number of accumulator 3255 registers required by 3256 each work-item for 3257 GFX90A, GFX908. 3258 ".max_flat_workgroup_size" integer Required Maximum flat 3259 work-group size 3260 supported by the 3261 kernel in work-items. 3262 Must be >=1 and 3263 consistent with 3264 ReqdWorkGroupSize if 3265 not 0, 0, 0. 3266 ".sgpr_spill_count" integer Number of stores from 3267 a scalar register to 3268 a register allocator 3269 created spill 3270 location. 3271 ".vgpr_spill_count" integer Number of stores from 3272 a vector register to 3273 a register allocator 3274 created spill 3275 location. 3276 ".kind" string The kind of the kernel 3277 with the following 3278 values: 3279 3280 "normal" 3281 Regular kernels. 3282 3283 "init" 3284 These kernels must be 3285 invoked after loading 3286 the containing code 3287 object and must 3288 complete before any 3289 normal and fini 3290 kernels in the same 3291 code object are 3292 invoked. 3293 3294 "fini" 3295 These kernels must be 3296 invoked before 3297 unloading the 3298 containing code object 3299 and after all init and 3300 normal kernels in the 3301 same code object have 3302 been invoked and 3303 completed. 3304 3305 If omitted, "normal" is 3306 assumed. 3307 =================================== ============== ========= ================================ 3308 3309.. 3310 3311 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map 3312 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3 3313 3314 ====================== ============== ========= ================================ 3315 String Key Value Type Required? Description 3316 ====================== ============== ========= ================================ 3317 ".name" string Kernel argument name. 3318 ".type_name" string Kernel argument type name. 3319 ".size" integer Required Kernel argument size in bytes. 3320 ".offset" integer Required Kernel argument offset in 3321 bytes. The offset must be a 3322 multiple of the alignment 3323 required by the argument. 3324 ".value_kind" string Required Kernel argument kind that 3325 specifies how to set up the 3326 corresponding argument. 3327 Values include: 3328 3329 "by_value" 3330 The argument is copied 3331 directly into the kernarg. 3332 3333 "global_buffer" 3334 A global address space pointer 3335 to the buffer data is passed 3336 in the kernarg. 3337 3338 "dynamic_shared_pointer" 3339 A group address space pointer 3340 to dynamically allocated LDS 3341 is passed in the kernarg. 3342 3343 "sampler" 3344 A global address space 3345 pointer to a S# is passed in 3346 the kernarg. 3347 3348 "image" 3349 A global address space 3350 pointer to a T# is passed in 3351 the kernarg. 3352 3353 "pipe" 3354 A global address space pointer 3355 to an OpenCL pipe is passed in 3356 the kernarg. 3357 3358 "queue" 3359 A global address space pointer 3360 to an OpenCL device enqueue 3361 queue is passed in the 3362 kernarg. 3363 3364 "hidden_global_offset_x" 3365 The OpenCL grid dispatch 3366 global offset for the X 3367 dimension is passed in the 3368 kernarg. 3369 3370 "hidden_global_offset_y" 3371 The OpenCL grid dispatch 3372 global offset for the Y 3373 dimension is passed in the 3374 kernarg. 3375 3376 "hidden_global_offset_z" 3377 The OpenCL grid dispatch 3378 global offset for the Z 3379 dimension is passed in the 3380 kernarg. 3381 3382 "hidden_none" 3383 An argument that is not used 3384 by the kernel. Space needs to 3385 be left for it, but it does 3386 not need to be set up. 3387 3388 "hidden_printf_buffer" 3389 A global address space pointer 3390 to the runtime printf buffer 3391 is passed in kernarg. Mutually 3392 exclusive with 3393 "hidden_hostcall_buffer" 3394 before Code Object V5. 3395 3396 "hidden_hostcall_buffer" 3397 A global address space pointer 3398 to the runtime hostcall buffer 3399 is passed in kernarg. Mutually 3400 exclusive with 3401 "hidden_printf_buffer" 3402 before Code Object V5. 3403 3404 "hidden_default_queue" 3405 A global address space pointer 3406 to the OpenCL device enqueue 3407 queue that should be used by 3408 the kernel by default is 3409 passed in the kernarg. 3410 3411 "hidden_completion_action" 3412 A global address space pointer 3413 to help link enqueued kernels into 3414 the ancestor tree for determining 3415 when the parent kernel has finished. 3416 3417 "hidden_multigrid_sync_arg" 3418 A global address space pointer for 3419 multi-grid synchronization is 3420 passed in the kernarg. 3421 3422 ".value_type" string Unused and deprecated. This should no longer 3423 be emitted, but is accepted for compatibility. 3424 3425 ".pointee_align" integer Alignment in bytes of pointee 3426 type for pointer type kernel 3427 argument. Must be a power 3428 of 2. Only present if 3429 ".value_kind" is 3430 "dynamic_shared_pointer". 3431 ".address_space" string Kernel argument address space 3432 qualifier. Only present if 3433 ".value_kind" is "global_buffer" or 3434 "dynamic_shared_pointer". Values 3435 are: 3436 3437 - "private" 3438 - "global" 3439 - "constant" 3440 - "local" 3441 - "generic" 3442 - "region" 3443 3444 .. TODO:: 3445 3446 Is "global_buffer" only "global" 3447 or "constant"? Is 3448 "dynamic_shared_pointer" always 3449 "local"? Can HCC allow "generic"? 3450 How can "private" or "region" 3451 ever happen? 3452 3453 ".access" string Kernel argument access 3454 qualifier. Only present if 3455 ".value_kind" is "image" or 3456 "pipe". Values 3457 are: 3458 3459 - "read_only" 3460 - "write_only" 3461 - "read_write" 3462 3463 .. TODO:: 3464 3465 Does this apply to 3466 "global_buffer"? 3467 3468 ".actual_access" string The actual memory accesses 3469 performed by the kernel on the 3470 kernel argument. Only present if 3471 ".value_kind" is "global_buffer", 3472 "image", or "pipe". This may be 3473 more restrictive than indicated 3474 by ".access" to reflect what the 3475 kernel actual does. If not 3476 present then the runtime must 3477 assume what is implied by 3478 ".access" and ".is_const" . Values 3479 are: 3480 3481 - "read_only" 3482 - "write_only" 3483 - "read_write" 3484 3485 ".is_const" boolean Indicates if the kernel argument 3486 is const qualified. Only present 3487 if ".value_kind" is 3488 "global_buffer". 3489 3490 ".is_restrict" boolean Indicates if the kernel argument 3491 is restrict qualified. Only 3492 present if ".value_kind" is 3493 "global_buffer". 3494 3495 ".is_volatile" boolean Indicates if the kernel argument 3496 is volatile qualified. Only 3497 present if ".value_kind" is 3498 "global_buffer". 3499 3500 ".is_pipe" boolean Indicates if the kernel argument 3501 is pipe qualified. Only present 3502 if ".value_kind" is "pipe". 3503 3504 .. TODO:: 3505 3506 Can "global_buffer" be pipe 3507 qualified? 3508 3509 ====================== ============== ========= ================================ 3510 3511.. _amdgpu-amdhsa-code-object-metadata-v4: 3512 3513Code Object V4 Metadata 3514+++++++++++++++++++++++ 3515 3516Code object V4 metadata is the same as 3517:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions 3518defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`. 3519 3520 .. table:: AMDHSA Code Object V4 Metadata Map Changes 3521 :name: amdgpu-amdhsa-code-object-metadata-map-table-v4 3522 3523 ================= ============== ========= ======================================= 3524 String Key Value Type Required? Description 3525 ================= ============== ========= ======================================= 3526 "amdhsa.version" sequence of Required - The first integer is the major 3527 2 integers version. Currently 1. 3528 - The second integer is the minor 3529 version. Currently 1. 3530 "amdhsa.target" string Required The target name of the code using the syntax: 3531 3532 .. code:: 3533 3534 <target-triple> [ "-" <target-id> ] 3535 3536 A canonical target ID must be 3537 used. See :ref:`amdgpu-target-triples` 3538 and :ref:`amdgpu-target-id`. 3539 ================= ============== ========= ======================================= 3540 3541.. _amdgpu-amdhsa-code-object-metadata-v5: 3542 3543Code Object V5 Metadata 3544+++++++++++++++++++++++ 3545 3546.. warning:: 3547 Code object V5 is not the default code object version emitted by this version 3548 of LLVM. 3549 3550 3551Code object V5 metadata is the same as 3552:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table 3553:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5` and table 3554:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`. 3555 3556 .. table:: AMDHSA Code Object V5 Metadata Map Changes 3557 :name: amdgpu-amdhsa-code-object-metadata-map-table-v5 3558 3559 ================= ============== ========= ======================================= 3560 String Key Value Type Required? Description 3561 ================= ============== ========= ======================================= 3562 "amdhsa.version" sequence of Required - The first integer is the major 3563 2 integers version. Currently 1. 3564 - The second integer is the minor 3565 version. Currently 2. 3566 ================= ============== ========= ======================================= 3567 3568.. 3569 3570 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes 3571 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5 3572 3573 ====================== ============== ========= ================================ 3574 String Key Value Type Required? Description 3575 ====================== ============== ========= ================================ 3576 ".value_kind" string Required Kernel argument kind that 3577 specifies how to set up the 3578 corresponding argument. 3579 Values include: 3580 the same as code object V3 metadata 3581 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`) 3582 with the following additions: 3583 3584 "hidden_block_count_x" 3585 The grid dispatch work-group count for the X dimension 3586 is passed in the kernarg. Some languages, such as OpenCL, 3587 support a last work-group in each dimension being partial. 3588 This count only includes the non-partial work-group count. 3589 This is not the same as the value in the AQL dispatch packet, 3590 which has the grid size in work-items. 3591 3592 "hidden_block_count_y" 3593 The grid dispatch work-group count for the Y dimension 3594 is passed in the kernarg. Some languages, such as OpenCL, 3595 support a last work-group in each dimension being partial. 3596 This count only includes the non-partial work-group count. 3597 This is not the same as the value in the AQL dispatch packet, 3598 which has the grid size in work-items. If the grid dimensionality 3599 is 1, then must be 1. 3600 3601 "hidden_block_count_z" 3602 The grid dispatch work-group count for the Z dimension 3603 is passed in the kernarg. Some languages, such as OpenCL, 3604 support a last work-group in each dimension being partial. 3605 This count only includes the non-partial work-group count. 3606 This is not the same as the value in the AQL dispatch packet, 3607 which has the grid size in work-items. If the grid dimensionality 3608 is 1 or 2, then must be 1. 3609 3610 "hidden_group_size_x" 3611 The grid dispatch work-group size for the X dimension is 3612 passed in the kernarg. This size only applies to the 3613 non-partial work-groups. This is the same value as the AQL 3614 dispatch packet work-group size. 3615 3616 "hidden_group_size_y" 3617 The grid dispatch work-group size for the Y dimension is 3618 passed in the kernarg. This size only applies to the 3619 non-partial work-groups. This is the same value as the AQL 3620 dispatch packet work-group size. If the grid dimensionality 3621 is 1, then must be 1. 3622 3623 "hidden_group_size_z" 3624 The grid dispatch work-group size for the Z dimension is 3625 passed in the kernarg. This size only applies to the 3626 non-partial work-groups. This is the same value as the AQL 3627 dispatch packet work-group size. If the grid dimensionality 3628 is 1 or 2, then must be 1. 3629 3630 "hidden_remainder_x" 3631 The grid dispatch work group size of the partial work group 3632 of the X dimension, if it exists. Must be zero if a partial 3633 work group does not exist in the X dimension. 3634 3635 "hidden_remainder_y" 3636 The grid dispatch work group size of the partial work group 3637 of the Y dimension, if it exists. Must be zero if a partial 3638 work group does not exist in the Y dimension. 3639 3640 "hidden_remainder_z" 3641 The grid dispatch work group size of the partial work group 3642 of the Z dimension, if it exists. Must be zero if a partial 3643 work group does not exist in the Z dimension. 3644 3645 "hidden_grid_dims" 3646 The grid dispatch dimensionality. This is the same value 3647 as the AQL dispatch packet dimensionality. Must be a value 3648 between 1 and 3. 3649 3650 "hidden_heap_v1" 3651 A global address space pointer to an initialized memory 3652 buffer that conforms to the requirements of the malloc/free 3653 device library V1 version implementation. 3654 3655 "hidden_private_base" 3656 The high 32 bits of the flat addressing private aperture base. 3657 Only used by GFX8 to allow conversion between private segment 3658 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3659 3660 "hidden_shared_base" 3661 The high 32 bits of the flat addressing shared aperture base. 3662 Only used by GFX8 to allow conversion between shared segment 3663 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 3664 3665 "hidden_queue_ptr" 3666 A global memory address space pointer to the ROCm runtime 3667 ``struct amd_queue_t`` structure for the HSA queue of the 3668 associated dispatch AQL packet. It is only required for pre-GFX9 3669 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`). 3670 3671 ====================== ============== ========= ================================ 3672 3673.. 3674 3675Kernel Dispatch 3676~~~~~~~~~~~~~~~ 3677 3678The HSA architected queuing language (AQL) defines a user space memory interface 3679that can be used to control the dispatch of kernels, in an agent independent 3680way. An agent can have zero or more AQL queues created for it using an HSA 3681compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which 3682are 64 bytes) can be placed. See the *HSA Platform System Architecture 3683Specification* [HSA]_ for the AQL queue mechanics and packet layouts. 3684 3685The packet processor of a kernel agent is responsible for detecting and 3686dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the 3687packet processor is implemented by the hardware command processor (CP), 3688asynchronous dispatch controller (ADC) and shader processor input controller 3689(SPI). 3690 3691An HSA compatible runtime can be used to allocate an AQL queue object. It uses 3692the kernel mode driver to initialize and register the AQL queue with CP. 3693 3694To dispatch a kernel the following actions are performed. This can occur in the 3695CPU host program, or from an HSA kernel executing on a GPU. 3696 36971. A pointer to an AQL queue for the kernel agent on which the kernel is to be 3698 executed is obtained. 36992. A pointer to the kernel descriptor (see 3700 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained. 3701 It must be for a kernel that is contained in a code object that was loaded 3702 by an HSA compatible runtime on the kernel agent with which the AQL queue is 3703 associated. 37043. Space is allocated for the kernel arguments using the HSA compatible runtime 3705 allocator for a memory region with the kernarg property for the kernel agent 3706 that will execute the kernel. It must be at least 16-byte aligned. 37074. Kernel argument values are assigned to the kernel argument memory 3708 allocation. The layout is defined in the *HSA Programmer's Language 3709 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the 3710 kernel argument memory in the same way constant memory is accessed. (Note 3711 that the HSA specification allows an implementation to copy the kernel 3712 argument contents to another location that is accessed by the kernel.) 37135. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible 3714 runtime api uses 64-bit atomic operations to reserve space in the AQL queue 3715 for the packet. The packet must be set up, and the final write must use an 3716 atomic store release to set the packet kind to ensure the packet contents are 3717 visible to the kernel agent. AQL defines a doorbell signal mechanism to 3718 notify the kernel agent that the AQL queue has been updated. These rules, and 3719 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA 3720 System Architecture Specification* [HSA]_. 37216. A kernel dispatch packet includes information about the actual dispatch, 3722 such as grid and work-group size, together with information from the code 3723 object about the kernel, such as segment sizes. The HSA compatible runtime 3724 queries on the kernel symbol can be used to obtain the code object values 3725 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`. 37267. CP executes micro-code and is responsible for detecting and setting up the 3727 GPU to execute the wavefronts of a kernel dispatch. 37288. CP ensures that when the a wavefront starts executing the kernel machine 3729 code, the scalar general purpose registers (SGPR) and vector general purpose 3730 registers (VGPR) are set up as required by the machine code. The required 3731 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial 3732 register state is defined in 3733 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 37349. The prolog of the kernel machine code (see 3735 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary 3736 before continuing executing the machine code that corresponds to the kernel. 373710. When the kernel dispatch has completed execution, CP signals the completion 3738 signal specified in the kernel dispatch packet if not 0. 3739 3740.. _amdgpu-amdhsa-memory-spaces: 3741 3742Memory Spaces 3743~~~~~~~~~~~~~ 3744 3745The memory space properties are: 3746 3747 .. table:: AMDHSA Memory Spaces 3748 :name: amdgpu-amdhsa-memory-spaces-table 3749 3750 ================= =========== ======== ======= ================== 3751 Memory Space Name HSA Segment Hardware Address NULL Value 3752 Name Name Size 3753 ================= =========== ======== ======= ================== 3754 Private private scratch 32 0x00000000 3755 Local group LDS 32 0xFFFFFFFF 3756 Global global global 64 0x0000000000000000 3757 Constant constant *same as 64 0x0000000000000000 3758 global* 3759 Generic flat flat 64 0x0000000000000000 3760 Region N/A GDS 32 *not implemented 3761 for AMDHSA* 3762 ================= =========== ======== ======= ================== 3763 3764The global and constant memory spaces both use global virtual addresses, which 3765are the same virtual address space used by the CPU. However, some virtual 3766addresses may only be accessible to the CPU, some only accessible by the GPU, 3767and some by both. 3768 3769Using the constant memory space indicates that the data will not change during 3770the execution of the kernel. This allows scalar read instructions to be 3771used. The vector and scalar L1 caches are invalidated of volatile data before 3772each kernel dispatch execution to allow constant memory to change values between 3773kernel dispatches. 3774 3775The local memory space uses the hardware Local Data Store (LDS) which is 3776automatically allocated when the hardware creates work-groups of wavefronts, and 3777freed when all the wavefronts of a work-group have terminated. The data store 3778(DS) instructions can be used to access it. 3779 3780The private memory space uses the hardware scratch memory support. If the kernel 3781uses scratch, then the hardware allocates memory that is accessed using 3782wavefront lane dword (4 byte) interleaving. The mapping used from private 3783address to physical address is: 3784 3785 ``wavefront-scratch-base + 3786 (private-address * wavefront-size * 4) + 3787 (wavefront-lane-id * 4)`` 3788 3789There are different ways that the wavefront scratch base address is determined 3790by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This 3791memory can be accessed in an interleaved manner using buffer instruction with 3792the scratch buffer descriptor and per wavefront scratch offset, by the scratch 3793instructions, or by flat instructions. If each lane of a wavefront accesses the 3794same private address, the interleaving results in adjacent dwords being accessed 3795and hence requires fewer cache lines to be fetched. Multi-dword access is not 3796supported except by flat and scratch instructions in GFX9-GFX10. 3797 3798The generic address space uses the hardware flat address support available in 3799GFX7-GFX10. This uses two fixed ranges of virtual addresses (the private and 3800local apertures), that are outside the range of addressible global memory, to 3801map from a flat address to a private or local address. 3802 3803FLAT instructions can take a flat address and access global, private (scratch) 3804and group (LDS) memory depending on if the address is within one of the 3805aperture ranges. Flat access to scratch requires hardware aperture setup and 3806setup in the kernel prologue (see 3807:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires 3808hardware aperture setup and M0 (GFX7-GFX8) register setup (see 3809:ref:`amdgpu-amdhsa-kernel-prolog-m0`). 3810 3811To convert between a segment address and a flat address the base address of the 3812apertures address can be used. For GFX7-GFX8 these are available in the 3813:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with 3814Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For 3815GFX9-GFX10 the aperture base addresses are directly available as inline constant 3816registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64 bit 3817address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^32 3818which makes it easier to convert from flat to segment or segment to flat. 3819 3820Image and Samplers 3821~~~~~~~~~~~~~~~~~~ 3822 3823Image and sample handles created by an HSA compatible runtime (see 3824:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S# 3825object respectively. In order to support the HSA ``query_sampler`` operations 3826two extra dwords are used to store the HSA BRIG enumeration values for the 3827queries that are not trivially deducible from the S# representation. 3828 3829HSA Signals 3830~~~~~~~~~~~ 3831 3832HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`) 3833are 64-bit addresses of a structure allocated in memory accessible from both the 3834CPU and GPU. The structure is defined by the runtime and subject to change 3835between releases. For example, see [AMD-ROCm-github]_. 3836 3837.. _amdgpu-amdhsa-hsa-aql-queue: 3838 3839HSA AQL Queue 3840~~~~~~~~~~~~~ 3841 3842The HSA AQL queue structure is defined by an HSA compatible runtime (see 3843:ref:`amdgpu-os`) and subject to change between releases. For example, see 3844[AMD-ROCm-github]_. For some processors it contains fields needed to implement 3845certain language features such as the flat address aperture bases. It also 3846contains fields used by CP such as managing the allocation of scratch memory. 3847 3848.. _amdgpu-amdhsa-kernel-descriptor: 3849 3850Kernel Descriptor 3851~~~~~~~~~~~~~~~~~ 3852 3853A kernel descriptor consists of the information needed by CP to initiate the 3854execution of a kernel, including the entry point address of the machine code 3855that implements the kernel. 3856 3857Code Object V3 Kernel Descriptor 3858++++++++++++++++++++++++++++++++ 3859 3860CP microcode requires the Kernel descriptor to be allocated on 64-byte 3861alignment. 3862 3863The fields used by CP for code objects before V3 also match those specified in 3864:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 3865 3866 .. table:: Code Object V3 Kernel Descriptor 3867 :name: amdgpu-amdhsa-kernel-descriptor-v3-table 3868 3869 ======= ======= =============================== ============================ 3870 Bits Size Field Name Description 3871 ======= ======= =============================== ============================ 3872 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local 3873 address space memory 3874 required for a work-group 3875 in bytes. This does not 3876 include any dynamically 3877 allocated local address 3878 space memory that may be 3879 added when the kernel is 3880 dispatched. 3881 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed 3882 private address space 3883 memory required for a 3884 work-item in bytes. 3885 Additional space may need to 3886 be added to this value if 3887 the call stack has 3888 non-inlined function calls. 3889 95:64 4 bytes KERNARG_SIZE The size of the kernarg 3890 memory pointed to by the 3891 AQL dispatch packet. The 3892 kernarg memory is used to 3893 pass arguments to the 3894 kernel. 3895 3896 * If the kernarg pointer in 3897 the dispatch packet is NULL 3898 then there are no kernel 3899 arguments. 3900 * If the kernarg pointer in 3901 the dispatch packet is 3902 not NULL and this value 3903 is 0 then the kernarg 3904 memory size is 3905 unspecified. 3906 * If the kernarg pointer in 3907 the dispatch packet is 3908 not NULL and this value 3909 is not 0 then the value 3910 specifies the kernarg 3911 memory size in bytes. It 3912 is recommended to provide 3913 a value as it may be used 3914 by CP to optimize making 3915 the kernarg memory 3916 visible to the kernel 3917 code. 3918 3919 127:96 4 bytes Reserved, must be 0. 3920 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly 3921 negative) from base 3922 address of kernel 3923 descriptor to kernel's 3924 entry point instruction 3925 which must be 256 byte 3926 aligned. 3927 351:272 20 Reserved, must be 0. 3928 bytes 3929 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX9 3930 Reserved, must be 0. 3931 GFX90A, GFX940 3932 Compute Shader (CS) 3933 program settings used by 3934 CP to set up 3935 ``COMPUTE_PGM_RSRC3`` 3936 configuration 3937 register. See 3938 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 3939 GFX10 3940 Compute Shader (CS) 3941 program settings used by 3942 CP to set up 3943 ``COMPUTE_PGM_RSRC3`` 3944 configuration 3945 register. See 3946 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 3947 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS) 3948 program settings used by 3949 CP to set up 3950 ``COMPUTE_PGM_RSRC1`` 3951 configuration 3952 register. See 3953 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 3954 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS) 3955 program settings used by 3956 CP to set up 3957 ``COMPUTE_PGM_RSRC2`` 3958 configuration 3959 register. See 3960 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 3961 458:448 7 bits *See separate bits below.* Enable the setup of the 3962 SGPR user data registers 3963 (see 3964 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 3965 3966 The total number of SGPR 3967 user data registers 3968 requested must not exceed 3969 16 and match value in 3970 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``. 3971 Any requests beyond 16 3972 will be ignored. 3973 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties* 3974 _BUFFER column of 3975 :ref:`amdgpu-processor-table` 3976 specifies *Architected flat 3977 scratch* then not supported 3978 and must be 0, 3979 >449 1 bit ENABLE_SGPR_DISPATCH_PTR 3980 >450 1 bit ENABLE_SGPR_QUEUE_PTR 3981 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR 3982 >452 1 bit ENABLE_SGPR_DISPATCH_ID 3983 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties* 3984 column of 3985 :ref:`amdgpu-processor-table` 3986 specifies *Architected flat 3987 scratch* then not supported 3988 and must be 0, 3989 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT 3990 _SIZE 3991 457:455 3 bits Reserved, must be 0. 3992 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX9 3993 Reserved, must be 0. 3994 GFX10 3995 - If 0 execute in 3996 wavefront size 64 mode. 3997 - If 1 execute in 3998 native wavefront size 3999 32 mode. 4000 463:459 1 bit Reserved, must be 0. 4001 464 1 bit RESERVED_464 Deprecated, must be 0. 4002 467:465 3 bits Reserved, must be 0. 4003 468 1 bit RESERVED_468 Deprecated, must be 0. 4004 469:471 3 bits Reserved, must be 0. 4005 511:472 5 bytes Reserved, must be 0. 4006 512 **Total size 64 bytes.** 4007 ======= ==================================================================== 4008 4009.. 4010 4011 .. table:: compute_pgm_rsrc1 for GFX6-GFX10 4012 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table 4013 4014 ======= ======= =============================== =========================================================================== 4015 Bits Size Field Name Description 4016 ======= ======= =============================== =========================================================================== 4017 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register 4018 blocks used by each work-item; 4019 granularity is device 4020 specific: 4021 4022 GFX6-GFX9 4023 - vgprs_used 0..256 4024 - max(0, ceil(vgprs_used / 4) - 1) 4025 GFX90A, GFX940 4026 - vgprs_used 0..512 4027 - vgprs_used = align(arch_vgprs, 4) 4028 + acc_vgprs 4029 - max(0, ceil(vgprs_used / 8) - 1) 4030 GFX10 (wavefront size 64) 4031 - max_vgpr 1..256 4032 - max(0, ceil(vgprs_used / 4) - 1) 4033 GFX10 (wavefront size 32) 4034 - max_vgpr 1..256 4035 - max(0, ceil(vgprs_used / 8) - 1) 4036 4037 Where vgprs_used is defined 4038 as the highest VGPR number 4039 explicitly referenced plus 4040 one. 4041 4042 Used by CP to set up 4043 ``COMPUTE_PGM_RSRC1.VGPRS``. 4044 4045 The 4046 :ref:`amdgpu-assembler` 4047 calculates this 4048 automatically for the 4049 selected processor from 4050 values provided to the 4051 `.amdhsa_kernel` directive 4052 by the 4053 `.amdhsa_next_free_vgpr` 4054 nested directive (see 4055 :ref:`amdhsa-kernel-directives-table`). 4056 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register 4057 blocks used by a wavefront; 4058 granularity is device 4059 specific: 4060 4061 GFX6-GFX8 4062 - sgprs_used 0..112 4063 - max(0, ceil(sgprs_used / 8) - 1) 4064 GFX9 4065 - sgprs_used 0..112 4066 - 2 * max(0, ceil(sgprs_used / 16) - 1) 4067 GFX10 4068 Reserved, must be 0. 4069 (128 SGPRs always 4070 allocated.) 4071 4072 Where sgprs_used is 4073 defined as the highest 4074 SGPR number explicitly 4075 referenced plus one, plus 4076 a target specific number 4077 of additional special 4078 SGPRs for VCC, 4079 FLAT_SCRATCH (GFX7+) and 4080 XNACK_MASK (GFX8+), and 4081 any additional 4082 target specific 4083 limitations. It does not 4084 include the 16 SGPRs added 4085 if a trap handler is 4086 enabled. 4087 4088 The target specific 4089 limitations and special 4090 SGPR layout are defined in 4091 the hardware 4092 documentation, which can 4093 be found in the 4094 :ref:`amdgpu-processors` 4095 table. 4096 4097 Used by CP to set up 4098 ``COMPUTE_PGM_RSRC1.SGPRS``. 4099 4100 The 4101 :ref:`amdgpu-assembler` 4102 calculates this 4103 automatically for the 4104 selected processor from 4105 values provided to the 4106 `.amdhsa_kernel` directive 4107 by the 4108 `.amdhsa_next_free_sgpr` 4109 and `.amdhsa_reserve_*` 4110 nested directives (see 4111 :ref:`amdhsa-kernel-directives-table`). 4112 11:10 2 bits PRIORITY Must be 0. 4113 4114 Start executing wavefront 4115 at the specified priority. 4116 4117 CP is responsible for 4118 filling in 4119 ``COMPUTE_PGM_RSRC1.PRIORITY``. 4120 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution 4121 with specified rounding 4122 mode for single (32 4123 bit) floating point 4124 precision floating point 4125 operations. 4126 4127 Floating point rounding 4128 mode values are defined in 4129 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4130 4131 Used by CP to set up 4132 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4133 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution 4134 with specified rounding 4135 denorm mode for half/double (16 4136 and 64-bit) floating point 4137 precision floating point 4138 operations. 4139 4140 Floating point rounding 4141 mode values are defined in 4142 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 4143 4144 Used by CP to set up 4145 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4146 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution 4147 with specified denorm mode 4148 for single (32 4149 bit) floating point 4150 precision floating point 4151 operations. 4152 4153 Floating point denorm mode 4154 values are defined in 4155 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4156 4157 Used by CP to set up 4158 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4159 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution 4160 with specified denorm mode 4161 for half/double (16 4162 and 64-bit) floating point 4163 precision floating point 4164 operations. 4165 4166 Floating point denorm mode 4167 values are defined in 4168 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 4169 4170 Used by CP to set up 4171 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``. 4172 20 1 bit PRIV Must be 0. 4173 4174 Start executing wavefront 4175 in privilege trap handler 4176 mode. 4177 4178 CP is responsible for 4179 filling in 4180 ``COMPUTE_PGM_RSRC1.PRIV``. 4181 21 1 bit ENABLE_DX10_CLAMP Wavefront starts execution 4182 with DX10 clamp mode 4183 enabled. Used by the vector 4184 ALU to force DX10 style 4185 treatment of NaN's (when 4186 set, clamp NaN to zero, 4187 otherwise pass NaN 4188 through). 4189 4190 Used by CP to set up 4191 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``. 4192 22 1 bit DEBUG_MODE Must be 0. 4193 4194 Start executing wavefront 4195 in single step mode. 4196 4197 CP is responsible for 4198 filling in 4199 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``. 4200 23 1 bit ENABLE_IEEE_MODE Wavefront starts execution 4201 with IEEE mode 4202 enabled. Floating point 4203 opcodes that support 4204 exception flag gathering 4205 will quiet and propagate 4206 signaling-NaN inputs per 4207 IEEE 754-2008. Min_dx10 and 4208 max_dx10 become IEEE 4209 754-2008 compliant due to 4210 signaling-NaN propagation 4211 and quieting. 4212 4213 Used by CP to set up 4214 ``COMPUTE_PGM_RSRC1.IEEE_MODE``. 4215 24 1 bit BULKY Must be 0. 4216 4217 Only one work-group allowed 4218 to execute on a compute 4219 unit. 4220 4221 CP is responsible for 4222 filling in 4223 ``COMPUTE_PGM_RSRC1.BULKY``. 4224 25 1 bit CDBG_USER Must be 0. 4225 4226 Flag that can be used to 4227 control debugging code. 4228 4229 CP is responsible for 4230 filling in 4231 ``COMPUTE_PGM_RSRC1.CDBG_USER``. 4232 26 1 bit FP16_OVFL GFX6-GFX8 4233 Reserved, must be 0. 4234 GFX9-GFX10 4235 Wavefront starts execution 4236 with specified fp16 overflow 4237 mode. 4238 4239 - If 0, fp16 overflow generates 4240 +/-INF values. 4241 - If 1, fp16 overflow that is the 4242 result of an +/-INF input value 4243 or divide by 0 produces a +/-INF, 4244 otherwise clamps computed 4245 overflow to +/-MAX_FP16 as 4246 appropriate. 4247 4248 Used by CP to set up 4249 ``COMPUTE_PGM_RSRC1.FP16_OVFL``. 4250 28:27 2 bits Reserved, must be 0. 4251 29 1 bit WGP_MODE GFX6-GFX9 4252 Reserved, must be 0. 4253 GFX10 4254 - If 0 execute work-groups in 4255 CU wavefront execution mode. 4256 - If 1 execute work-groups on 4257 in WGP wavefront execution mode. 4258 4259 See :ref:`amdgpu-amdhsa-memory-model`. 4260 4261 Used by CP to set up 4262 ``COMPUTE_PGM_RSRC1.WGP_MODE``. 4263 30 1 bit MEM_ORDERED GFX6-GFX9 4264 Reserved, must be 0. 4265 GFX10 4266 Controls the behavior of the 4267 s_waitcnt's vmcnt and vscnt 4268 counters. 4269 4270 - If 0 vmcnt reports completion 4271 of load and atomic with return 4272 out of order with sample 4273 instructions, and the vscnt 4274 reports the completion of 4275 store and atomic without 4276 return in order. 4277 - If 1 vmcnt reports completion 4278 of load, atomic with return 4279 and sample instructions in 4280 order, and the vscnt reports 4281 the completion of store and 4282 atomic without return in order. 4283 4284 Used by CP to set up 4285 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``. 4286 31 1 bit FWD_PROGRESS GFX6-GFX9 4287 Reserved, must be 0. 4288 GFX10 4289 - If 0 execute SIMD wavefronts 4290 using oldest first policy. 4291 - If 1 execute SIMD wavefronts to 4292 ensure wavefronts will make some 4293 forward progress. 4294 4295 Used by CP to set up 4296 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``. 4297 32 **Total size 4 bytes** 4298 ======= =================================================================================================================== 4299 4300.. 4301 4302 .. table:: compute_pgm_rsrc2 for GFX6-GFX10 4303 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table 4304 4305 ======= ======= =============================== =========================================================================== 4306 Bits Size Field Name Description 4307 ======= ======= =============================== =========================================================================== 4308 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the 4309 private segment. 4310 * If the *Target Properties* 4311 column of 4312 :ref:`amdgpu-processor-table` 4313 does not specify 4314 *Architected flat 4315 scratch* then enable the 4316 setup of the SGPR 4317 wavefront scratch offset 4318 system register (see 4319 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4320 * If the *Target Properties* 4321 column of 4322 :ref:`amdgpu-processor-table` 4323 specifies *Architected 4324 flat scratch* then enable 4325 the setup of the 4326 FLAT_SCRATCH register 4327 pair (see 4328 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4329 4330 Used by CP to set up 4331 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``. 4332 5:1 5 bits USER_SGPR_COUNT The total number of SGPR 4333 user data 4334 registers requested. This 4335 number must be greater than 4336 or equal to the number of user 4337 data registers enabled. 4338 4339 Used by CP to set up 4340 ``COMPUTE_PGM_RSRC2.USER_SGPR``. 4341 6 1 bit ENABLE_TRAP_HANDLER Must be 0. 4342 4343 This bit represents 4344 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``, 4345 which is set by the CP if 4346 the runtime has installed a 4347 trap handler. 4348 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the 4349 system SGPR register for 4350 the work-group id in the X 4351 dimension (see 4352 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4353 4354 Used by CP to set up 4355 ``COMPUTE_PGM_RSRC2.TGID_X_EN``. 4356 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the 4357 system SGPR register for 4358 the work-group id in the Y 4359 dimension (see 4360 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4361 4362 Used by CP to set up 4363 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``. 4364 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the 4365 system SGPR register for 4366 the work-group id in the Z 4367 dimension (see 4368 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4369 4370 Used by CP to set up 4371 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``. 4372 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the 4373 system SGPR register for 4374 work-group information (see 4375 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4376 4377 Used by CP to set up 4378 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``. 4379 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the 4380 VGPR system registers used 4381 for the work-item ID. 4382 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table` 4383 defines the values. 4384 4385 Used by CP to set up 4386 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``. 4387 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0. 4388 4389 Wavefront starts execution 4390 with address watch 4391 exceptions enabled which 4392 are generated when L1 has 4393 witnessed a thread access 4394 an *address of 4395 interest*. 4396 4397 CP is responsible for 4398 filling in the address 4399 watch bit in 4400 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4401 according to what the 4402 runtime requests. 4403 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0. 4404 4405 Wavefront starts execution 4406 with memory violation 4407 exceptions exceptions 4408 enabled which are generated 4409 when a memory violation has 4410 occurred for this wavefront from 4411 L1 or LDS 4412 (write-to-read-only-memory, 4413 mis-aligned atomic, LDS 4414 address out of range, 4415 illegal address, etc.). 4416 4417 CP sets the memory 4418 violation bit in 4419 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB`` 4420 according to what the 4421 runtime requests. 4422 23:15 9 bits GRANULATED_LDS_SIZE Must be 0. 4423 4424 CP uses the rounded value 4425 from the dispatch packet, 4426 not this value, as the 4427 dispatch may contain 4428 dynamically allocated group 4429 segment memory. CP writes 4430 directly to 4431 ``COMPUTE_PGM_RSRC2.LDS_SIZE``. 4432 4433 Amount of group segment 4434 (LDS) to allocate for each 4435 work-group. Granularity is 4436 device specific: 4437 4438 GFX6 4439 roundup(lds-size / (64 * 4)) 4440 GFX7-GFX10 4441 roundup(lds-size / (128 * 4)) 4442 4443 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution 4444 _INVALID_OPERATION with specified exceptions 4445 enabled. 4446 4447 Used by CP to set up 4448 ``COMPUTE_PGM_RSRC2.EXCP_EN`` 4449 (set from bits 0..6). 4450 4451 IEEE 754 FP Invalid 4452 Operation 4453 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more 4454 _SOURCE input operands is a 4455 denormal number 4456 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by 4457 _DIVISION_BY_ZERO Zero 4458 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow 4459 _OVERFLOW 4460 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow 4461 _UNDERFLOW 4462 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact 4463 _INEXACT 4464 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero 4465 _ZERO (rcp_iflag_f32 instruction 4466 only) 4467 31 1 bit Reserved, must be 0. 4468 32 **Total size 4 bytes.** 4469 ======= =================================================================================================================== 4470 4471.. 4472 4473 .. table:: compute_pgm_rsrc3 for GFX90A, GFX940 4474 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table 4475 4476 ======= ======= =============================== =========================================================================== 4477 Bits Size Field Name Description 4478 ======= ======= =============================== =========================================================================== 4479 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4. 4480 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ..., 4481 63 - accum-offset = 256. 4482 6:15 10 Reserved, must be 0. 4483 bits 4484 16 1 bit TG_SPLIT - If 0 the waves of a work-group are 4485 launched in the same CU. 4486 - If 1 the waves of a work-group can be 4487 launched in different CUs. The waves 4488 cannot use S_BARRIER or LDS. 4489 17:31 15 Reserved, must be 0. 4490 bits 4491 32 **Total size 4 bytes.** 4492 ======= =================================================================================================================== 4493 4494.. 4495 4496 .. table:: compute_pgm_rsrc3 for GFX10 4497 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table 4498 4499 ======= ======= =============================== =========================================================================== 4500 Bits Size Field Name Description 4501 ======= ======= =============================== =========================================================================== 4502 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For 4503 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity 4504 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does 4505 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0. 4506 31:4 28 Reserved, must be 0. 4507 bits 4508 32 **Total size 4 bytes.** 4509 ======= =================================================================================================================== 4510 4511.. 4512 4513 .. table:: Floating Point Rounding Mode Enumeration Values 4514 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table 4515 4516 ====================================== ===== ============================== 4517 Enumeration Name Value Description 4518 ====================================== ===== ============================== 4519 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even 4520 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity 4521 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity 4522 FLOAT_ROUND_MODE_ZERO 3 Round Toward 0 4523 ====================================== ===== ============================== 4524 4525.. 4526 4527 .. table:: Floating Point Denorm Mode Enumeration Values 4528 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table 4529 4530 ====================================== ===== ============================== 4531 Enumeration Name Value Description 4532 ====================================== ===== ============================== 4533 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination 4534 Denorms 4535 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms 4536 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms 4537 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush 4538 ====================================== ===== ============================== 4539 4540.. 4541 4542 .. table:: System VGPR Work-Item ID Enumeration Values 4543 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table 4544 4545 ======================================== ===== ============================ 4546 Enumeration Name Value Description 4547 ======================================== ===== ============================ 4548 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension 4549 ID. 4550 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y 4551 dimensions ID. 4552 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z 4553 dimensions ID. 4554 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined. 4555 ======================================== ===== ============================ 4556 4557.. _amdgpu-amdhsa-initial-kernel-execution-state: 4558 4559Initial Kernel Execution State 4560~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 4561 4562This section defines the register state that will be set up by the packet 4563processor prior to the start of execution of every wavefront. This is limited by 4564the constraints of the hardware controllers of CP/ADC/SPI. 4565 4566The order of the SGPR registers is defined, but the compiler can specify which 4567ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit 4568fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4569for enabled registers are dense starting at SGPR0: the first enabled register is 4570SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have 4571an SGPR number. 4572 4573The initial SGPRs comprise up to 16 User SRGPs that are set by CP and apply to 4574all wavefronts of the grid. It is possible to specify more than 16 User SGPRs 4575using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are 4576actually initialized. These are then immediately followed by the System SGPRs 4577that are set up by ADC/SPI and can have different values for each wavefront of 4578the grid dispatch. 4579 4580SGPR register initial state is defined in 4581:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 4582 4583 .. table:: SGPR Register Set Up Order 4584 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table 4585 4586 ========== ========================== ====== ============================== 4587 SGPR Order Name Number Description 4588 (kernel descriptor enable of 4589 field) SGPRs 4590 ========== ========================== ====== ============================== 4591 First Private Segment Buffer 4 See 4592 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4593 _segment_buffer) 4594 then Dispatch Ptr 2 64-bit address of AQL dispatch 4595 (enable_sgpr_dispatch_ptr) packet for kernel dispatch 4596 actually executing. 4597 then Queue Ptr 2 64-bit address of amd_queue_t 4598 (enable_sgpr_queue_ptr) object for AQL queue on which 4599 the dispatch packet was 4600 queued. 4601 then Kernarg Segment Ptr 2 64-bit address of Kernarg 4602 (enable_sgpr_kernarg segment. This is directly 4603 _segment_ptr) copied from the 4604 kernarg_address in the kernel 4605 dispatch packet. 4606 4607 Having CP load it once avoids 4608 loading it at the beginning of 4609 every wavefront. 4610 then Dispatch Id 2 64-bit Dispatch ID of the 4611 (enable_sgpr_dispatch_id) dispatch packet being 4612 executed. 4613 then Flat Scratch Init 2 See 4614 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4615 _init) 4616 then Private Segment Size 1 The 32-bit byte size of a 4617 (enable_sgpr_private single work-item's memory 4618 _segment_size) allocation. This is the 4619 value from the kernel 4620 dispatch packet Private 4621 Segment Byte Size rounded up 4622 by CP to a multiple of 4623 DWORD. 4624 4625 Having CP load it once avoids 4626 loading it at the beginning of 4627 every wavefront. 4628 4629 This is not used for 4630 GFX7-GFX8 since it is the same 4631 value as the second SGPR of 4632 Flat Scratch Init. However, it 4633 may be needed for GFX9-GFX10 which 4634 changes the meaning of the 4635 Flat Scratch Init value. 4636 then Work-Group Id X 1 32-bit work-group id in X 4637 (enable_sgpr_workgroup_id dimension of grid for 4638 _X) wavefront. 4639 then Work-Group Id Y 1 32-bit work-group id in Y 4640 (enable_sgpr_workgroup_id dimension of grid for 4641 _Y) wavefront. 4642 then Work-Group Id Z 1 32-bit work-group id in Z 4643 (enable_sgpr_workgroup_id dimension of grid for 4644 _Z) wavefront. 4645 then Work-Group Info 1 {first_wavefront, 14'b0000, 4646 (enable_sgpr_workgroup ordered_append_term[10:0], 4647 _info) threadgroup_size_in_wavefronts[5:0]} 4648 then Scratch Wavefront Offset 1 See 4649 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4650 _segment_wavefront_offset) and 4651 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`. 4652 ========== ========================== ====== ============================== 4653 4654The order of the VGPR registers is defined, but the compiler can specify which 4655ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit 4656fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used 4657for enabled registers are dense starting at VGPR0: the first enabled register is 4658VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a 4659VGPR number. 4660 4661There are different methods used for the VGPR initial state: 4662 4663* Unless the *Target Properties* column of :ref:`amdgpu-processor-table` 4664 specifies otherwise, a separate VGPR register is used per work-item ID. The 4665 VGPR register initial state for this method is defined in 4666 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`. 4667* If *Target Properties* column of :ref:`amdgpu-processor-table` 4668 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used 4669 for all work-item IDs. The register layout for this method is defined in 4670 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`. 4671 4672 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method 4673 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table 4674 4675 ========== ========================== ====== ============================== 4676 VGPR Order Name Number Description 4677 (kernel descriptor enable of 4678 field) VGPRs 4679 ========== ========================== ====== ============================== 4680 First Work-Item Id X 1 32-bit work-item id in X 4681 (Always initialized) dimension of work-group for 4682 wavefront lane. 4683 then Work-Item Id Y 1 32-bit work-item id in Y 4684 (enable_vgpr_workitem_id dimension of work-group for 4685 > 0) wavefront lane. 4686 then Work-Item Id Z 1 32-bit work-item id in Z 4687 (enable_vgpr_workitem_id dimension of work-group for 4688 > 1) wavefront lane. 4689 ========== ========================== ====== ============================== 4690 4691.. 4692 4693 .. table:: Register Layout for Packed Work-Item ID Method 4694 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table 4695 4696 ======= ======= ================ ========================================= 4697 Bits Size Field Name Description 4698 ======= ======= ================ ========================================= 4699 0:9 10 bits Work-Item Id X Work-item id in X 4700 dimension of work-group for 4701 wavefront lane. 4702 4703 Always initialized. 4704 4705 10:19 10 bits Work-Item Id Y Work-item id in Y 4706 dimension of work-group for 4707 wavefront lane. 4708 4709 Initialized if enable_vgpr_workitem_id > 4710 0, otherwise set to 0. 4711 20:29 10 bits Work-Item Id Z Work-item id in Z 4712 dimension of work-group for 4713 wavefront lane. 4714 4715 Initialized if enable_vgpr_workitem_id > 4716 1, otherwise set to 0. 4717 30:31 2 bits Reserved, set to 0. 4718 ======= ======= ================ ========================================= 4719 4720The setting of registers is done by GPU CP/ADC/SPI hardware as follows: 4721 47221. SGPRs before the Work-Group Ids are set by CP using the 16 User Data 4723 registers. 47242. Work-group Id registers X, Y, Z are set by ADC which supports any 4725 combination including none. 47263. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why 4727 its value cannot be included with the flat scratch init value which is per 4728 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). 47294. The VGPRs are set by SPI which only supports specifying either (X), (X, Y) 4730 or (X, Y, Z). 47315. Flat Scratch register pair initialization is described in 4732 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 4733 4734The global segment can be accessed either using buffer instructions (GFX6 which 4735has V# 64-bit address support), flat instructions (GFX7-GFX10), or global 4736instructions (GFX9-GFX10). 4737 4738If buffer operations are used, then the compiler can generate a V# with the 4739following properties: 4740 4741* base address of 0 4742* no swizzle 4743* ATC: 1 if IOMMU present (such as APU) 4744* ptr64: 1 4745* MTYPE set to support memory coherence that matches the runtime (such as CC for 4746 APU and NC for dGPU). 4747 4748.. _amdgpu-amdhsa-kernel-prolog: 4749 4750Kernel Prolog 4751~~~~~~~~~~~~~ 4752 4753The compiler performs initialization in the kernel prologue depending on the 4754target and information about things like stack usage in the kernel and called 4755functions. Some of this initialization requires the compiler to request certain 4756User and System SGPRs be present in the 4757:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the 4758:ref:`amdgpu-amdhsa-kernel-descriptor`. 4759 4760.. _amdgpu-amdhsa-kernel-prolog-cfi: 4761 4762CFI 4763+++ 4764 47651. The CFI return address is undefined. 4766 47672. The CFI CFA is defined using an expression which evaluates to a location 4768 description that comprises one memory location description for the 4769 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``. 4770 4771.. _amdgpu-amdhsa-kernel-prolog-m0: 4772 4773M0 4774++ 4775 4776GFX6-GFX8 4777 The M0 register must be initialized with a value at least the total LDS size 4778 if the kernel may access LDS via DS or flat operations. Total LDS size is 4779 available in dispatch packet. For M0, it is also possible to use maximum 4780 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for 4781 GFX7-GFX8). 4782GFX9-GFX10 4783 The M0 register is not used for range checking LDS accesses and so does not 4784 need to be initialized in the prolog. 4785 4786.. _amdgpu-amdhsa-kernel-prolog-stack-pointer: 4787 4788Stack Pointer 4789+++++++++++++ 4790 4791If the kernel has function calls it must set up the ABI stack pointer described 4792in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting 4793SGPR32 to the unswizzled scratch offset of the address past the last local 4794allocation. 4795 4796.. _amdgpu-amdhsa-kernel-prolog-frame-pointer: 4797 4798Frame Pointer 4799+++++++++++++ 4800 4801If the kernel needs a frame pointer for the reasons defined in 4802``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the 4803kernel prolog. If a frame pointer is not required then all uses of the frame 4804pointer are replaced with immediate ``0`` offsets. 4805 4806.. _amdgpu-amdhsa-kernel-prolog-flat-scratch: 4807 4808Flat Scratch 4809++++++++++++ 4810 4811There are different methods used for initializing flat scratch: 4812 4813* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4814 specifies *Does not support generic address space*: 4815 4816 Flat scratch is not supported and there is no flat scratch register pair. 4817 4818* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4819 specifies *Offset flat scratch*: 4820 4821 If the kernel or any function it calls may use flat operations to access 4822 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4823 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and 4824 Scratch Wavefront Offset SGPR registers (see 4825 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4826 4827 1. The low word of Flat Scratch Init is the 32-bit byte offset from 4828 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory 4829 being managed by SPI for the queue executing the kernel dispatch. This is 4830 the same value used in the Scratch Segment Buffer V# base address. 4831 4832 CP obtains this from the runtime. (The Scratch Segment Buffer base address 4833 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.) 4834 4835 The prolog must add the value of Scratch Wavefront Offset to get the 4836 wavefront's byte scratch backing memory offset from 4837 ``SH_HIDDEN_PRIVATE_BASE_VIMID``. 4838 4839 The Scratch Wavefront Offset must also be used as an offset with Private 4840 segment address when using the Scratch Segment Buffer. 4841 4842 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right 4843 shifted by 8 before moving into FLAT_SCRATCH_HI. 4844 4845 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where 4846 SGPRn is the highest numbered SGPR allocated to the wavefront). 4847 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and 4848 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront 4849 FLAT SCRATCH BASE in flat memory instructions that access the scratch 4850 aperture. 4851 2. The second word of Flat Scratch Init is 32-bit byte size of a single 4852 work-items scratch memory usage. 4853 4854 CP obtains this from the runtime, and it is always a multiple of DWORD. CP 4855 checks that the value in the kernel dispatch packet Private Segment Byte 4856 Size is not larger and requests the runtime to increase the queue's scratch 4857 size if necessary. 4858 4859 CP directly loads from the kernel dispatch packet Private Segment Byte Size 4860 field and rounds up to a multiple of DWORD. Having CP load it once avoids 4861 loading it at the beginning of every wavefront. 4862 4863 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on 4864 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE 4865 in flat memory instructions. 4866 4867* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4868 specifies *Absolute flat scratch*: 4869 4870 If the kernel or any function it calls may use flat operations to access 4871 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair 4872 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization 4873 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see 4874 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`): 4875 4876 The Flat Scratch Init is the 64-bit address of the base of scratch backing 4877 memory being managed by SPI for the queue executing the kernel dispatch. 4878 4879 CP obtains this from the runtime. 4880 4881 The kernel prolog must add the value of the wave's Scratch Wavefront Offset 4882 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair 4883 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat 4884 memory instructions. 4885 4886 The Scratch Wavefront Offset must also be used as an offset with Private 4887 segment address when using the Scratch Segment Buffer (see 4888 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`). 4889 4890* If the *Target Properties* column of :ref:`amdgpu-processor-table` 4891 specifies *Architected flat scratch*: 4892 4893 If ENABLE_PRIVATE_SEGMENT is enabled in 4894 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` then the FLAT_SCRATCH 4895 register pair will be initialized to the 64-bit address of the base of scratch 4896 backing memory being managed by SPI for the queue executing the kernel 4897 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the 4898 flat scratch base in flat memory instructions. 4899 4900.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer: 4901 4902Private Segment Buffer 4903++++++++++++++++++++++ 4904 4905If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies 4906*Architected flat scratch* then a Private Segment Buffer is not supported. 4907Instead the flat SCRATCH instructions are used. 4908 4909Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs 4910that are used as a V# to access scratch. CP uses the value provided by the 4911runtime. It is used, together with Scratch Wavefront Offset as an offset, to 4912access the private memory space using a segment address. See 4913:ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 4914 4915The scratch V# is a four-aligned SGPR and always selected for the kernel as 4916follows: 4917 4918 - If it is known during instruction selection that there is stack usage, 4919 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if 4920 optimizations are disabled (``-O0``), if stack objects already exist (for 4921 locals, etc.), or if there are any function calls. 4922 4923 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index 4924 are reserved for the tentative scratch V#. These will be used if it is 4925 determined that spilling is needed. 4926 4927 - If no use is made of the tentative scratch V#, then it is unreserved, 4928 and the register count is determined ignoring it. 4929 - If use is made of the tentative scratch V#, then its register numbers 4930 are shifted to the first four-aligned SGPR index after the highest one 4931 allocated by the register allocator, and all uses are updated. The 4932 register count includes them in the shifted location. 4933 - In either case, if the processor has the SGPR allocation bug, the 4934 tentative allocation is not shifted or unreserved in order to ensure 4935 the register count is higher to workaround the bug. 4936 4937 .. note:: 4938 4939 This approach of using a tentative scratch V# and shifting the register 4940 numbers if used avoids having to perform register allocation a second 4941 time if the tentative V# is eliminated. This is more efficient and 4942 avoids the problem that the second register allocation may perform 4943 spilling which will fail as there is no longer a scratch V#. 4944 4945When the kernel prolog code is being emitted it is known whether the scratch V# 4946described above is actually used. If it is, the prolog code must set it up by 4947copying the Private Segment Buffer to the scratch V# registers and then adding 4948the Private Segment Wavefront Offset to the queue base address in the V#. The 4949result is a V# with a base address pointing to the beginning of the wavefront 4950scratch backing memory. 4951 4952The Private Segment Buffer is always requested, but the Private Segment 4953Wavefront Offset is only requested if it is used (see 4954:ref:`amdgpu-amdhsa-initial-kernel-execution-state`). 4955 4956.. _amdgpu-amdhsa-memory-model: 4957 4958Memory Model 4959~~~~~~~~~~~~ 4960 4961This section describes the mapping of the LLVM memory model onto AMDGPU machine 4962code (see :ref:`memmodel`). 4963 4964The AMDGPU backend supports the memory synchronization scopes specified in 4965:ref:`amdgpu-memory-scopes`. 4966 4967The code sequences used to implement the memory model specify the order of 4968instructions that a single thread must execute. The ``s_waitcnt`` and cache 4969management instructions such as ``buffer_wbinvl1_vol`` are defined with respect 4970to other memory instructions executed by the same thread. This allows them to be 4971moved earlier or later which can allow them to be combined with other instances 4972of the same instruction, or hoisted/sunk out of loops to improve performance. 4973Only the instructions related to the memory model are given; additional 4974``s_waitcnt`` instructions are required to ensure registers are defined before 4975being used. These may be able to be combined with the memory model ``s_waitcnt`` 4976instructions as described above. 4977 4978The AMDGPU backend supports the following memory models: 4979 4980 HSA Memory Model [HSA]_ 4981 The HSA memory model uses a single happens-before relation for all address 4982 spaces (see :ref:`amdgpu-address-spaces`). 4983 OpenCL Memory Model [OpenCL]_ 4984 The OpenCL memory model which has separate happens-before relations for the 4985 global and local address spaces. Only a fence specifying both global and 4986 local address space, and seq_cst instructions join the relationships. Since 4987 the LLVM ``memfence`` instruction does not allow an address space to be 4988 specified the OpenCL fence has to conservatively assume both local and 4989 global address space was specified. However, optimizations can often be 4990 done to eliminate the additional ``s_waitcnt`` instructions when there are 4991 no intervening memory instructions which access the corresponding address 4992 space. The code sequences in the table indicate what can be omitted for the 4993 OpenCL memory. The target triple environment is used to determine if the 4994 source language is OpenCL (see :ref:`amdgpu-opencl`). 4995 4996``ds/flat_load/store/atomic`` instructions to local memory are termed LDS 4997operations. 4998 4999``buffer/global/flat_load/store/atomic`` instructions to global memory are 5000termed vector memory operations. 5001 5002Private address space uses ``buffer_load/store`` using the scratch V# 5003(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX10). Since only a single thread 5004is accessing the memory, atomic memory orderings are not meaningful, and all 5005accesses are treated as non-atomic. 5006 5007Constant address space uses ``buffer/global_load`` instructions (or equivalent 5008scalar memory instructions). Since the constant address space contents do not 5009change during the execution of a kernel dispatch it is not legal to perform 5010stores, and atomic memory orderings are not meaningful, and all accesses are 5011treated as non-atomic. 5012 5013A memory synchronization scope wider than work-group is not meaningful for the 5014group (LDS) address space and is treated as work-group. 5015 5016The memory model does not support the region address space which is treated as 5017non-atomic. 5018 5019Acquire memory ordering is not meaningful on store atomic instructions and is 5020treated as non-atomic. 5021 5022Release memory ordering is not meaningful on load atomic instructions and is 5023treated a non-atomic. 5024 5025Acquire-release memory ordering is not meaningful on load or store atomic 5026instructions and is treated as acquire and release respectively. 5027 5028The memory order also adds the single thread optimization constraints defined in 5029table 5030:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`. 5031 5032 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints 5033 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table 5034 5035 ============ ============================================================== 5036 LLVM Memory Optimization Constraints 5037 Ordering 5038 ============ ============================================================== 5039 unordered *none* 5040 monotonic *none* 5041 acquire - If a load atomic/atomicrmw then no following load/load 5042 atomic/store/store atomic/atomicrmw/fence instruction can be 5043 moved before the acquire. 5044 - If a fence then same as load atomic, plus no preceding 5045 associated fence-paired-atomic can be moved after the fence. 5046 release - If a store atomic/atomicrmw then no preceding load/load 5047 atomic/store/store atomic/atomicrmw/fence instruction can be 5048 moved after the release. 5049 - If a fence then same as store atomic, plus no following 5050 associated fence-paired-atomic can be moved before the 5051 fence. 5052 acq_rel Same constraints as both acquire and release. 5053 seq_cst - If a load atomic then same constraints as acquire, plus no 5054 preceding sequentially consistent load atomic/store 5055 atomic/atomicrmw/fence instruction can be moved after the 5056 seq_cst. 5057 - If a store atomic then the same constraints as release, plus 5058 no following sequentially consistent load atomic/store 5059 atomic/atomicrmw/fence instruction can be moved before the 5060 seq_cst. 5061 - If an atomicrmw/fence then same constraints as acq_rel. 5062 ============ ============================================================== 5063 5064The code sequences used to implement the memory model are defined in the 5065following sections: 5066 5067* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9` 5068* :ref:`amdgpu-amdhsa-memory-model-gfx90a` 5069* :ref:`amdgpu-amdhsa-memory-model-gfx940` 5070* :ref:`amdgpu-amdhsa-memory-model-gfx10` 5071 5072.. _amdgpu-amdhsa-memory-model-gfx6-gfx9: 5073 5074Memory Model GFX6-GFX9 5075++++++++++++++++++++++ 5076 5077For GFX6-GFX9: 5078 5079* Each agent has multiple shader arrays (SA). 5080* Each SA has multiple compute units (CU). 5081* Each CU has multiple SIMDs that execute wavefronts. 5082* The wavefronts for a single work-group are executed in the same CU but may be 5083 executed by different SIMDs. 5084* Each CU has a single LDS memory shared by the wavefronts of the work-groups 5085 executing on it. 5086* All LDS operations of a CU are performed as wavefront wide operations in a 5087 global order and involve no caching. Completion is reported to a wavefront in 5088 execution order. 5089* The LDS memory has multiple request queues shared by the SIMDs of a 5090 CU. Therefore, the LDS operations performed by different wavefronts of a 5091 work-group can be reordered relative to each other, which can result in 5092 reordering the visibility of vector memory operations with respect to LDS 5093 operations of other wavefronts in the same work-group. A ``s_waitcnt 5094 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 5095 vector memory operations between wavefronts of a work-group, but not between 5096 operations performed by the same wavefront. 5097* The vector memory operations are performed as wavefront wide operations and 5098 completion is reported to a wavefront in execution order. The exception is 5099 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of 5100 vector memory order if they access LDS memory, and out of LDS operation order 5101 if they access global memory. 5102* The vector memory operations access a single vector L1 cache shared by all 5103 SIMDs a CU. Therefore, no special action is required for coherence between the 5104 lanes of a single wavefront, or for coherence between wavefronts in the same 5105 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between 5106 wavefronts executing in different work-groups as they may be executing on 5107 different CUs. 5108* The scalar memory operations access a scalar L1 cache shared by all wavefronts 5109 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 5110 scalar operations are used in a restricted way so do not impact the memory 5111 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 5112* The vector and scalar memory operations use an L2 cache shared by all CUs on 5113 the same agent. 5114* The L2 cache has independent channels to service disjoint ranges of virtual 5115 addresses. 5116* Each CU has a separate request queue per channel. Therefore, the vector and 5117 scalar memory operations performed by wavefronts executing in different 5118 work-groups (which may be executing on different CUs) of an agent can be 5119 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to 5120 ensure synchronization between vector memory operations of different CUs. It 5121 ensures a previous vector memory operation has completed before executing a 5122 subsequent vector memory or LDS operation and so can be used to meet the 5123 requirements of acquire and release. 5124* The L2 cache can be kept coherent with other agents on some targets, or ranges 5125 of virtual addresses can be set up to bypass it to ensure system coherence. 5126 5127Scalar memory operations are only used to access memory that is proven to not 5128change during the execution of the kernel dispatch. This includes constant 5129address space and global address space for program scope ``const`` variables. 5130Therefore, the kernel machine code does not have to maintain the scalar cache to 5131ensure it is coherent with the vector caches. The scalar and vector caches are 5132invalidated between kernel dispatches by CP since constant address space data 5133may change between kernel dispatch executions. See 5134:ref:`amdgpu-amdhsa-memory-spaces`. 5135 5136The one exception is if scalar writes are used to spill SGPR registers. In this 5137case the AMDGPU backend ensures the memory location used to spill is never 5138accessed by vector memory operations at the same time. If scalar writes are used 5139then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 5140return since the locations may be used for vector memory instructions by a 5141future wavefront that uses the same scratch area, or a function call that 5142creates a frame at the same address, respectively. There is no need for a 5143``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 5144 5145For kernarg backing memory: 5146 5147* CP invalidates the L1 cache at the start of each kernel dispatch. 5148* On dGPU the kernarg backing memory is allocated in host memory accessed as 5149 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also 5150 causes it to be treated as non-volatile and so is not invalidated by 5151 ``*_vol``. 5152* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent) 5153 and so the L2 cache will be coherent with the CPU and other agents. 5154 5155Scratch backing memory (which is used for the private address space) is accessed 5156with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 5157only accessed by a single thread, and is always write-before-read, there is 5158never a need to invalidate these entries from the L1 cache. Hence all cache 5159invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 5160 5161The code sequences used to implement the memory model for GFX6-GFX9 are defined 5162in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`. 5163 5164 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX9 5165 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table 5166 5167 ============ ============ ============== ========== ================================ 5168 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 5169 Ordering Sync Scope Address GFX6-GFX9 5170 Space 5171 ============ ============ ============== ========== ================================ 5172 **Non-Atomic** 5173 ------------------------------------------------------------------------------------ 5174 load *none* *none* - global - !volatile & !nontemporal 5175 - generic 5176 - private 1. buffer/global/flat_load 5177 - constant 5178 - !volatile & nontemporal 5179 5180 1. buffer/global/flat_load 5181 glc=1 slc=1 5182 5183 - volatile 5184 5185 1. buffer/global/flat_load 5186 glc=1 5187 2. s_waitcnt vmcnt(0) 5188 5189 - Must happen before 5190 any following volatile 5191 global/generic 5192 load/store. 5193 - Ensures that 5194 volatile 5195 operations to 5196 different 5197 addresses will not 5198 be reordered by 5199 hardware. 5200 5201 load *none* *none* - local 1. ds_load 5202 store *none* *none* - global - !volatile & !nontemporal 5203 - generic 5204 - private 1. buffer/global/flat_store 5205 - constant 5206 - !volatile & nontemporal 5207 5208 1. buffer/global/flat_store 5209 glc=1 slc=1 5210 5211 - volatile 5212 5213 1. buffer/global/flat_store 5214 2. s_waitcnt vmcnt(0) 5215 5216 - Must happen before 5217 any following volatile 5218 global/generic 5219 load/store. 5220 - Ensures that 5221 volatile 5222 operations to 5223 different 5224 addresses will not 5225 be reordered by 5226 hardware. 5227 5228 store *none* *none* - local 1. ds_store 5229 **Unordered Atomic** 5230 ------------------------------------------------------------------------------------ 5231 load atomic unordered *any* *any* *Same as non-atomic*. 5232 store atomic unordered *any* *any* *Same as non-atomic*. 5233 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 5234 **Monotonic Atomic** 5235 ------------------------------------------------------------------------------------ 5236 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load 5237 - wavefront - local 5238 - workgroup - generic 5239 load atomic monotonic - agent - global 1. buffer/global/flat_load 5240 - system - generic glc=1 5241 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 5242 - wavefront - generic 5243 - workgroup 5244 - agent 5245 - system 5246 store atomic monotonic - singlethread - local 1. ds_store 5247 - wavefront 5248 - workgroup 5249 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 5250 - wavefront - generic 5251 - workgroup 5252 - agent 5253 - system 5254 atomicrmw monotonic - singlethread - local 1. ds_atomic 5255 - wavefront 5256 - workgroup 5257 **Acquire Atomic** 5258 ------------------------------------------------------------------------------------ 5259 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 5260 - wavefront - local 5261 - generic 5262 load atomic acquire - workgroup - global 1. buffer/global_load 5263 load atomic acquire - workgroup - local 1. ds/flat_load 5264 - generic 2. s_waitcnt lgkmcnt(0) 5265 5266 - If OpenCL, omit. 5267 - Must happen before 5268 any following 5269 global/generic 5270 load/load 5271 atomic/store/store 5272 atomic/atomicrmw. 5273 - Ensures any 5274 following global 5275 data read is no 5276 older than a local load 5277 atomic value being 5278 acquired. 5279 5280 load atomic acquire - agent - global 1. buffer/global_load 5281 - system glc=1 5282 2. s_waitcnt vmcnt(0) 5283 5284 - Must happen before 5285 following 5286 buffer_wbinvl1_vol. 5287 - Ensures the load 5288 has completed 5289 before invalidating 5290 the cache. 5291 5292 3. buffer_wbinvl1_vol 5293 5294 - Must happen before 5295 any following 5296 global/generic 5297 load/load 5298 atomic/atomicrmw. 5299 - Ensures that 5300 following 5301 loads will not see 5302 stale global data. 5303 5304 load atomic acquire - agent - generic 1. flat_load glc=1 5305 - system 2. s_waitcnt vmcnt(0) & 5306 lgkmcnt(0) 5307 5308 - If OpenCL omit 5309 lgkmcnt(0). 5310 - Must happen before 5311 following 5312 buffer_wbinvl1_vol. 5313 - Ensures the flat_load 5314 has completed 5315 before invalidating 5316 the cache. 5317 5318 3. buffer_wbinvl1_vol 5319 5320 - Must happen before 5321 any following 5322 global/generic 5323 load/load 5324 atomic/atomicrmw. 5325 - Ensures that 5326 following loads 5327 will not see stale 5328 global data. 5329 5330 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 5331 - wavefront - local 5332 - generic 5333 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 5334 atomicrmw acquire - workgroup - local 1. ds/flat_atomic 5335 - generic 2. s_waitcnt lgkmcnt(0) 5336 5337 - If OpenCL, omit. 5338 - Must happen before 5339 any following 5340 global/generic 5341 load/load 5342 atomic/store/store 5343 atomic/atomicrmw. 5344 - Ensures any 5345 following global 5346 data read is no 5347 older than a local 5348 atomicrmw value 5349 being acquired. 5350 5351 atomicrmw acquire - agent - global 1. buffer/global_atomic 5352 - system 2. s_waitcnt vmcnt(0) 5353 5354 - Must happen before 5355 following 5356 buffer_wbinvl1_vol. 5357 - Ensures the 5358 atomicrmw has 5359 completed before 5360 invalidating the 5361 cache. 5362 5363 3. buffer_wbinvl1_vol 5364 5365 - Must happen before 5366 any following 5367 global/generic 5368 load/load 5369 atomic/atomicrmw. 5370 - Ensures that 5371 following loads 5372 will not see stale 5373 global data. 5374 5375 atomicrmw acquire - agent - generic 1. flat_atomic 5376 - system 2. s_waitcnt vmcnt(0) & 5377 lgkmcnt(0) 5378 5379 - If OpenCL, omit 5380 lgkmcnt(0). 5381 - Must happen before 5382 following 5383 buffer_wbinvl1_vol. 5384 - Ensures the 5385 atomicrmw has 5386 completed before 5387 invalidating the 5388 cache. 5389 5390 3. buffer_wbinvl1_vol 5391 5392 - Must happen before 5393 any following 5394 global/generic 5395 load/load 5396 atomic/atomicrmw. 5397 - Ensures that 5398 following loads 5399 will not see stale 5400 global data. 5401 5402 fence acquire - singlethread *none* *none* 5403 - wavefront 5404 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5405 5406 - If OpenCL and 5407 address space is 5408 not generic, omit. 5409 - However, since LLVM 5410 currently has no 5411 address space on 5412 the fence need to 5413 conservatively 5414 always generate. If 5415 fence had an 5416 address space then 5417 set to address 5418 space of OpenCL 5419 fence flag, or to 5420 generic if both 5421 local and global 5422 flags are 5423 specified. 5424 - Must happen after 5425 any preceding 5426 local/generic load 5427 atomic/atomicrmw 5428 with an equal or 5429 wider sync scope 5430 and memory ordering 5431 stronger than 5432 unordered (this is 5433 termed the 5434 fence-paired-atomic). 5435 - Must happen before 5436 any following 5437 global/generic 5438 load/load 5439 atomic/store/store 5440 atomic/atomicrmw. 5441 - Ensures any 5442 following global 5443 data read is no 5444 older than the 5445 value read by the 5446 fence-paired-atomic. 5447 5448 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 5449 - system vmcnt(0) 5450 5451 - If OpenCL and 5452 address space is 5453 not generic, omit 5454 lgkmcnt(0). 5455 - However, since LLVM 5456 currently has no 5457 address space on 5458 the fence need to 5459 conservatively 5460 always generate 5461 (see comment for 5462 previous fence). 5463 - Could be split into 5464 separate s_waitcnt 5465 vmcnt(0) and 5466 s_waitcnt 5467 lgkmcnt(0) to allow 5468 them to be 5469 independently moved 5470 according to the 5471 following rules. 5472 - s_waitcnt vmcnt(0) 5473 must happen after 5474 any preceding 5475 global/generic load 5476 atomic/atomicrmw 5477 with an equal or 5478 wider sync scope 5479 and memory ordering 5480 stronger than 5481 unordered (this is 5482 termed the 5483 fence-paired-atomic). 5484 - s_waitcnt lgkmcnt(0) 5485 must happen after 5486 any preceding 5487 local/generic load 5488 atomic/atomicrmw 5489 with an equal or 5490 wider sync scope 5491 and memory ordering 5492 stronger than 5493 unordered (this is 5494 termed the 5495 fence-paired-atomic). 5496 - Must happen before 5497 the following 5498 buffer_wbinvl1_vol. 5499 - Ensures that the 5500 fence-paired atomic 5501 has completed 5502 before invalidating 5503 the 5504 cache. Therefore 5505 any following 5506 locations read must 5507 be no older than 5508 the value read by 5509 the 5510 fence-paired-atomic. 5511 5512 2. buffer_wbinvl1_vol 5513 5514 - Must happen before any 5515 following global/generic 5516 load/load 5517 atomic/store/store 5518 atomic/atomicrmw. 5519 - Ensures that 5520 following loads 5521 will not see stale 5522 global data. 5523 5524 **Release Atomic** 5525 ------------------------------------------------------------------------------------ 5526 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 5527 - wavefront - local 5528 - generic 5529 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5530 - generic 5531 - If OpenCL, omit. 5532 - Must happen after 5533 any preceding 5534 local/generic 5535 load/store/load 5536 atomic/store 5537 atomic/atomicrmw. 5538 - Must happen before 5539 the following 5540 store. 5541 - Ensures that all 5542 memory operations 5543 to local have 5544 completed before 5545 performing the 5546 store that is being 5547 released. 5548 5549 2. buffer/global/flat_store 5550 store atomic release - workgroup - local 1. ds_store 5551 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 5552 - system - generic vmcnt(0) 5553 5554 - If OpenCL and 5555 address space is 5556 not generic, omit 5557 lgkmcnt(0). 5558 - Could be split into 5559 separate s_waitcnt 5560 vmcnt(0) and 5561 s_waitcnt 5562 lgkmcnt(0) to allow 5563 them to be 5564 independently moved 5565 according to the 5566 following rules. 5567 - s_waitcnt vmcnt(0) 5568 must happen after 5569 any preceding 5570 global/generic 5571 load/store/load 5572 atomic/store 5573 atomic/atomicrmw. 5574 - s_waitcnt lgkmcnt(0) 5575 must happen after 5576 any preceding 5577 local/generic 5578 load/store/load 5579 atomic/store 5580 atomic/atomicrmw. 5581 - Must happen before 5582 the following 5583 store. 5584 - Ensures that all 5585 memory operations 5586 to memory have 5587 completed before 5588 performing the 5589 store that is being 5590 released. 5591 5592 2. buffer/global/flat_store 5593 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 5594 - wavefront - local 5595 - generic 5596 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) 5597 - generic 5598 - If OpenCL, omit. 5599 - Must happen after 5600 any preceding 5601 local/generic 5602 load/store/load 5603 atomic/store 5604 atomic/atomicrmw. 5605 - Must happen before 5606 the following 5607 atomicrmw. 5608 - Ensures that all 5609 memory operations 5610 to local have 5611 completed before 5612 performing the 5613 atomicrmw that is 5614 being released. 5615 5616 2. buffer/global/flat_atomic 5617 atomicrmw release - workgroup - local 1. ds_atomic 5618 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 5619 - system - generic vmcnt(0) 5620 5621 - If OpenCL, omit 5622 lgkmcnt(0). 5623 - Could be split into 5624 separate s_waitcnt 5625 vmcnt(0) and 5626 s_waitcnt 5627 lgkmcnt(0) to allow 5628 them to be 5629 independently moved 5630 according to the 5631 following rules. 5632 - s_waitcnt vmcnt(0) 5633 must happen after 5634 any preceding 5635 global/generic 5636 load/store/load 5637 atomic/store 5638 atomic/atomicrmw. 5639 - s_waitcnt lgkmcnt(0) 5640 must happen after 5641 any preceding 5642 local/generic 5643 load/store/load 5644 atomic/store 5645 atomic/atomicrmw. 5646 - Must happen before 5647 the following 5648 atomicrmw. 5649 - Ensures that all 5650 memory operations 5651 to global and local 5652 have completed 5653 before performing 5654 the atomicrmw that 5655 is being released. 5656 5657 2. buffer/global/flat_atomic 5658 fence release - singlethread *none* *none* 5659 - wavefront 5660 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5661 5662 - If OpenCL and 5663 address space is 5664 not generic, omit. 5665 - However, since LLVM 5666 currently has no 5667 address space on 5668 the fence need to 5669 conservatively 5670 always generate. If 5671 fence had an 5672 address space then 5673 set to address 5674 space of OpenCL 5675 fence flag, or to 5676 generic if both 5677 local and global 5678 flags are 5679 specified. 5680 - Must happen after 5681 any preceding 5682 local/generic 5683 load/load 5684 atomic/store/store 5685 atomic/atomicrmw. 5686 - Must happen before 5687 any following store 5688 atomic/atomicrmw 5689 with an equal or 5690 wider sync scope 5691 and memory ordering 5692 stronger than 5693 unordered (this is 5694 termed the 5695 fence-paired-atomic). 5696 - Ensures that all 5697 memory operations 5698 to local have 5699 completed before 5700 performing the 5701 following 5702 fence-paired-atomic. 5703 5704 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 5705 - system vmcnt(0) 5706 5707 - If OpenCL and 5708 address space is 5709 not generic, omit 5710 lgkmcnt(0). 5711 - If OpenCL and 5712 address space is 5713 local, omit 5714 vmcnt(0). 5715 - However, since LLVM 5716 currently has no 5717 address space on 5718 the fence need to 5719 conservatively 5720 always generate. If 5721 fence had an 5722 address space then 5723 set to address 5724 space of OpenCL 5725 fence flag, or to 5726 generic if both 5727 local and global 5728 flags are 5729 specified. 5730 - Could be split into 5731 separate s_waitcnt 5732 vmcnt(0) and 5733 s_waitcnt 5734 lgkmcnt(0) to allow 5735 them to be 5736 independently moved 5737 according to the 5738 following rules. 5739 - s_waitcnt vmcnt(0) 5740 must happen after 5741 any preceding 5742 global/generic 5743 load/store/load 5744 atomic/store 5745 atomic/atomicrmw. 5746 - s_waitcnt lgkmcnt(0) 5747 must happen after 5748 any preceding 5749 local/generic 5750 load/store/load 5751 atomic/store 5752 atomic/atomicrmw. 5753 - Must happen before 5754 any following store 5755 atomic/atomicrmw 5756 with an equal or 5757 wider sync scope 5758 and memory ordering 5759 stronger than 5760 unordered (this is 5761 termed the 5762 fence-paired-atomic). 5763 - Ensures that all 5764 memory operations 5765 have 5766 completed before 5767 performing the 5768 following 5769 fence-paired-atomic. 5770 5771 **Acquire-Release Atomic** 5772 ------------------------------------------------------------------------------------ 5773 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 5774 - wavefront - local 5775 - generic 5776 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) 5777 5778 - If OpenCL, omit. 5779 - Must happen after 5780 any preceding 5781 local/generic 5782 load/store/load 5783 atomic/store 5784 atomic/atomicrmw. 5785 - Must happen before 5786 the following 5787 atomicrmw. 5788 - Ensures that all 5789 memory operations 5790 to local have 5791 completed before 5792 performing the 5793 atomicrmw that is 5794 being released. 5795 5796 2. buffer/global_atomic 5797 5798 atomicrmw acq_rel - workgroup - local 1. ds_atomic 5799 2. s_waitcnt lgkmcnt(0) 5800 5801 - If OpenCL, omit. 5802 - Must happen before 5803 any following 5804 global/generic 5805 load/load 5806 atomic/store/store 5807 atomic/atomicrmw. 5808 - Ensures any 5809 following global 5810 data read is no 5811 older than the local load 5812 atomic value being 5813 acquired. 5814 5815 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) 5816 5817 - If OpenCL, omit. 5818 - Must happen after 5819 any preceding 5820 local/generic 5821 load/store/load 5822 atomic/store 5823 atomic/atomicrmw. 5824 - Must happen before 5825 the following 5826 atomicrmw. 5827 - Ensures that all 5828 memory operations 5829 to local have 5830 completed before 5831 performing the 5832 atomicrmw that is 5833 being released. 5834 5835 2. flat_atomic 5836 3. s_waitcnt lgkmcnt(0) 5837 5838 - If OpenCL, omit. 5839 - Must happen before 5840 any following 5841 global/generic 5842 load/load 5843 atomic/store/store 5844 atomic/atomicrmw. 5845 - Ensures any 5846 following global 5847 data read is no 5848 older than a local load 5849 atomic value being 5850 acquired. 5851 5852 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 5853 - system vmcnt(0) 5854 5855 - If OpenCL, omit 5856 lgkmcnt(0). 5857 - Could be split into 5858 separate s_waitcnt 5859 vmcnt(0) and 5860 s_waitcnt 5861 lgkmcnt(0) to allow 5862 them to be 5863 independently moved 5864 according to the 5865 following rules. 5866 - s_waitcnt vmcnt(0) 5867 must happen after 5868 any preceding 5869 global/generic 5870 load/store/load 5871 atomic/store 5872 atomic/atomicrmw. 5873 - s_waitcnt lgkmcnt(0) 5874 must happen after 5875 any preceding 5876 local/generic 5877 load/store/load 5878 atomic/store 5879 atomic/atomicrmw. 5880 - Must happen before 5881 the following 5882 atomicrmw. 5883 - Ensures that all 5884 memory operations 5885 to global have 5886 completed before 5887 performing the 5888 atomicrmw that is 5889 being released. 5890 5891 2. buffer/global_atomic 5892 3. s_waitcnt vmcnt(0) 5893 5894 - Must happen before 5895 following 5896 buffer_wbinvl1_vol. 5897 - Ensures the 5898 atomicrmw has 5899 completed before 5900 invalidating the 5901 cache. 5902 5903 4. buffer_wbinvl1_vol 5904 5905 - Must happen before 5906 any following 5907 global/generic 5908 load/load 5909 atomic/atomicrmw. 5910 - Ensures that 5911 following loads 5912 will not see stale 5913 global data. 5914 5915 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 5916 - system vmcnt(0) 5917 5918 - If OpenCL, omit 5919 lgkmcnt(0). 5920 - Could be split into 5921 separate s_waitcnt 5922 vmcnt(0) and 5923 s_waitcnt 5924 lgkmcnt(0) to allow 5925 them to be 5926 independently moved 5927 according to the 5928 following rules. 5929 - s_waitcnt vmcnt(0) 5930 must happen after 5931 any preceding 5932 global/generic 5933 load/store/load 5934 atomic/store 5935 atomic/atomicrmw. 5936 - s_waitcnt lgkmcnt(0) 5937 must happen after 5938 any preceding 5939 local/generic 5940 load/store/load 5941 atomic/store 5942 atomic/atomicrmw. 5943 - Must happen before 5944 the following 5945 atomicrmw. 5946 - Ensures that all 5947 memory operations 5948 to global have 5949 completed before 5950 performing the 5951 atomicrmw that is 5952 being released. 5953 5954 2. flat_atomic 5955 3. s_waitcnt vmcnt(0) & 5956 lgkmcnt(0) 5957 5958 - If OpenCL, omit 5959 lgkmcnt(0). 5960 - Must happen before 5961 following 5962 buffer_wbinvl1_vol. 5963 - Ensures the 5964 atomicrmw has 5965 completed before 5966 invalidating the 5967 cache. 5968 5969 4. buffer_wbinvl1_vol 5970 5971 - Must happen before 5972 any following 5973 global/generic 5974 load/load 5975 atomic/atomicrmw. 5976 - Ensures that 5977 following loads 5978 will not see stale 5979 global data. 5980 5981 fence acq_rel - singlethread *none* *none* 5982 - wavefront 5983 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) 5984 5985 - If OpenCL and 5986 address space is 5987 not generic, omit. 5988 - However, 5989 since LLVM 5990 currently has no 5991 address space on 5992 the fence need to 5993 conservatively 5994 always generate 5995 (see comment for 5996 previous fence). 5997 - Must happen after 5998 any preceding 5999 local/generic 6000 load/load 6001 atomic/store/store 6002 atomic/atomicrmw. 6003 - Must happen before 6004 any following 6005 global/generic 6006 load/load 6007 atomic/store/store 6008 atomic/atomicrmw. 6009 - Ensures that all 6010 memory operations 6011 to local have 6012 completed before 6013 performing any 6014 following global 6015 memory operations. 6016 - Ensures that the 6017 preceding 6018 local/generic load 6019 atomic/atomicrmw 6020 with an equal or 6021 wider sync scope 6022 and memory ordering 6023 stronger than 6024 unordered (this is 6025 termed the 6026 acquire-fence-paired-atomic) 6027 has completed 6028 before following 6029 global memory 6030 operations. This 6031 satisfies the 6032 requirements of 6033 acquire. 6034 - Ensures that all 6035 previous memory 6036 operations have 6037 completed before a 6038 following 6039 local/generic store 6040 atomic/atomicrmw 6041 with an equal or 6042 wider sync scope 6043 and memory ordering 6044 stronger than 6045 unordered (this is 6046 termed the 6047 release-fence-paired-atomic). 6048 This satisfies the 6049 requirements of 6050 release. 6051 6052 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 6053 - system vmcnt(0) 6054 6055 - If OpenCL and 6056 address space is 6057 not generic, omit 6058 lgkmcnt(0). 6059 - However, since LLVM 6060 currently has no 6061 address space on 6062 the fence need to 6063 conservatively 6064 always generate 6065 (see comment for 6066 previous fence). 6067 - Could be split into 6068 separate s_waitcnt 6069 vmcnt(0) and 6070 s_waitcnt 6071 lgkmcnt(0) to allow 6072 them to be 6073 independently moved 6074 according to the 6075 following rules. 6076 - s_waitcnt vmcnt(0) 6077 must happen after 6078 any preceding 6079 global/generic 6080 load/store/load 6081 atomic/store 6082 atomic/atomicrmw. 6083 - s_waitcnt lgkmcnt(0) 6084 must happen after 6085 any preceding 6086 local/generic 6087 load/store/load 6088 atomic/store 6089 atomic/atomicrmw. 6090 - Must happen before 6091 the following 6092 buffer_wbinvl1_vol. 6093 - Ensures that the 6094 preceding 6095 global/local/generic 6096 load 6097 atomic/atomicrmw 6098 with an equal or 6099 wider sync scope 6100 and memory ordering 6101 stronger than 6102 unordered (this is 6103 termed the 6104 acquire-fence-paired-atomic) 6105 has completed 6106 before invalidating 6107 the cache. This 6108 satisfies the 6109 requirements of 6110 acquire. 6111 - Ensures that all 6112 previous memory 6113 operations have 6114 completed before a 6115 following 6116 global/local/generic 6117 store 6118 atomic/atomicrmw 6119 with an equal or 6120 wider sync scope 6121 and memory ordering 6122 stronger than 6123 unordered (this is 6124 termed the 6125 release-fence-paired-atomic). 6126 This satisfies the 6127 requirements of 6128 release. 6129 6130 2. buffer_wbinvl1_vol 6131 6132 - Must happen before 6133 any following 6134 global/generic 6135 load/load 6136 atomic/store/store 6137 atomic/atomicrmw. 6138 - Ensures that 6139 following loads 6140 will not see stale 6141 global data. This 6142 satisfies the 6143 requirements of 6144 acquire. 6145 6146 **Sequential Consistent Atomic** 6147 ------------------------------------------------------------------------------------ 6148 load atomic seq_cst - singlethread - global *Same as corresponding 6149 - wavefront - local load atomic acquire, 6150 - generic except must generate 6151 all instructions even 6152 for OpenCL.* 6153 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) 6154 - generic 6155 6156 - Must 6157 happen after 6158 preceding 6159 local/generic load 6160 atomic/store 6161 atomic/atomicrmw 6162 with memory 6163 ordering of seq_cst 6164 and with equal or 6165 wider sync scope. 6166 (Note that seq_cst 6167 fences have their 6168 own s_waitcnt 6169 lgkmcnt(0) and so do 6170 not need to be 6171 considered.) 6172 - Ensures any 6173 preceding 6174 sequential 6175 consistent local 6176 memory instructions 6177 have completed 6178 before executing 6179 this sequentially 6180 consistent 6181 instruction. This 6182 prevents reordering 6183 a seq_cst store 6184 followed by a 6185 seq_cst load. (Note 6186 that seq_cst is 6187 stronger than 6188 acquire/release as 6189 the reordering of 6190 load acquire 6191 followed by a store 6192 release is 6193 prevented by the 6194 s_waitcnt of 6195 the release, but 6196 there is nothing 6197 preventing a store 6198 release followed by 6199 load acquire from 6200 completing out of 6201 order. The s_waitcnt 6202 could be placed after 6203 seq_store or before 6204 the seq_load. We 6205 choose the load to 6206 make the s_waitcnt be 6207 as late as possible 6208 so that the store 6209 may have already 6210 completed.) 6211 6212 2. *Following 6213 instructions same as 6214 corresponding load 6215 atomic acquire, 6216 except must generate 6217 all instructions even 6218 for OpenCL.* 6219 load atomic seq_cst - workgroup - local *Same as corresponding 6220 load atomic acquire, 6221 except must generate 6222 all instructions even 6223 for OpenCL.* 6224 6225 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 6226 - system - generic vmcnt(0) 6227 6228 - Could be split into 6229 separate s_waitcnt 6230 vmcnt(0) 6231 and s_waitcnt 6232 lgkmcnt(0) to allow 6233 them to be 6234 independently moved 6235 according to the 6236 following rules. 6237 - s_waitcnt lgkmcnt(0) 6238 must happen after 6239 preceding 6240 global/generic load 6241 atomic/store 6242 atomic/atomicrmw 6243 with memory 6244 ordering of seq_cst 6245 and with equal or 6246 wider sync scope. 6247 (Note that seq_cst 6248 fences have their 6249 own s_waitcnt 6250 lgkmcnt(0) and so do 6251 not need to be 6252 considered.) 6253 - s_waitcnt vmcnt(0) 6254 must happen after 6255 preceding 6256 global/generic load 6257 atomic/store 6258 atomic/atomicrmw 6259 with memory 6260 ordering of seq_cst 6261 and with equal or 6262 wider sync scope. 6263 (Note that seq_cst 6264 fences have their 6265 own s_waitcnt 6266 vmcnt(0) and so do 6267 not need to be 6268 considered.) 6269 - Ensures any 6270 preceding 6271 sequential 6272 consistent global 6273 memory instructions 6274 have completed 6275 before executing 6276 this sequentially 6277 consistent 6278 instruction. This 6279 prevents reordering 6280 a seq_cst store 6281 followed by a 6282 seq_cst load. (Note 6283 that seq_cst is 6284 stronger than 6285 acquire/release as 6286 the reordering of 6287 load acquire 6288 followed by a store 6289 release is 6290 prevented by the 6291 s_waitcnt of 6292 the release, but 6293 there is nothing 6294 preventing a store 6295 release followed by 6296 load acquire from 6297 completing out of 6298 order. The s_waitcnt 6299 could be placed after 6300 seq_store or before 6301 the seq_load. We 6302 choose the load to 6303 make the s_waitcnt be 6304 as late as possible 6305 so that the store 6306 may have already 6307 completed.) 6308 6309 2. *Following 6310 instructions same as 6311 corresponding load 6312 atomic acquire, 6313 except must generate 6314 all instructions even 6315 for OpenCL.* 6316 store atomic seq_cst - singlethread - global *Same as corresponding 6317 - wavefront - local store atomic release, 6318 - workgroup - generic except must generate 6319 - agent all instructions even 6320 - system for OpenCL.* 6321 atomicrmw seq_cst - singlethread - global *Same as corresponding 6322 - wavefront - local atomicrmw acq_rel, 6323 - workgroup - generic except must generate 6324 - agent all instructions even 6325 - system for OpenCL.* 6326 fence seq_cst - singlethread *none* *Same as corresponding 6327 - wavefront fence acq_rel, 6328 - workgroup except must generate 6329 - agent all instructions even 6330 - system for OpenCL.* 6331 ============ ============ ============== ========== ================================ 6332 6333.. _amdgpu-amdhsa-memory-model-gfx90a: 6334 6335Memory Model GFX90A 6336+++++++++++++++++++ 6337 6338For GFX90A: 6339 6340* Each agent has multiple shader arrays (SA). 6341* Each SA has multiple compute units (CU). 6342* Each CU has multiple SIMDs that execute wavefronts. 6343* The wavefronts for a single work-group are executed in the same CU but may be 6344 executed by different SIMDs. The exception is when in tgsplit execution mode 6345 when the wavefronts may be executed by different SIMDs in different CUs. 6346* Each CU has a single LDS memory shared by the wavefronts of the work-groups 6347 executing on it. The exception is when in tgsplit execution mode when no LDS 6348 is allocated as wavefronts of the same work-group can be in different CUs. 6349* All LDS operations of a CU are performed as wavefront wide operations in a 6350 global order and involve no caching. Completion is reported to a wavefront in 6351 execution order. 6352* The LDS memory has multiple request queues shared by the SIMDs of a 6353 CU. Therefore, the LDS operations performed by different wavefronts of a 6354 work-group can be reordered relative to each other, which can result in 6355 reordering the visibility of vector memory operations with respect to LDS 6356 operations of other wavefronts in the same work-group. A ``s_waitcnt 6357 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 6358 vector memory operations between wavefronts of a work-group, but not between 6359 operations performed by the same wavefront. 6360* The vector memory operations are performed as wavefront wide operations and 6361 completion is reported to a wavefront in execution order. The exception is 6362 that ``flat_load/store/atomic`` instructions can report out of vector memory 6363 order if they access LDS memory, and out of LDS operation order if they access 6364 global memory. 6365* The vector memory operations access a single vector L1 cache shared by all 6366 SIMDs a CU. Therefore: 6367 6368 * No special action is required for coherence between the lanes of a single 6369 wavefront. 6370 6371 * No special action is required for coherence between wavefronts in the same 6372 work-group since they execute on the same CU. The exception is when in 6373 tgsplit execution mode as wavefronts of the same work-group can be in 6374 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in 6375 the following item. 6376 6377 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts 6378 executing in different work-groups as they may be executing on different 6379 CUs. 6380 6381* The scalar memory operations access a scalar L1 cache shared by all wavefronts 6382 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 6383 scalar operations are used in a restricted way so do not impact the memory 6384 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 6385* The vector and scalar memory operations use an L2 cache shared by all CUs on 6386 the same agent. 6387 6388 * The L2 cache has independent channels to service disjoint ranges of virtual 6389 addresses. 6390 * Each CU has a separate request queue per channel. Therefore, the vector and 6391 scalar memory operations performed by wavefronts executing in different 6392 work-groups (which may be executing on different CUs), or the same 6393 work-group if executing in tgsplit mode, of an agent can be reordered 6394 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure 6395 synchronization between vector memory operations of different CUs. It 6396 ensures a previous vector memory operation has completed before executing a 6397 subsequent vector memory or LDS operation and so can be used to meet the 6398 requirements of acquire and release. 6399 * The L2 cache of one agent can be kept coherent with other agents by: 6400 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE 6401 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with 6402 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2. 6403 6404 * Any local memory cache lines will be automatically invalidated by writes 6405 from CUs associated with other L2 caches, or writes from the CPU, due to 6406 the cache probe caused by coherent requests. Coherent requests are caused 6407 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over 6408 XGMI, and by PCIe requests that are configured to be coherent requests. 6409 * XGMI accesses from the CPU to local memory may be cached on the CPU. 6410 Subsequent access from the GPU will automatically invalidate or writeback 6411 the CPU cache due to the L2 probe filter and and the PTE C-bit being set. 6412 * Since all work-groups on the same agent share the same L2, no L2 6413 invalidation or writeback is required for coherence. 6414 * To ensure coherence of local and remote memory writes of work-groups in 6415 different agents a ``buffer_wbl2`` is required. It will writeback dirty L2 6416 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC 6417 ()used for remote coarse grain memory). Note that MTYPE CC (used for local 6418 fine grain memory) causes write through to DRAM, and MTYPE UC (used for 6419 remote fine grain memory) bypasses the L2, so both will never result in 6420 dirty L2 cache lines. 6421 * To ensure coherence of local and remote memory reads of work-groups in 6422 different agents a ``buffer_invl2`` is required. It will invalidate L2 6423 cache lines with MTYPE NC (used for remote coarse grain memory). Note that 6424 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local 6425 coarse memory) cause local reads to be invalidated by remote writes with 6426 with the PTE C-bit so these cache lines are not invalidated. Note that 6427 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will 6428 never result in L2 cache lines that need to be invalidated. 6429 6430 * PCIe access from the GPU to the CPU memory is kept coherent by using the 6431 MTYPE UC (uncached) which bypasses the L2. 6432 6433Scalar memory operations are only used to access memory that is proven to not 6434change during the execution of the kernel dispatch. This includes constant 6435address space and global address space for program scope ``const`` variables. 6436Therefore, the kernel machine code does not have to maintain the scalar cache to 6437ensure it is coherent with the vector caches. The scalar and vector caches are 6438invalidated between kernel dispatches by CP since constant address space data 6439may change between kernel dispatch executions. See 6440:ref:`amdgpu-amdhsa-memory-spaces`. 6441 6442The one exception is if scalar writes are used to spill SGPR registers. In this 6443case the AMDGPU backend ensures the memory location used to spill is never 6444accessed by vector memory operations at the same time. If scalar writes are used 6445then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 6446return since the locations may be used for vector memory instructions by a 6447future wavefront that uses the same scratch area, or a function call that 6448creates a frame at the same address, respectively. There is no need for a 6449``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 6450 6451For kernarg backing memory: 6452 6453* CP invalidates the L1 cache at the start of each kernel dispatch. 6454* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 6455 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 6456 cache. This also causes it to be treated as non-volatile and so is not 6457 invalidated by ``*_vol``. 6458* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 6459 so the L2 cache will be coherent with the CPU and other agents. 6460 6461Scratch backing memory (which is used for the private address space) is accessed 6462with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 6463only accessed by a single thread, and is always write-before-read, there is 6464never a need to invalidate these entries from the L1 cache. Hence all cache 6465invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 6466 6467The code sequences used to implement the memory model for GFX90A are defined 6468in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`. 6469 6470 .. table:: AMDHSA Memory Model Code Sequences GFX90A 6471 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table 6472 6473 ============ ============ ============== ========== ================================ 6474 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 6475 Ordering Sync Scope Address GFX90A 6476 Space 6477 ============ ============ ============== ========== ================================ 6478 **Non-Atomic** 6479 ------------------------------------------------------------------------------------ 6480 load *none* *none* - global - !volatile & !nontemporal 6481 - generic 6482 - private 1. buffer/global/flat_load 6483 - constant 6484 - !volatile & nontemporal 6485 6486 1. buffer/global/flat_load 6487 glc=1 slc=1 6488 6489 - volatile 6490 6491 1. buffer/global/flat_load 6492 glc=1 6493 2. s_waitcnt vmcnt(0) 6494 6495 - Must happen before 6496 any following volatile 6497 global/generic 6498 load/store. 6499 - Ensures that 6500 volatile 6501 operations to 6502 different 6503 addresses will not 6504 be reordered by 6505 hardware. 6506 6507 load *none* *none* - local 1. ds_load 6508 store *none* *none* - global - !volatile & !nontemporal 6509 - generic 6510 - private 1. buffer/global/flat_store 6511 - constant 6512 - !volatile & nontemporal 6513 6514 1. buffer/global/flat_store 6515 glc=1 slc=1 6516 6517 - volatile 6518 6519 1. buffer/global/flat_store 6520 2. s_waitcnt vmcnt(0) 6521 6522 - Must happen before 6523 any following volatile 6524 global/generic 6525 load/store. 6526 - Ensures that 6527 volatile 6528 operations to 6529 different 6530 addresses will not 6531 be reordered by 6532 hardware. 6533 6534 store *none* *none* - local 1. ds_store 6535 **Unordered Atomic** 6536 ------------------------------------------------------------------------------------ 6537 load atomic unordered *any* *any* *Same as non-atomic*. 6538 store atomic unordered *any* *any* *Same as non-atomic*. 6539 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 6540 **Monotonic Atomic** 6541 ------------------------------------------------------------------------------------ 6542 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 6543 - wavefront - generic 6544 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 6545 - generic glc=1 6546 6547 - If not TgSplit execution 6548 mode, omit glc=1. 6549 6550 load atomic monotonic - singlethread - local *If TgSplit execution mode, 6551 - wavefront local address space cannot 6552 - workgroup be used.* 6553 6554 1. ds_load 6555 load atomic monotonic - agent - global 1. buffer/global/flat_load 6556 - generic glc=1 6557 load atomic monotonic - system - global 1. buffer/global/flat_load 6558 - generic glc=1 6559 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 6560 - wavefront - generic 6561 - workgroup 6562 - agent 6563 store atomic monotonic - system - global 1. buffer/global/flat_store 6564 - generic 6565 store atomic monotonic - singlethread - local *If TgSplit execution mode, 6566 - wavefront local address space cannot 6567 - workgroup be used.* 6568 6569 1. ds_store 6570 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 6571 - wavefront - generic 6572 - workgroup 6573 - agent 6574 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 6575 - generic 6576 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 6577 - wavefront local address space cannot 6578 - workgroup be used.* 6579 6580 1. ds_atomic 6581 **Acquire Atomic** 6582 ------------------------------------------------------------------------------------ 6583 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 6584 - wavefront - local 6585 - generic 6586 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 6587 6588 - If not TgSplit execution 6589 mode, omit glc=1. 6590 6591 2. s_waitcnt vmcnt(0) 6592 6593 - If not TgSplit execution 6594 mode, omit. 6595 - Must happen before the 6596 following buffer_wbinvl1_vol. 6597 6598 3. buffer_wbinvl1_vol 6599 6600 - If not TgSplit execution 6601 mode, omit. 6602 - Must happen before 6603 any following 6604 global/generic 6605 load/load 6606 atomic/store/store 6607 atomic/atomicrmw. 6608 - Ensures that 6609 following 6610 loads will not see 6611 stale data. 6612 6613 load atomic acquire - workgroup - local *If TgSplit execution mode, 6614 local address space cannot 6615 be used.* 6616 6617 1. ds_load 6618 2. s_waitcnt lgkmcnt(0) 6619 6620 - If OpenCL, omit. 6621 - Must happen before 6622 any following 6623 global/generic 6624 load/load 6625 atomic/store/store 6626 atomic/atomicrmw. 6627 - Ensures any 6628 following global 6629 data read is no 6630 older than the local load 6631 atomic value being 6632 acquired. 6633 6634 load atomic acquire - workgroup - generic 1. flat_load glc=1 6635 6636 - If not TgSplit execution 6637 mode, omit glc=1. 6638 6639 2. s_waitcnt lgkm/vmcnt(0) 6640 6641 - Use lgkmcnt(0) if not 6642 TgSplit execution mode 6643 and vmcnt(0) if TgSplit 6644 execution mode. 6645 - If OpenCL, omit lgkmcnt(0). 6646 - Must happen before 6647 the following 6648 buffer_wbinvl1_vol and any 6649 following global/generic 6650 load/load 6651 atomic/store/store 6652 atomic/atomicrmw. 6653 - Ensures any 6654 following global 6655 data read is no 6656 older than a local load 6657 atomic value being 6658 acquired. 6659 6660 3. buffer_wbinvl1_vol 6661 6662 - If not TgSplit execution 6663 mode, omit. 6664 - Ensures that 6665 following 6666 loads will not see 6667 stale data. 6668 6669 load atomic acquire - agent - global 1. buffer/global_load 6670 glc=1 6671 2. s_waitcnt vmcnt(0) 6672 6673 - Must happen before 6674 following 6675 buffer_wbinvl1_vol. 6676 - Ensures the load 6677 has completed 6678 before invalidating 6679 the cache. 6680 6681 3. buffer_wbinvl1_vol 6682 6683 - Must happen before 6684 any following 6685 global/generic 6686 load/load 6687 atomic/atomicrmw. 6688 - Ensures that 6689 following 6690 loads will not see 6691 stale global data. 6692 6693 load atomic acquire - system - global 1. buffer/global/flat_load 6694 glc=1 6695 2. s_waitcnt vmcnt(0) 6696 6697 - Must happen before 6698 following buffer_invl2 and 6699 buffer_wbinvl1_vol. 6700 - Ensures the load 6701 has completed 6702 before invalidating 6703 the cache. 6704 6705 3. buffer_invl2; 6706 buffer_wbinvl1_vol 6707 6708 - Must happen before 6709 any following 6710 global/generic 6711 load/load 6712 atomic/atomicrmw. 6713 - Ensures that 6714 following 6715 loads will not see 6716 stale L1 global data, 6717 nor see stale L2 MTYPE 6718 NC global data. 6719 MTYPE RW and CC memory will 6720 never be stale in L2 due to 6721 the memory probes. 6722 6723 load atomic acquire - agent - generic 1. flat_load glc=1 6724 2. s_waitcnt vmcnt(0) & 6725 lgkmcnt(0) 6726 6727 - If TgSplit execution mode, 6728 omit lgkmcnt(0). 6729 - If OpenCL omit 6730 lgkmcnt(0). 6731 - Must happen before 6732 following 6733 buffer_wbinvl1_vol. 6734 - Ensures the flat_load 6735 has completed 6736 before invalidating 6737 the cache. 6738 6739 3. buffer_wbinvl1_vol 6740 6741 - Must happen before 6742 any following 6743 global/generic 6744 load/load 6745 atomic/atomicrmw. 6746 - Ensures that 6747 following loads 6748 will not see stale 6749 global data. 6750 6751 load atomic acquire - system - generic 1. flat_load glc=1 6752 2. s_waitcnt vmcnt(0) & 6753 lgkmcnt(0) 6754 6755 - If TgSplit execution mode, 6756 omit lgkmcnt(0). 6757 - If OpenCL omit 6758 lgkmcnt(0). 6759 - Must happen before 6760 following 6761 buffer_invl2 and 6762 buffer_wbinvl1_vol. 6763 - Ensures the flat_load 6764 has completed 6765 before invalidating 6766 the caches. 6767 6768 3. buffer_invl2; 6769 buffer_wbinvl1_vol 6770 6771 - Must happen before 6772 any following 6773 global/generic 6774 load/load 6775 atomic/atomicrmw. 6776 - Ensures that 6777 following 6778 loads will not see 6779 stale L1 global data, 6780 nor see stale L2 MTYPE 6781 NC global data. 6782 MTYPE RW and CC memory will 6783 never be stale in L2 due to 6784 the memory probes. 6785 6786 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 6787 - wavefront - generic 6788 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 6789 - wavefront local address space cannot 6790 be used.* 6791 6792 1. ds_atomic 6793 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 6794 2. s_waitcnt vmcnt(0) 6795 6796 - If not TgSplit execution 6797 mode, omit. 6798 - Must happen before the 6799 following buffer_wbinvl1_vol. 6800 - Ensures the atomicrmw 6801 has completed 6802 before invalidating 6803 the cache. 6804 6805 3. buffer_wbinvl1_vol 6806 6807 - If not TgSplit execution 6808 mode, omit. 6809 - Must happen before 6810 any following 6811 global/generic 6812 load/load 6813 atomic/atomicrmw. 6814 - Ensures that 6815 following loads 6816 will not see stale 6817 global data. 6818 6819 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 6820 local address space cannot 6821 be used.* 6822 6823 1. ds_atomic 6824 2. s_waitcnt lgkmcnt(0) 6825 6826 - If OpenCL, omit. 6827 - Must happen before 6828 any following 6829 global/generic 6830 load/load 6831 atomic/store/store 6832 atomic/atomicrmw. 6833 - Ensures any 6834 following global 6835 data read is no 6836 older than the local 6837 atomicrmw value 6838 being acquired. 6839 6840 atomicrmw acquire - workgroup - generic 1. flat_atomic 6841 2. s_waitcnt lgkm/vmcnt(0) 6842 6843 - Use lgkmcnt(0) if not 6844 TgSplit execution mode 6845 and vmcnt(0) if TgSplit 6846 execution mode. 6847 - If OpenCL, omit lgkmcnt(0). 6848 - Must happen before 6849 the following 6850 buffer_wbinvl1_vol and 6851 any following 6852 global/generic 6853 load/load 6854 atomic/store/store 6855 atomic/atomicrmw. 6856 - Ensures any 6857 following global 6858 data read is no 6859 older than a local 6860 atomicrmw value 6861 being acquired. 6862 6863 3. buffer_wbinvl1_vol 6864 6865 - If not TgSplit execution 6866 mode, omit. 6867 - Ensures that 6868 following 6869 loads will not see 6870 stale data. 6871 6872 atomicrmw acquire - agent - global 1. buffer/global_atomic 6873 2. s_waitcnt vmcnt(0) 6874 6875 - Must happen before 6876 following 6877 buffer_wbinvl1_vol. 6878 - Ensures the 6879 atomicrmw has 6880 completed before 6881 invalidating the 6882 cache. 6883 6884 3. buffer_wbinvl1_vol 6885 6886 - Must happen before 6887 any following 6888 global/generic 6889 load/load 6890 atomic/atomicrmw. 6891 - Ensures that 6892 following loads 6893 will not see stale 6894 global data. 6895 6896 atomicrmw acquire - system - global 1. buffer/global_atomic 6897 2. s_waitcnt vmcnt(0) 6898 6899 - Must happen before 6900 following buffer_invl2 and 6901 buffer_wbinvl1_vol. 6902 - Ensures the 6903 atomicrmw has 6904 completed before 6905 invalidating the 6906 caches. 6907 6908 3. buffer_invl2; 6909 buffer_wbinvl1_vol 6910 6911 - Must happen before 6912 any following 6913 global/generic 6914 load/load 6915 atomic/atomicrmw. 6916 - Ensures that 6917 following 6918 loads will not see 6919 stale L1 global data, 6920 nor see stale L2 MTYPE 6921 NC global data. 6922 MTYPE RW and CC memory will 6923 never be stale in L2 due to 6924 the memory probes. 6925 6926 atomicrmw acquire - agent - generic 1. flat_atomic 6927 2. s_waitcnt vmcnt(0) & 6928 lgkmcnt(0) 6929 6930 - If TgSplit execution mode, 6931 omit lgkmcnt(0). 6932 - If OpenCL, omit 6933 lgkmcnt(0). 6934 - Must happen before 6935 following 6936 buffer_wbinvl1_vol. 6937 - Ensures the 6938 atomicrmw has 6939 completed before 6940 invalidating the 6941 cache. 6942 6943 3. buffer_wbinvl1_vol 6944 6945 - Must happen before 6946 any following 6947 global/generic 6948 load/load 6949 atomic/atomicrmw. 6950 - Ensures that 6951 following loads 6952 will not see stale 6953 global data. 6954 6955 atomicrmw acquire - system - generic 1. flat_atomic 6956 2. s_waitcnt vmcnt(0) & 6957 lgkmcnt(0) 6958 6959 - If TgSplit execution mode, 6960 omit lgkmcnt(0). 6961 - If OpenCL, omit 6962 lgkmcnt(0). 6963 - Must happen before 6964 following 6965 buffer_invl2 and 6966 buffer_wbinvl1_vol. 6967 - Ensures the 6968 atomicrmw has 6969 completed before 6970 invalidating the 6971 caches. 6972 6973 3. buffer_invl2; 6974 buffer_wbinvl1_vol 6975 6976 - Must happen before 6977 any following 6978 global/generic 6979 load/load 6980 atomic/atomicrmw. 6981 - Ensures that 6982 following 6983 loads will not see 6984 stale L1 global data, 6985 nor see stale L2 MTYPE 6986 NC global data. 6987 MTYPE RW and CC memory will 6988 never be stale in L2 due to 6989 the memory probes. 6990 6991 fence acquire - singlethread *none* *none* 6992 - wavefront 6993 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 6994 6995 - Use lgkmcnt(0) if not 6996 TgSplit execution mode 6997 and vmcnt(0) if TgSplit 6998 execution mode. 6999 - If OpenCL and 7000 address space is 7001 not generic, omit 7002 lgkmcnt(0). 7003 - If OpenCL and 7004 address space is 7005 local, omit 7006 vmcnt(0). 7007 - However, since LLVM 7008 currently has no 7009 address space on 7010 the fence need to 7011 conservatively 7012 always generate. If 7013 fence had an 7014 address space then 7015 set to address 7016 space of OpenCL 7017 fence flag, or to 7018 generic if both 7019 local and global 7020 flags are 7021 specified. 7022 - s_waitcnt vmcnt(0) 7023 must happen after 7024 any preceding 7025 global/generic load 7026 atomic/ 7027 atomicrmw 7028 with an equal or 7029 wider sync scope 7030 and memory ordering 7031 stronger than 7032 unordered (this is 7033 termed the 7034 fence-paired-atomic). 7035 - s_waitcnt lgkmcnt(0) 7036 must happen after 7037 any preceding 7038 local/generic load 7039 atomic/atomicrmw 7040 with an equal or 7041 wider sync scope 7042 and memory ordering 7043 stronger than 7044 unordered (this is 7045 termed the 7046 fence-paired-atomic). 7047 - Must happen before 7048 the following 7049 buffer_wbinvl1_vol and 7050 any following 7051 global/generic 7052 load/load 7053 atomic/store/store 7054 atomic/atomicrmw. 7055 - Ensures any 7056 following global 7057 data read is no 7058 older than the 7059 value read by the 7060 fence-paired-atomic. 7061 7062 2. buffer_wbinvl1_vol 7063 7064 - If not TgSplit execution 7065 mode, omit. 7066 - Ensures that 7067 following 7068 loads will not see 7069 stale data. 7070 7071 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 7072 vmcnt(0) 7073 7074 - If TgSplit execution mode, 7075 omit lgkmcnt(0). 7076 - If OpenCL and 7077 address space is 7078 not generic, omit 7079 lgkmcnt(0). 7080 - However, since LLVM 7081 currently has no 7082 address space on 7083 the fence need to 7084 conservatively 7085 always generate 7086 (see comment for 7087 previous fence). 7088 - Could be split into 7089 separate s_waitcnt 7090 vmcnt(0) and 7091 s_waitcnt 7092 lgkmcnt(0) to allow 7093 them to be 7094 independently moved 7095 according to the 7096 following rules. 7097 - s_waitcnt vmcnt(0) 7098 must happen after 7099 any preceding 7100 global/generic load 7101 atomic/atomicrmw 7102 with an equal or 7103 wider sync scope 7104 and memory ordering 7105 stronger than 7106 unordered (this is 7107 termed the 7108 fence-paired-atomic). 7109 - s_waitcnt lgkmcnt(0) 7110 must happen after 7111 any preceding 7112 local/generic load 7113 atomic/atomicrmw 7114 with an equal or 7115 wider sync scope 7116 and memory ordering 7117 stronger than 7118 unordered (this is 7119 termed the 7120 fence-paired-atomic). 7121 - Must happen before 7122 the following 7123 buffer_wbinvl1_vol. 7124 - Ensures that the 7125 fence-paired atomic 7126 has completed 7127 before invalidating 7128 the 7129 cache. Therefore 7130 any following 7131 locations read must 7132 be no older than 7133 the value read by 7134 the 7135 fence-paired-atomic. 7136 7137 2. buffer_wbinvl1_vol 7138 7139 - Must happen before any 7140 following global/generic 7141 load/load 7142 atomic/store/store 7143 atomic/atomicrmw. 7144 - Ensures that 7145 following loads 7146 will not see stale 7147 global data. 7148 7149 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 7150 vmcnt(0) 7151 7152 - If TgSplit execution mode, 7153 omit lgkmcnt(0). 7154 - If OpenCL and 7155 address space is 7156 not generic, omit 7157 lgkmcnt(0). 7158 - However, since LLVM 7159 currently has no 7160 address space on 7161 the fence need to 7162 conservatively 7163 always generate 7164 (see comment for 7165 previous fence). 7166 - Could be split into 7167 separate s_waitcnt 7168 vmcnt(0) and 7169 s_waitcnt 7170 lgkmcnt(0) to allow 7171 them to be 7172 independently moved 7173 according to the 7174 following rules. 7175 - s_waitcnt vmcnt(0) 7176 must happen after 7177 any preceding 7178 global/generic load 7179 atomic/atomicrmw 7180 with an equal or 7181 wider sync scope 7182 and memory ordering 7183 stronger than 7184 unordered (this is 7185 termed the 7186 fence-paired-atomic). 7187 - s_waitcnt lgkmcnt(0) 7188 must happen after 7189 any preceding 7190 local/generic load 7191 atomic/atomicrmw 7192 with an equal or 7193 wider sync scope 7194 and memory ordering 7195 stronger than 7196 unordered (this is 7197 termed the 7198 fence-paired-atomic). 7199 - Must happen before 7200 the following buffer_invl2 and 7201 buffer_wbinvl1_vol. 7202 - Ensures that the 7203 fence-paired atomic 7204 has completed 7205 before invalidating 7206 the 7207 cache. Therefore 7208 any following 7209 locations read must 7210 be no older than 7211 the value read by 7212 the 7213 fence-paired-atomic. 7214 7215 2. buffer_invl2; 7216 buffer_wbinvl1_vol 7217 7218 - Must happen before any 7219 following global/generic 7220 load/load 7221 atomic/store/store 7222 atomic/atomicrmw. 7223 - Ensures that 7224 following 7225 loads will not see 7226 stale L1 global data, 7227 nor see stale L2 MTYPE 7228 NC global data. 7229 MTYPE RW and CC memory will 7230 never be stale in L2 due to 7231 the memory probes. 7232 **Release Atomic** 7233 ------------------------------------------------------------------------------------ 7234 store atomic release - singlethread - global 1. buffer/global/flat_store 7235 - wavefront - generic 7236 store atomic release - singlethread - local *If TgSplit execution mode, 7237 - wavefront local address space cannot 7238 be used.* 7239 7240 1. ds_store 7241 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7242 - generic 7243 - Use lgkmcnt(0) if not 7244 TgSplit execution mode 7245 and vmcnt(0) if TgSplit 7246 execution mode. 7247 - If OpenCL, omit lgkmcnt(0). 7248 - s_waitcnt vmcnt(0) 7249 must happen after 7250 any preceding 7251 global/generic load/store/ 7252 load atomic/store atomic/ 7253 atomicrmw. 7254 - s_waitcnt lgkmcnt(0) 7255 must happen after 7256 any preceding 7257 local/generic 7258 load/store/load 7259 atomic/store 7260 atomic/atomicrmw. 7261 - Must happen before 7262 the following 7263 store. 7264 - Ensures that all 7265 memory operations 7266 have 7267 completed before 7268 performing the 7269 store that is being 7270 released. 7271 7272 2. buffer/global/flat_store 7273 store atomic release - workgroup - local *If TgSplit execution mode, 7274 local address space cannot 7275 be used.* 7276 7277 1. ds_store 7278 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 7279 - generic vmcnt(0) 7280 7281 - If TgSplit execution mode, 7282 omit lgkmcnt(0). 7283 - If OpenCL and 7284 address space is 7285 not generic, omit 7286 lgkmcnt(0). 7287 - Could be split into 7288 separate s_waitcnt 7289 vmcnt(0) and 7290 s_waitcnt 7291 lgkmcnt(0) to allow 7292 them to be 7293 independently moved 7294 according to the 7295 following rules. 7296 - s_waitcnt vmcnt(0) 7297 must happen after 7298 any preceding 7299 global/generic 7300 load/store/load 7301 atomic/store 7302 atomic/atomicrmw. 7303 - s_waitcnt lgkmcnt(0) 7304 must happen after 7305 any preceding 7306 local/generic 7307 load/store/load 7308 atomic/store 7309 atomic/atomicrmw. 7310 - Must happen before 7311 the following 7312 store. 7313 - Ensures that all 7314 memory operations 7315 to memory have 7316 completed before 7317 performing the 7318 store that is being 7319 released. 7320 7321 2. buffer/global/flat_store 7322 store atomic release - system - global 1. buffer_wbl2 7323 - generic 7324 - Must happen before 7325 following s_waitcnt. 7326 - Performs L2 writeback to 7327 ensure previous 7328 global/generic 7329 store/atomicrmw are 7330 visible at system scope. 7331 7332 2. s_waitcnt lgkmcnt(0) & 7333 vmcnt(0) 7334 7335 - If TgSplit execution mode, 7336 omit lgkmcnt(0). 7337 - If OpenCL and 7338 address space is 7339 not generic, omit 7340 lgkmcnt(0). 7341 - Could be split into 7342 separate s_waitcnt 7343 vmcnt(0) and 7344 s_waitcnt 7345 lgkmcnt(0) to allow 7346 them to be 7347 independently moved 7348 according to the 7349 following rules. 7350 - s_waitcnt vmcnt(0) 7351 must happen after any 7352 preceding 7353 global/generic 7354 load/store/load 7355 atomic/store 7356 atomic/atomicrmw. 7357 - s_waitcnt lgkmcnt(0) 7358 must happen after any 7359 preceding 7360 local/generic 7361 load/store/load 7362 atomic/store 7363 atomic/atomicrmw. 7364 - Must happen before 7365 the following 7366 store. 7367 - Ensures that all 7368 memory operations 7369 to memory and the L2 7370 writeback have 7371 completed before 7372 performing the 7373 store that is being 7374 released. 7375 7376 3. buffer/global/flat_store 7377 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 7378 - wavefront - generic 7379 atomicrmw release - singlethread - local *If TgSplit execution mode, 7380 - wavefront local address space cannot 7381 be used.* 7382 7383 1. ds_atomic 7384 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7385 - generic 7386 - Use lgkmcnt(0) if not 7387 TgSplit execution mode 7388 and vmcnt(0) if TgSplit 7389 execution mode. 7390 - If OpenCL, omit 7391 lgkmcnt(0). 7392 - s_waitcnt vmcnt(0) 7393 must happen after 7394 any preceding 7395 global/generic load/store/ 7396 load atomic/store atomic/ 7397 atomicrmw. 7398 - s_waitcnt lgkmcnt(0) 7399 must happen after 7400 any preceding 7401 local/generic 7402 load/store/load 7403 atomic/store 7404 atomic/atomicrmw. 7405 - Must happen before 7406 the following 7407 atomicrmw. 7408 - Ensures that all 7409 memory operations 7410 have 7411 completed before 7412 performing the 7413 atomicrmw that is 7414 being released. 7415 7416 2. buffer/global/flat_atomic 7417 atomicrmw release - workgroup - local *If TgSplit execution mode, 7418 local address space cannot 7419 be used.* 7420 7421 1. ds_atomic 7422 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 7423 - generic vmcnt(0) 7424 7425 - If TgSplit execution mode, 7426 omit lgkmcnt(0). 7427 - If OpenCL, omit 7428 lgkmcnt(0). 7429 - Could be split into 7430 separate s_waitcnt 7431 vmcnt(0) and 7432 s_waitcnt 7433 lgkmcnt(0) to allow 7434 them to be 7435 independently moved 7436 according to the 7437 following rules. 7438 - s_waitcnt vmcnt(0) 7439 must happen after 7440 any preceding 7441 global/generic 7442 load/store/load 7443 atomic/store 7444 atomic/atomicrmw. 7445 - s_waitcnt lgkmcnt(0) 7446 must happen after 7447 any preceding 7448 local/generic 7449 load/store/load 7450 atomic/store 7451 atomic/atomicrmw. 7452 - Must happen before 7453 the following 7454 atomicrmw. 7455 - Ensures that all 7456 memory operations 7457 to global and local 7458 have completed 7459 before performing 7460 the atomicrmw that 7461 is being released. 7462 7463 2. buffer/global/flat_atomic 7464 atomicrmw release - system - global 1. buffer_wbl2 7465 - generic 7466 - Must happen before 7467 following s_waitcnt. 7468 - Performs L2 writeback to 7469 ensure previous 7470 global/generic 7471 store/atomicrmw are 7472 visible at system scope. 7473 7474 2. s_waitcnt lgkmcnt(0) & 7475 vmcnt(0) 7476 7477 - If TgSplit execution mode, 7478 omit lgkmcnt(0). 7479 - If OpenCL, omit 7480 lgkmcnt(0). 7481 - Could be split into 7482 separate s_waitcnt 7483 vmcnt(0) and 7484 s_waitcnt 7485 lgkmcnt(0) to allow 7486 them to be 7487 independently moved 7488 according to the 7489 following rules. 7490 - s_waitcnt vmcnt(0) 7491 must happen after 7492 any preceding 7493 global/generic 7494 load/store/load 7495 atomic/store 7496 atomic/atomicrmw. 7497 - s_waitcnt lgkmcnt(0) 7498 must happen after 7499 any preceding 7500 local/generic 7501 load/store/load 7502 atomic/store 7503 atomic/atomicrmw. 7504 - Must happen before 7505 the following 7506 atomicrmw. 7507 - Ensures that all 7508 memory operations 7509 to memory and the L2 7510 writeback have 7511 completed before 7512 performing the 7513 store that is being 7514 released. 7515 7516 3. buffer/global/flat_atomic 7517 fence release - singlethread *none* *none* 7518 - wavefront 7519 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 7520 7521 - Use lgkmcnt(0) if not 7522 TgSplit execution mode 7523 and vmcnt(0) if TgSplit 7524 execution mode. 7525 - If OpenCL and 7526 address space is 7527 not generic, omit 7528 lgkmcnt(0). 7529 - If OpenCL and 7530 address space is 7531 local, omit 7532 vmcnt(0). 7533 - However, since LLVM 7534 currently has no 7535 address space on 7536 the fence need to 7537 conservatively 7538 always generate. If 7539 fence had an 7540 address space then 7541 set to address 7542 space of OpenCL 7543 fence flag, or to 7544 generic if both 7545 local and global 7546 flags are 7547 specified. 7548 - s_waitcnt vmcnt(0) 7549 must happen after 7550 any preceding 7551 global/generic 7552 load/store/ 7553 load atomic/store atomic/ 7554 atomicrmw. 7555 - s_waitcnt lgkmcnt(0) 7556 must happen after 7557 any preceding 7558 local/generic 7559 load/load 7560 atomic/store/store 7561 atomic/atomicrmw. 7562 - Must happen before 7563 any following store 7564 atomic/atomicrmw 7565 with an equal or 7566 wider sync scope 7567 and memory ordering 7568 stronger than 7569 unordered (this is 7570 termed the 7571 fence-paired-atomic). 7572 - Ensures that all 7573 memory operations 7574 have 7575 completed before 7576 performing the 7577 following 7578 fence-paired-atomic. 7579 7580 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 7581 vmcnt(0) 7582 7583 - If TgSplit execution mode, 7584 omit lgkmcnt(0). 7585 - If OpenCL and 7586 address space is 7587 not generic, omit 7588 lgkmcnt(0). 7589 - If OpenCL and 7590 address space is 7591 local, omit 7592 vmcnt(0). 7593 - However, since LLVM 7594 currently has no 7595 address space on 7596 the fence need to 7597 conservatively 7598 always generate. If 7599 fence had an 7600 address space then 7601 set to address 7602 space of OpenCL 7603 fence flag, or to 7604 generic if both 7605 local and global 7606 flags are 7607 specified. 7608 - Could be split into 7609 separate s_waitcnt 7610 vmcnt(0) and 7611 s_waitcnt 7612 lgkmcnt(0) to allow 7613 them to be 7614 independently moved 7615 according to the 7616 following rules. 7617 - s_waitcnt vmcnt(0) 7618 must happen after 7619 any preceding 7620 global/generic 7621 load/store/load 7622 atomic/store 7623 atomic/atomicrmw. 7624 - s_waitcnt lgkmcnt(0) 7625 must happen after 7626 any preceding 7627 local/generic 7628 load/store/load 7629 atomic/store 7630 atomic/atomicrmw. 7631 - Must happen before 7632 any following store 7633 atomic/atomicrmw 7634 with an equal or 7635 wider sync scope 7636 and memory ordering 7637 stronger than 7638 unordered (this is 7639 termed the 7640 fence-paired-atomic). 7641 - Ensures that all 7642 memory operations 7643 have 7644 completed before 7645 performing the 7646 following 7647 fence-paired-atomic. 7648 7649 fence release - system *none* 1. buffer_wbl2 7650 7651 - If OpenCL and 7652 address space is 7653 local, omit. 7654 - Must happen before 7655 following s_waitcnt. 7656 - Performs L2 writeback to 7657 ensure previous 7658 global/generic 7659 store/atomicrmw are 7660 visible at system scope. 7661 7662 2. s_waitcnt lgkmcnt(0) & 7663 vmcnt(0) 7664 7665 - If TgSplit execution mode, 7666 omit lgkmcnt(0). 7667 - If OpenCL and 7668 address space is 7669 not generic, omit 7670 lgkmcnt(0). 7671 - If OpenCL and 7672 address space is 7673 local, omit 7674 vmcnt(0). 7675 - However, since LLVM 7676 currently has no 7677 address space on 7678 the fence need to 7679 conservatively 7680 always generate. If 7681 fence had an 7682 address space then 7683 set to address 7684 space of OpenCL 7685 fence flag, or to 7686 generic if both 7687 local and global 7688 flags are 7689 specified. 7690 - Could be split into 7691 separate s_waitcnt 7692 vmcnt(0) and 7693 s_waitcnt 7694 lgkmcnt(0) to allow 7695 them to be 7696 independently moved 7697 according to the 7698 following rules. 7699 - s_waitcnt vmcnt(0) 7700 must happen after 7701 any preceding 7702 global/generic 7703 load/store/load 7704 atomic/store 7705 atomic/atomicrmw. 7706 - s_waitcnt lgkmcnt(0) 7707 must happen after 7708 any preceding 7709 local/generic 7710 load/store/load 7711 atomic/store 7712 atomic/atomicrmw. 7713 - Must happen before 7714 any following store 7715 atomic/atomicrmw 7716 with an equal or 7717 wider sync scope 7718 and memory ordering 7719 stronger than 7720 unordered (this is 7721 termed the 7722 fence-paired-atomic). 7723 - Ensures that all 7724 memory operations 7725 have 7726 completed before 7727 performing the 7728 following 7729 fence-paired-atomic. 7730 7731 **Acquire-Release Atomic** 7732 ------------------------------------------------------------------------------------ 7733 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 7734 - wavefront - generic 7735 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 7736 - wavefront local address space cannot 7737 be used.* 7738 7739 1. ds_atomic 7740 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 7741 7742 - Use lgkmcnt(0) if not 7743 TgSplit execution mode 7744 and vmcnt(0) if TgSplit 7745 execution mode. 7746 - If OpenCL, omit 7747 lgkmcnt(0). 7748 - Must happen after 7749 any preceding 7750 local/generic 7751 load/store/load 7752 atomic/store 7753 atomic/atomicrmw. 7754 - s_waitcnt vmcnt(0) 7755 must happen after 7756 any preceding 7757 global/generic load/store/ 7758 load atomic/store atomic/ 7759 atomicrmw. 7760 - s_waitcnt lgkmcnt(0) 7761 must happen after 7762 any preceding 7763 local/generic 7764 load/store/load 7765 atomic/store 7766 atomic/atomicrmw. 7767 - Must happen before 7768 the following 7769 atomicrmw. 7770 - Ensures that all 7771 memory operations 7772 have 7773 completed before 7774 performing the 7775 atomicrmw that is 7776 being released. 7777 7778 2. buffer/global_atomic 7779 3. s_waitcnt vmcnt(0) 7780 7781 - If not TgSplit execution 7782 mode, omit. 7783 - Must happen before 7784 the following 7785 buffer_wbinvl1_vol. 7786 - Ensures any 7787 following global 7788 data read is no 7789 older than the 7790 atomicrmw value 7791 being acquired. 7792 7793 4. buffer_wbinvl1_vol 7794 7795 - If not TgSplit execution 7796 mode, omit. 7797 - Ensures that 7798 following 7799 loads will not see 7800 stale data. 7801 7802 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 7803 local address space cannot 7804 be used.* 7805 7806 1. ds_atomic 7807 2. s_waitcnt lgkmcnt(0) 7808 7809 - If OpenCL, omit. 7810 - Must happen before 7811 any following 7812 global/generic 7813 load/load 7814 atomic/store/store 7815 atomic/atomicrmw. 7816 - Ensures any 7817 following global 7818 data read is no 7819 older than the local load 7820 atomic value being 7821 acquired. 7822 7823 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 7824 7825 - Use lgkmcnt(0) if not 7826 TgSplit execution mode 7827 and vmcnt(0) if TgSplit 7828 execution mode. 7829 - If OpenCL, omit 7830 lgkmcnt(0). 7831 - s_waitcnt vmcnt(0) 7832 must happen after 7833 any preceding 7834 global/generic load/store/ 7835 load atomic/store atomic/ 7836 atomicrmw. 7837 - s_waitcnt lgkmcnt(0) 7838 must happen after 7839 any preceding 7840 local/generic 7841 load/store/load 7842 atomic/store 7843 atomic/atomicrmw. 7844 - Must happen before 7845 the following 7846 atomicrmw. 7847 - Ensures that all 7848 memory operations 7849 have 7850 completed before 7851 performing the 7852 atomicrmw that is 7853 being released. 7854 7855 2. flat_atomic 7856 3. s_waitcnt lgkmcnt(0) & 7857 vmcnt(0) 7858 7859 - If not TgSplit execution 7860 mode, omit vmcnt(0). 7861 - If OpenCL, omit 7862 lgkmcnt(0). 7863 - Must happen before 7864 the following 7865 buffer_wbinvl1_vol and 7866 any following 7867 global/generic 7868 load/load 7869 atomic/store/store 7870 atomic/atomicrmw. 7871 - Ensures any 7872 following global 7873 data read is no 7874 older than a local load 7875 atomic value being 7876 acquired. 7877 7878 3. buffer_wbinvl1_vol 7879 7880 - If not TgSplit execution 7881 mode, omit. 7882 - Ensures that 7883 following 7884 loads will not see 7885 stale data. 7886 7887 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 7888 vmcnt(0) 7889 7890 - If TgSplit execution mode, 7891 omit lgkmcnt(0). 7892 - If OpenCL, omit 7893 lgkmcnt(0). 7894 - Could be split into 7895 separate s_waitcnt 7896 vmcnt(0) and 7897 s_waitcnt 7898 lgkmcnt(0) to allow 7899 them to be 7900 independently moved 7901 according to the 7902 following rules. 7903 - s_waitcnt vmcnt(0) 7904 must happen after 7905 any preceding 7906 global/generic 7907 load/store/load 7908 atomic/store 7909 atomic/atomicrmw. 7910 - s_waitcnt lgkmcnt(0) 7911 must happen after 7912 any preceding 7913 local/generic 7914 load/store/load 7915 atomic/store 7916 atomic/atomicrmw. 7917 - Must happen before 7918 the following 7919 atomicrmw. 7920 - Ensures that all 7921 memory operations 7922 to global have 7923 completed before 7924 performing the 7925 atomicrmw that is 7926 being released. 7927 7928 2. buffer/global_atomic 7929 3. s_waitcnt vmcnt(0) 7930 7931 - Must happen before 7932 following 7933 buffer_wbinvl1_vol. 7934 - Ensures the 7935 atomicrmw has 7936 completed before 7937 invalidating the 7938 cache. 7939 7940 4. buffer_wbinvl1_vol 7941 7942 - Must happen before 7943 any following 7944 global/generic 7945 load/load 7946 atomic/atomicrmw. 7947 - Ensures that 7948 following loads 7949 will not see stale 7950 global data. 7951 7952 atomicrmw acq_rel - system - global 1. buffer_wbl2 7953 7954 - Must happen before 7955 following s_waitcnt. 7956 - Performs L2 writeback to 7957 ensure previous 7958 global/generic 7959 store/atomicrmw are 7960 visible at system scope. 7961 7962 2. s_waitcnt lgkmcnt(0) & 7963 vmcnt(0) 7964 7965 - If TgSplit execution mode, 7966 omit lgkmcnt(0). 7967 - If OpenCL, omit 7968 lgkmcnt(0). 7969 - Could be split into 7970 separate s_waitcnt 7971 vmcnt(0) and 7972 s_waitcnt 7973 lgkmcnt(0) to allow 7974 them to be 7975 independently moved 7976 according to the 7977 following rules. 7978 - s_waitcnt vmcnt(0) 7979 must happen after 7980 any preceding 7981 global/generic 7982 load/store/load 7983 atomic/store 7984 atomic/atomicrmw. 7985 - s_waitcnt lgkmcnt(0) 7986 must happen after 7987 any preceding 7988 local/generic 7989 load/store/load 7990 atomic/store 7991 atomic/atomicrmw. 7992 - Must happen before 7993 the following 7994 atomicrmw. 7995 - Ensures that all 7996 memory operations 7997 to global and L2 writeback 7998 have completed before 7999 performing the 8000 atomicrmw that is 8001 being released. 8002 8003 3. buffer/global_atomic 8004 4. s_waitcnt vmcnt(0) 8005 8006 - Must happen before 8007 following buffer_invl2 and 8008 buffer_wbinvl1_vol. 8009 - Ensures the 8010 atomicrmw has 8011 completed before 8012 invalidating the 8013 caches. 8014 8015 5. buffer_invl2; 8016 buffer_wbinvl1_vol 8017 8018 - Must happen before 8019 any following 8020 global/generic 8021 load/load 8022 atomic/atomicrmw. 8023 - Ensures that 8024 following 8025 loads will not see 8026 stale L1 global data, 8027 nor see stale L2 MTYPE 8028 NC global data. 8029 MTYPE RW and CC memory will 8030 never be stale in L2 due to 8031 the memory probes. 8032 8033 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 8034 vmcnt(0) 8035 8036 - If TgSplit execution mode, 8037 omit lgkmcnt(0). 8038 - If OpenCL, omit 8039 lgkmcnt(0). 8040 - Could be split into 8041 separate s_waitcnt 8042 vmcnt(0) and 8043 s_waitcnt 8044 lgkmcnt(0) to allow 8045 them to be 8046 independently moved 8047 according to the 8048 following rules. 8049 - s_waitcnt vmcnt(0) 8050 must happen after 8051 any preceding 8052 global/generic 8053 load/store/load 8054 atomic/store 8055 atomic/atomicrmw. 8056 - s_waitcnt lgkmcnt(0) 8057 must happen after 8058 any preceding 8059 local/generic 8060 load/store/load 8061 atomic/store 8062 atomic/atomicrmw. 8063 - Must happen before 8064 the following 8065 atomicrmw. 8066 - Ensures that all 8067 memory operations 8068 to global have 8069 completed before 8070 performing the 8071 atomicrmw that is 8072 being released. 8073 8074 2. flat_atomic 8075 3. s_waitcnt vmcnt(0) & 8076 lgkmcnt(0) 8077 8078 - If TgSplit execution mode, 8079 omit lgkmcnt(0). 8080 - If OpenCL, omit 8081 lgkmcnt(0). 8082 - Must happen before 8083 following 8084 buffer_wbinvl1_vol. 8085 - Ensures the 8086 atomicrmw has 8087 completed before 8088 invalidating the 8089 cache. 8090 8091 4. buffer_wbinvl1_vol 8092 8093 - Must happen before 8094 any following 8095 global/generic 8096 load/load 8097 atomic/atomicrmw. 8098 - Ensures that 8099 following loads 8100 will not see stale 8101 global data. 8102 8103 atomicrmw acq_rel - system - generic 1. buffer_wbl2 8104 8105 - Must happen before 8106 following s_waitcnt. 8107 - Performs L2 writeback to 8108 ensure previous 8109 global/generic 8110 store/atomicrmw are 8111 visible at system scope. 8112 8113 2. s_waitcnt lgkmcnt(0) & 8114 vmcnt(0) 8115 8116 - If TgSplit execution mode, 8117 omit lgkmcnt(0). 8118 - If OpenCL, omit 8119 lgkmcnt(0). 8120 - Could be split into 8121 separate s_waitcnt 8122 vmcnt(0) and 8123 s_waitcnt 8124 lgkmcnt(0) to allow 8125 them to be 8126 independently moved 8127 according to the 8128 following rules. 8129 - s_waitcnt vmcnt(0) 8130 must happen after 8131 any preceding 8132 global/generic 8133 load/store/load 8134 atomic/store 8135 atomic/atomicrmw. 8136 - s_waitcnt lgkmcnt(0) 8137 must happen after 8138 any preceding 8139 local/generic 8140 load/store/load 8141 atomic/store 8142 atomic/atomicrmw. 8143 - Must happen before 8144 the following 8145 atomicrmw. 8146 - Ensures that all 8147 memory operations 8148 to global and L2 writeback 8149 have completed before 8150 performing the 8151 atomicrmw that is 8152 being released. 8153 8154 3. flat_atomic 8155 4. s_waitcnt vmcnt(0) & 8156 lgkmcnt(0) 8157 8158 - If TgSplit execution mode, 8159 omit lgkmcnt(0). 8160 - If OpenCL, omit 8161 lgkmcnt(0). 8162 - Must happen before 8163 following buffer_invl2 and 8164 buffer_wbinvl1_vol. 8165 - Ensures the 8166 atomicrmw has 8167 completed before 8168 invalidating the 8169 caches. 8170 8171 5. buffer_invl2; 8172 buffer_wbinvl1_vol 8173 8174 - Must happen before 8175 any following 8176 global/generic 8177 load/load 8178 atomic/atomicrmw. 8179 - Ensures that 8180 following 8181 loads will not see 8182 stale L1 global data, 8183 nor see stale L2 MTYPE 8184 NC global data. 8185 MTYPE RW and CC memory will 8186 never be stale in L2 due to 8187 the memory probes. 8188 8189 fence acq_rel - singlethread *none* *none* 8190 - wavefront 8191 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 8192 8193 - Use lgkmcnt(0) if not 8194 TgSplit execution mode 8195 and vmcnt(0) if TgSplit 8196 execution mode. 8197 - If OpenCL and 8198 address space is 8199 not generic, omit 8200 lgkmcnt(0). 8201 - If OpenCL and 8202 address space is 8203 local, omit 8204 vmcnt(0). 8205 - However, 8206 since LLVM 8207 currently has no 8208 address space on 8209 the fence need to 8210 conservatively 8211 always generate 8212 (see comment for 8213 previous fence). 8214 - s_waitcnt vmcnt(0) 8215 must happen after 8216 any preceding 8217 global/generic 8218 load/store/ 8219 load atomic/store atomic/ 8220 atomicrmw. 8221 - s_waitcnt lgkmcnt(0) 8222 must happen after 8223 any preceding 8224 local/generic 8225 load/load 8226 atomic/store/store 8227 atomic/atomicrmw. 8228 - Must happen before 8229 any following 8230 global/generic 8231 load/load 8232 atomic/store/store 8233 atomic/atomicrmw. 8234 - Ensures that all 8235 memory operations 8236 have 8237 completed before 8238 performing any 8239 following global 8240 memory operations. 8241 - Ensures that the 8242 preceding 8243 local/generic load 8244 atomic/atomicrmw 8245 with an equal or 8246 wider sync scope 8247 and memory ordering 8248 stronger than 8249 unordered (this is 8250 termed the 8251 acquire-fence-paired-atomic) 8252 has completed 8253 before following 8254 global memory 8255 operations. This 8256 satisfies the 8257 requirements of 8258 acquire. 8259 - Ensures that all 8260 previous memory 8261 operations have 8262 completed before a 8263 following 8264 local/generic store 8265 atomic/atomicrmw 8266 with an equal or 8267 wider sync scope 8268 and memory ordering 8269 stronger than 8270 unordered (this is 8271 termed the 8272 release-fence-paired-atomic). 8273 This satisfies the 8274 requirements of 8275 release. 8276 - Must happen before 8277 the following 8278 buffer_wbinvl1_vol. 8279 - Ensures that the 8280 acquire-fence-paired 8281 atomic has completed 8282 before invalidating 8283 the 8284 cache. Therefore 8285 any following 8286 locations read must 8287 be no older than 8288 the value read by 8289 the 8290 acquire-fence-paired-atomic. 8291 8292 2. buffer_wbinvl1_vol 8293 8294 - If not TgSplit execution 8295 mode, omit. 8296 - Ensures that 8297 following 8298 loads will not see 8299 stale data. 8300 8301 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 8302 vmcnt(0) 8303 8304 - If TgSplit execution mode, 8305 omit lgkmcnt(0). 8306 - If OpenCL and 8307 address space is 8308 not generic, omit 8309 lgkmcnt(0). 8310 - However, since LLVM 8311 currently has no 8312 address space on 8313 the fence need to 8314 conservatively 8315 always generate 8316 (see comment for 8317 previous fence). 8318 - Could be split into 8319 separate s_waitcnt 8320 vmcnt(0) and 8321 s_waitcnt 8322 lgkmcnt(0) to allow 8323 them to be 8324 independently moved 8325 according to the 8326 following rules. 8327 - s_waitcnt vmcnt(0) 8328 must happen after 8329 any preceding 8330 global/generic 8331 load/store/load 8332 atomic/store 8333 atomic/atomicrmw. 8334 - s_waitcnt lgkmcnt(0) 8335 must happen after 8336 any preceding 8337 local/generic 8338 load/store/load 8339 atomic/store 8340 atomic/atomicrmw. 8341 - Must happen before 8342 the following 8343 buffer_wbinvl1_vol. 8344 - Ensures that the 8345 preceding 8346 global/local/generic 8347 load 8348 atomic/atomicrmw 8349 with an equal or 8350 wider sync scope 8351 and memory ordering 8352 stronger than 8353 unordered (this is 8354 termed the 8355 acquire-fence-paired-atomic) 8356 has completed 8357 before invalidating 8358 the cache. This 8359 satisfies the 8360 requirements of 8361 acquire. 8362 - Ensures that all 8363 previous memory 8364 operations have 8365 completed before a 8366 following 8367 global/local/generic 8368 store 8369 atomic/atomicrmw 8370 with an equal or 8371 wider sync scope 8372 and memory ordering 8373 stronger than 8374 unordered (this is 8375 termed the 8376 release-fence-paired-atomic). 8377 This satisfies the 8378 requirements of 8379 release. 8380 8381 2. buffer_wbinvl1_vol 8382 8383 - Must happen before 8384 any following 8385 global/generic 8386 load/load 8387 atomic/store/store 8388 atomic/atomicrmw. 8389 - Ensures that 8390 following loads 8391 will not see stale 8392 global data. This 8393 satisfies the 8394 requirements of 8395 acquire. 8396 8397 fence acq_rel - system *none* 1. buffer_wbl2 8398 8399 - If OpenCL and 8400 address space is 8401 local, omit. 8402 - Must happen before 8403 following s_waitcnt. 8404 - Performs L2 writeback to 8405 ensure previous 8406 global/generic 8407 store/atomicrmw are 8408 visible at system scope. 8409 8410 2. s_waitcnt lgkmcnt(0) & 8411 vmcnt(0) 8412 8413 - If TgSplit execution mode, 8414 omit lgkmcnt(0). 8415 - If OpenCL and 8416 address space is 8417 not generic, omit 8418 lgkmcnt(0). 8419 - However, since LLVM 8420 currently has no 8421 address space on 8422 the fence need to 8423 conservatively 8424 always generate 8425 (see comment for 8426 previous fence). 8427 - Could be split into 8428 separate s_waitcnt 8429 vmcnt(0) and 8430 s_waitcnt 8431 lgkmcnt(0) to allow 8432 them to be 8433 independently moved 8434 according to the 8435 following rules. 8436 - s_waitcnt vmcnt(0) 8437 must happen after 8438 any preceding 8439 global/generic 8440 load/store/load 8441 atomic/store 8442 atomic/atomicrmw. 8443 - s_waitcnt lgkmcnt(0) 8444 must happen after 8445 any preceding 8446 local/generic 8447 load/store/load 8448 atomic/store 8449 atomic/atomicrmw. 8450 - Must happen before 8451 the following buffer_invl2 and 8452 buffer_wbinvl1_vol. 8453 - Ensures that the 8454 preceding 8455 global/local/generic 8456 load 8457 atomic/atomicrmw 8458 with an equal or 8459 wider sync scope 8460 and memory ordering 8461 stronger than 8462 unordered (this is 8463 termed the 8464 acquire-fence-paired-atomic) 8465 has completed 8466 before invalidating 8467 the cache. This 8468 satisfies the 8469 requirements of 8470 acquire. 8471 - Ensures that all 8472 previous memory 8473 operations have 8474 completed before a 8475 following 8476 global/local/generic 8477 store 8478 atomic/atomicrmw 8479 with an equal or 8480 wider sync scope 8481 and memory ordering 8482 stronger than 8483 unordered (this is 8484 termed the 8485 release-fence-paired-atomic). 8486 This satisfies the 8487 requirements of 8488 release. 8489 8490 3. buffer_invl2; 8491 buffer_wbinvl1_vol 8492 8493 - Must happen before 8494 any following 8495 global/generic 8496 load/load 8497 atomic/store/store 8498 atomic/atomicrmw. 8499 - Ensures that 8500 following 8501 loads will not see 8502 stale L1 global data, 8503 nor see stale L2 MTYPE 8504 NC global data. 8505 MTYPE RW and CC memory will 8506 never be stale in L2 due to 8507 the memory probes. 8508 8509 **Sequential Consistent Atomic** 8510 ------------------------------------------------------------------------------------ 8511 load atomic seq_cst - singlethread - global *Same as corresponding 8512 - wavefront - local load atomic acquire, 8513 - generic except must generate 8514 all instructions even 8515 for OpenCL.* 8516 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 8517 - generic 8518 - Use lgkmcnt(0) if not 8519 TgSplit execution mode 8520 and vmcnt(0) if TgSplit 8521 execution mode. 8522 - s_waitcnt lgkmcnt(0) must 8523 happen after 8524 preceding 8525 local/generic load 8526 atomic/store 8527 atomic/atomicrmw 8528 with memory 8529 ordering of seq_cst 8530 and with equal or 8531 wider sync scope. 8532 (Note that seq_cst 8533 fences have their 8534 own s_waitcnt 8535 lgkmcnt(0) and so do 8536 not need to be 8537 considered.) 8538 - s_waitcnt vmcnt(0) 8539 must happen after 8540 preceding 8541 global/generic load 8542 atomic/store 8543 atomic/atomicrmw 8544 with memory 8545 ordering of seq_cst 8546 and with equal or 8547 wider sync scope. 8548 (Note that seq_cst 8549 fences have their 8550 own s_waitcnt 8551 vmcnt(0) and so do 8552 not need to be 8553 considered.) 8554 - Ensures any 8555 preceding 8556 sequential 8557 consistent global/local 8558 memory instructions 8559 have completed 8560 before executing 8561 this sequentially 8562 consistent 8563 instruction. This 8564 prevents reordering 8565 a seq_cst store 8566 followed by a 8567 seq_cst load. (Note 8568 that seq_cst is 8569 stronger than 8570 acquire/release as 8571 the reordering of 8572 load acquire 8573 followed by a store 8574 release is 8575 prevented by the 8576 s_waitcnt of 8577 the release, but 8578 there is nothing 8579 preventing a store 8580 release followed by 8581 load acquire from 8582 completing out of 8583 order. The s_waitcnt 8584 could be placed after 8585 seq_store or before 8586 the seq_load. We 8587 choose the load to 8588 make the s_waitcnt be 8589 as late as possible 8590 so that the store 8591 may have already 8592 completed.) 8593 8594 2. *Following 8595 instructions same as 8596 corresponding load 8597 atomic acquire, 8598 except must generate 8599 all instructions even 8600 for OpenCL.* 8601 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 8602 local address space cannot 8603 be used.* 8604 8605 *Same as corresponding 8606 load atomic acquire, 8607 except must generate 8608 all instructions even 8609 for OpenCL.* 8610 8611 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 8612 - system - generic vmcnt(0) 8613 8614 - If TgSplit execution mode, 8615 omit lgkmcnt(0). 8616 - Could be split into 8617 separate s_waitcnt 8618 vmcnt(0) 8619 and s_waitcnt 8620 lgkmcnt(0) to allow 8621 them to be 8622 independently moved 8623 according to the 8624 following rules. 8625 - s_waitcnt lgkmcnt(0) 8626 must happen after 8627 preceding 8628 global/generic load 8629 atomic/store 8630 atomic/atomicrmw 8631 with memory 8632 ordering of seq_cst 8633 and with equal or 8634 wider sync scope. 8635 (Note that seq_cst 8636 fences have their 8637 own s_waitcnt 8638 lgkmcnt(0) and so do 8639 not need to be 8640 considered.) 8641 - s_waitcnt vmcnt(0) 8642 must happen after 8643 preceding 8644 global/generic load 8645 atomic/store 8646 atomic/atomicrmw 8647 with memory 8648 ordering of seq_cst 8649 and with equal or 8650 wider sync scope. 8651 (Note that seq_cst 8652 fences have their 8653 own s_waitcnt 8654 vmcnt(0) and so do 8655 not need to be 8656 considered.) 8657 - Ensures any 8658 preceding 8659 sequential 8660 consistent global 8661 memory instructions 8662 have completed 8663 before executing 8664 this sequentially 8665 consistent 8666 instruction. This 8667 prevents reordering 8668 a seq_cst store 8669 followed by a 8670 seq_cst load. (Note 8671 that seq_cst is 8672 stronger than 8673 acquire/release as 8674 the reordering of 8675 load acquire 8676 followed by a store 8677 release is 8678 prevented by the 8679 s_waitcnt of 8680 the release, but 8681 there is nothing 8682 preventing a store 8683 release followed by 8684 load acquire from 8685 completing out of 8686 order. The s_waitcnt 8687 could be placed after 8688 seq_store or before 8689 the seq_load. We 8690 choose the load to 8691 make the s_waitcnt be 8692 as late as possible 8693 so that the store 8694 may have already 8695 completed.) 8696 8697 2. *Following 8698 instructions same as 8699 corresponding load 8700 atomic acquire, 8701 except must generate 8702 all instructions even 8703 for OpenCL.* 8704 store atomic seq_cst - singlethread - global *Same as corresponding 8705 - wavefront - local store atomic release, 8706 - workgroup - generic except must generate 8707 - agent all instructions even 8708 - system for OpenCL.* 8709 atomicrmw seq_cst - singlethread - global *Same as corresponding 8710 - wavefront - local atomicrmw acq_rel, 8711 - workgroup - generic except must generate 8712 - agent all instructions even 8713 - system for OpenCL.* 8714 fence seq_cst - singlethread *none* *Same as corresponding 8715 - wavefront fence acq_rel, 8716 - workgroup except must generate 8717 - agent all instructions even 8718 - system for OpenCL.* 8719 ============ ============ ============== ========== ================================ 8720 8721.. _amdgpu-amdhsa-memory-model-gfx940: 8722 8723Memory Model GFX940 8724+++++++++++++++++++ 8725 8726For GFX940: 8727 8728* Each agent has multiple shader arrays (SA). 8729* Each SA has multiple compute units (CU). 8730* Each CU has multiple SIMDs that execute wavefronts. 8731* The wavefronts for a single work-group are executed in the same CU but may be 8732 executed by different SIMDs. The exception is when in tgsplit execution mode 8733 when the wavefronts may be executed by different SIMDs in different CUs. 8734* Each CU has a single LDS memory shared by the wavefronts of the work-groups 8735 executing on it. The exception is when in tgsplit execution mode when no LDS 8736 is allocated as wavefronts of the same work-group can be in different CUs. 8737* All LDS operations of a CU are performed as wavefront wide operations in a 8738 global order and involve no caching. Completion is reported to a wavefront in 8739 execution order. 8740* The LDS memory has multiple request queues shared by the SIMDs of a 8741 CU. Therefore, the LDS operations performed by different wavefronts of a 8742 work-group can be reordered relative to each other, which can result in 8743 reordering the visibility of vector memory operations with respect to LDS 8744 operations of other wavefronts in the same work-group. A ``s_waitcnt 8745 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 8746 vector memory operations between wavefronts of a work-group, but not between 8747 operations performed by the same wavefront. 8748* The vector memory operations are performed as wavefront wide operations and 8749 completion is reported to a wavefront in execution order. The exception is 8750 that ``flat_load/store/atomic`` instructions can report out of vector memory 8751 order if they access LDS memory, and out of LDS operation order if they access 8752 global memory. 8753* The vector memory operations access a single vector L1 cache shared by all 8754 SIMDs a CU. Therefore: 8755 8756 * No special action is required for coherence between the lanes of a single 8757 wavefront. 8758 8759 * No special action is required for coherence between wavefronts in the same 8760 work-group since they execute on the same CU. The exception is when in 8761 tgsplit execution mode as wavefronts of the same work-group can be in 8762 different CUs and so a ``buffer_inv sc0`` is required which will invalidate 8763 the L1 cache. 8764 8765 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence 8766 between wavefronts executing in different work-groups as they may be 8767 executing on different CUs. 8768 8769 * Atomic read-modify-write instructions implicitly bypass the L1 cache. 8770 Therefore, they do not use the sc0 bit for coherence and instead use it to 8771 indicate if the instruction returns the original value being updated. They 8772 do use sc1 to indicate system or agent scope coherence. 8773 8774* The scalar memory operations access a scalar L1 cache shared by all wavefronts 8775 on a group of CUs. The scalar and vector L1 caches are not coherent. However, 8776 scalar operations are used in a restricted way so do not impact the memory 8777 model. See :ref:`amdgpu-amdhsa-memory-spaces`. 8778* The vector and scalar memory operations use an L2 cache. 8779 8780 * The gfx940 can be configured as a number of smaller agents with each having 8781 a single L2 shared by all CUs on the same agent, or as fewer (possibly one) 8782 larger agents with groups of CUs on each agent each sharing separate L2 8783 caches. 8784 * The L2 cache has independent channels to service disjoint ranges of virtual 8785 addresses. 8786 * Each CU has a separate request queue per channel for its associated L2. 8787 Therefore, the vector and scalar memory operations performed by wavefronts 8788 executing with different L1 caches and the same L2 cache can be reordered 8789 relative to each other. 8790 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between 8791 vector memory operations of different CUs. It ensures a previous vector 8792 memory operation has completed before executing a subsequent vector memory 8793 or LDS operation and so can be used to meet the requirements of acquire and 8794 release. 8795 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW 8796 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with 8797 the PTE C-bit set for memory not local to the L2. 8798 8799 * Any local memory cache lines will be automatically invalidated by writes 8800 from CUs associated with other L2 caches, or writes from the CPU, due to 8801 the cache probe caused by the PTE C-bit. 8802 * XGMI accesses from the CPU to local memory may be cached on the CPU. 8803 Subsequent access from the GPU will automatically invalidate or writeback 8804 the CPU cache due to the L2 probe filter. 8805 * To ensure coherence of local memory writes of CUs with different L1 caches 8806 in the same agent a ``buffer_wbl2`` is required. It does nothing if the 8807 agent is configured to have a single L2, or will writeback dirty L2 cache 8808 lines if configured to have multiple L2 caches. 8809 * To ensure coherence of local memory writes of CUs in different agents a 8810 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines. 8811 * To ensure coherence of local memory reads of CUs with different L1 caches 8812 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the 8813 agent is configured to have a single L2, or will invalidate non-local L2 8814 cache lines if configured to have multiple L2 caches. 8815 * To ensure coherence of local memory reads of CUs in different agents a 8816 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache 8817 lines if configured to have multiple L2 caches. 8818 8819 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE 8820 UC (uncached) which bypasses the L2. 8821 8822Scalar memory operations are only used to access memory that is proven to not 8823change during the execution of the kernel dispatch. This includes constant 8824address space and global address space for program scope ``const`` variables. 8825Therefore, the kernel machine code does not have to maintain the scalar cache to 8826ensure it is coherent with the vector caches. The scalar and vector caches are 8827invalidated between kernel dispatches by CP since constant address space data 8828may change between kernel dispatch executions. See 8829:ref:`amdgpu-amdhsa-memory-spaces`. 8830 8831The one exception is if scalar writes are used to spill SGPR registers. In this 8832case the AMDGPU backend ensures the memory location used to spill is never 8833accessed by vector memory operations at the same time. If scalar writes are used 8834then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 8835return since the locations may be used for vector memory instructions by a 8836future wavefront that uses the same scratch area, or a function call that 8837creates a frame at the same address, respectively. There is no need for a 8838``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 8839 8840For kernarg backing memory: 8841 8842* CP invalidates the L1 cache at the start of each kernel dispatch. 8843* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host 8844 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L2 8845 cache. This also causes it to be treated as non-volatile and so is not 8846 invalidated by ``*_vol``. 8847* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 8848 so the L2 cache will be coherent with the CPU and other agents. 8849 8850Scratch backing memory (which is used for the private address space) is accessed 8851with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is 8852only accessed by a single thread, and is always write-before-read, there is 8853never a need to invalidate these entries from the L1 cache. Hence all cache 8854invalidates are done as ``*_vol`` to only invalidate the volatile cache lines. 8855 8856The code sequences used to implement the memory model for GFX940 are defined 8857in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx940-table`. 8858 8859 .. table:: AMDHSA Memory Model Code Sequences GFX940 8860 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx940-table 8861 8862 ============ ============ ============== ========== ================================ 8863 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 8864 Ordering Sync Scope Address GFX940 8865 Space 8866 ============ ============ ============== ========== ================================ 8867 **Non-Atomic** 8868 ------------------------------------------------------------------------------------ 8869 load *none* *none* - global - !volatile & !nontemporal 8870 - generic 8871 - private 1. buffer/global/flat_load 8872 - constant 8873 - !volatile & nontemporal 8874 8875 1. buffer/global/flat_load 8876 nt=1 8877 8878 - volatile 8879 8880 1. buffer/global/flat_load 8881 sc0=1 sc1=1 8882 2. s_waitcnt vmcnt(0) 8883 8884 - Must happen before 8885 any following volatile 8886 global/generic 8887 load/store. 8888 - Ensures that 8889 volatile 8890 operations to 8891 different 8892 addresses will not 8893 be reordered by 8894 hardware. 8895 8896 load *none* *none* - local 1. ds_load 8897 store *none* *none* - global - !volatile & !nontemporal 8898 - generic 8899 - private 1. buffer/global/flat_store 8900 - constant 8901 - !volatile & nontemporal 8902 8903 1. buffer/global/flat_store 8904 nt=1 8905 8906 - volatile 8907 8908 1. buffer/global/flat_store 8909 sc0=1 sc1=1 8910 2. s_waitcnt vmcnt(0) 8911 8912 - Must happen before 8913 any following volatile 8914 global/generic 8915 load/store. 8916 - Ensures that 8917 volatile 8918 operations to 8919 different 8920 addresses will not 8921 be reordered by 8922 hardware. 8923 8924 store *none* *none* - local 1. ds_store 8925 **Unordered Atomic** 8926 ------------------------------------------------------------------------------------ 8927 load atomic unordered *any* *any* *Same as non-atomic*. 8928 store atomic unordered *any* *any* *Same as non-atomic*. 8929 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 8930 **Monotonic Atomic** 8931 ------------------------------------------------------------------------------------ 8932 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 8933 - wavefront - generic 8934 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 8935 - generic sc0=1 8936 load atomic monotonic - singlethread - local *If TgSplit execution mode, 8937 - wavefront local address space cannot 8938 - workgroup be used.* 8939 8940 1. ds_load 8941 load atomic monotonic - agent - global 1. buffer/global/flat_load 8942 - generic sc1=1 8943 load atomic monotonic - system - global 1. buffer/global/flat_load 8944 - generic sc0=1 sc1=1 8945 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 8946 - wavefront - generic 8947 store atomic monotonic - workgroup - global 1. buffer/global/flat_store 8948 - generic sc0=1 8949 store atomic monotonic - agent - global 1. buffer/global/flat_store 8950 - generic sc1=1 8951 store atomic monotonic - system - global 1. buffer/global/flat_store 8952 - generic sc0=1 sc1=1 8953 store atomic monotonic - singlethread - local *If TgSplit execution mode, 8954 - wavefront local address space cannot 8955 - workgroup be used.* 8956 8957 1. ds_store 8958 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 8959 - wavefront - generic 8960 - workgroup 8961 - agent 8962 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic 8963 - generic sc1=1 8964 atomicrmw monotonic - singlethread - local *If TgSplit execution mode, 8965 - wavefront local address space cannot 8966 - workgroup be used.* 8967 8968 1. ds_atomic 8969 **Acquire Atomic** 8970 ------------------------------------------------------------------------------------ 8971 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 8972 - wavefront - local 8973 - generic 8974 load atomic acquire - workgroup - global 1. buffer/global_load sc0=1 8975 2. s_waitcnt vmcnt(0) 8976 8977 - If not TgSplit execution 8978 mode, omit. 8979 - Must happen before the 8980 following buffer_inv. 8981 8982 3. buffer_inv sc0=1 8983 8984 - If not TgSplit execution 8985 mode, omit. 8986 - Must happen before 8987 any following 8988 global/generic 8989 load/load 8990 atomic/store/store 8991 atomic/atomicrmw. 8992 - Ensures that 8993 following 8994 loads will not see 8995 stale data. 8996 8997 load atomic acquire - workgroup - local *If TgSplit execution mode, 8998 local address space cannot 8999 be used.* 9000 9001 1. ds_load 9002 2. s_waitcnt lgkmcnt(0) 9003 9004 - If OpenCL, omit. 9005 - Must happen before 9006 any following 9007 global/generic 9008 load/load 9009 atomic/store/store 9010 atomic/atomicrmw. 9011 - Ensures any 9012 following global 9013 data read is no 9014 older than the local load 9015 atomic value being 9016 acquired. 9017 9018 load atomic acquire - workgroup - generic 1. flat_load sc0=1 9019 2. s_waitcnt lgkm/vmcnt(0) 9020 9021 - Use lgkmcnt(0) if not 9022 TgSplit execution mode 9023 and vmcnt(0) if TgSplit 9024 execution mode. 9025 - If OpenCL, omit lgkmcnt(0). 9026 - Must happen before 9027 the following 9028 buffer_inv and any 9029 following global/generic 9030 load/load 9031 atomic/store/store 9032 atomic/atomicrmw. 9033 - Ensures any 9034 following global 9035 data read is no 9036 older than a local load 9037 atomic value being 9038 acquired. 9039 9040 3. buffer_inv sc0=1 9041 9042 - If not TgSplit execution 9043 mode, omit. 9044 - Ensures that 9045 following 9046 loads will not see 9047 stale data. 9048 9049 load atomic acquire - agent - global 1. buffer/global_load 9050 sc1=1 9051 2. s_waitcnt vmcnt(0) 9052 9053 - Must happen before 9054 following 9055 buffer_inv. 9056 - Ensures the load 9057 has completed 9058 before invalidating 9059 the cache. 9060 9061 3. buffer_inv sc1=1 9062 9063 - Must happen before 9064 any following 9065 global/generic 9066 load/load 9067 atomic/atomicrmw. 9068 - Ensures that 9069 following 9070 loads will not see 9071 stale global data. 9072 9073 load atomic acquire - system - global 1. buffer/global/flat_load 9074 sc0=1 sc1=1 9075 2. s_waitcnt vmcnt(0) 9076 9077 - Must happen before 9078 following 9079 buffer_inv. 9080 - Ensures the load 9081 has completed 9082 before invalidating 9083 the cache. 9084 9085 3. buffer_inv sc0=1 sc1=1 9086 9087 - Must happen before 9088 any following 9089 global/generic 9090 load/load 9091 atomic/atomicrmw. 9092 - Ensures that 9093 following 9094 loads will not see 9095 stale MTYPE NC global data. 9096 MTYPE RW and CC memory will 9097 never be stale due to the 9098 memory probes. 9099 9100 load atomic acquire - agent - generic 1. flat_load sc1=1 9101 2. s_waitcnt vmcnt(0) & 9102 lgkmcnt(0) 9103 9104 - If TgSplit execution mode, 9105 omit lgkmcnt(0). 9106 - If OpenCL omit 9107 lgkmcnt(0). 9108 - Must happen before 9109 following 9110 buffer_inv. 9111 - Ensures the flat_load 9112 has completed 9113 before invalidating 9114 the cache. 9115 9116 3. buffer_inv sc1=1 9117 9118 - Must happen before 9119 any following 9120 global/generic 9121 load/load 9122 atomic/atomicrmw. 9123 - Ensures that 9124 following loads 9125 will not see stale 9126 global data. 9127 9128 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=1 9129 2. s_waitcnt vmcnt(0) & 9130 lgkmcnt(0) 9131 9132 - If TgSplit execution mode, 9133 omit lgkmcnt(0). 9134 - If OpenCL omit 9135 lgkmcnt(0). 9136 - Must happen before 9137 the following 9138 buffer_inv. 9139 - Ensures the flat_load 9140 has completed 9141 before invalidating 9142 the caches. 9143 9144 3. buffer_inv sc0=1 sc1=1 9145 9146 - Must happen before 9147 any following 9148 global/generic 9149 load/load 9150 atomic/atomicrmw. 9151 - Ensures that 9152 following 9153 loads will not see 9154 stale MTYPE NC global data. 9155 MTYPE RW and CC memory will 9156 never be stale due to the 9157 memory probes. 9158 9159 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic 9160 - wavefront - generic 9161 atomicrmw acquire - singlethread - local *If TgSplit execution mode, 9162 - wavefront local address space cannot 9163 be used.* 9164 9165 1. ds_atomic 9166 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 9167 2. s_waitcnt vmcnt(0) 9168 9169 - If not TgSplit execution 9170 mode, omit. 9171 - Must happen before the 9172 following buffer_inv. 9173 - Ensures the atomicrmw 9174 has completed 9175 before invalidating 9176 the cache. 9177 9178 3. buffer_inv sc0=1 9179 9180 - If not TgSplit execution 9181 mode, omit. 9182 - Must happen before 9183 any following 9184 global/generic 9185 load/load 9186 atomic/atomicrmw. 9187 - Ensures that 9188 following loads 9189 will not see stale 9190 global data. 9191 9192 atomicrmw acquire - workgroup - local *If TgSplit execution mode, 9193 local address space cannot 9194 be used.* 9195 9196 1. ds_atomic 9197 2. s_waitcnt lgkmcnt(0) 9198 9199 - If OpenCL, omit. 9200 - Must happen before 9201 any following 9202 global/generic 9203 load/load 9204 atomic/store/store 9205 atomic/atomicrmw. 9206 - Ensures any 9207 following global 9208 data read is no 9209 older than the local 9210 atomicrmw value 9211 being acquired. 9212 9213 atomicrmw acquire - workgroup - generic 1. flat_atomic 9214 2. s_waitcnt lgkm/vmcnt(0) 9215 9216 - Use lgkmcnt(0) if not 9217 TgSplit execution mode 9218 and vmcnt(0) if TgSplit 9219 execution mode. 9220 - If OpenCL, omit lgkmcnt(0). 9221 - Must happen before 9222 the following 9223 buffer_inv and 9224 any following 9225 global/generic 9226 load/load 9227 atomic/store/store 9228 atomic/atomicrmw. 9229 - Ensures any 9230 following global 9231 data read is no 9232 older than a local 9233 atomicrmw value 9234 being acquired. 9235 9236 3. buffer_inv sc0=1 9237 9238 - If not TgSplit execution 9239 mode, omit. 9240 - Ensures that 9241 following 9242 loads will not see 9243 stale data. 9244 9245 atomicrmw acquire - agent - global 1. buffer/global_atomic 9246 2. s_waitcnt vmcnt(0) 9247 9248 - Must happen before 9249 following 9250 buffer_inv. 9251 - Ensures the 9252 atomicrmw has 9253 completed before 9254 invalidating the 9255 cache. 9256 9257 3. buffer_inv sc1=1 9258 9259 - Must happen before 9260 any following 9261 global/generic 9262 load/load 9263 atomic/atomicrmw. 9264 - Ensures that 9265 following loads 9266 will not see stale 9267 global data. 9268 9269 atomicrmw acquire - system - global 1. buffer/global_atomic 9270 sc1=1 9271 2. s_waitcnt vmcnt(0) 9272 9273 - Must happen before 9274 following 9275 buffer_inv. 9276 - Ensures the 9277 atomicrmw has 9278 completed before 9279 invalidating the 9280 caches. 9281 9282 3. buffer_inv sc0=1 sc1=1 9283 9284 - Must happen before 9285 any following 9286 global/generic 9287 load/load 9288 atomic/atomicrmw. 9289 - Ensures that 9290 following 9291 loads will not see 9292 stale MTYPE NC global data. 9293 MTYPE RW and CC memory will 9294 never be stale due to the 9295 memory probes. 9296 9297 atomicrmw acquire - agent - generic 1. flat_atomic 9298 2. s_waitcnt vmcnt(0) & 9299 lgkmcnt(0) 9300 9301 - If TgSplit execution mode, 9302 omit lgkmcnt(0). 9303 - If OpenCL, omit 9304 lgkmcnt(0). 9305 - Must happen before 9306 following 9307 buffer_inv. 9308 - Ensures the 9309 atomicrmw has 9310 completed before 9311 invalidating the 9312 cache. 9313 9314 3. buffer_inv sc1=1 9315 9316 - Must happen before 9317 any following 9318 global/generic 9319 load/load 9320 atomic/atomicrmw. 9321 - Ensures that 9322 following loads 9323 will not see stale 9324 global data. 9325 9326 atomicrmw acquire - system - generic 1. flat_atomic sc1=1 9327 2. s_waitcnt vmcnt(0) & 9328 lgkmcnt(0) 9329 9330 - If TgSplit execution mode, 9331 omit lgkmcnt(0). 9332 - If OpenCL, omit 9333 lgkmcnt(0). 9334 - Must happen before 9335 following 9336 buffer_inv. 9337 - Ensures the 9338 atomicrmw has 9339 completed before 9340 invalidating the 9341 caches. 9342 9343 3. buffer_inv sc0=1 sc1=1 9344 9345 - Must happen before 9346 any following 9347 global/generic 9348 load/load 9349 atomic/atomicrmw. 9350 - Ensures that 9351 following 9352 loads will not see 9353 stale MTYPE NC global data. 9354 MTYPE RW and CC memory will 9355 never be stale due to the 9356 memory probes. 9357 9358 fence acquire - singlethread *none* *none* 9359 - wavefront 9360 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9361 9362 - Use lgkmcnt(0) if not 9363 TgSplit execution mode 9364 and vmcnt(0) if TgSplit 9365 execution mode. 9366 - If OpenCL and 9367 address space is 9368 not generic, omit 9369 lgkmcnt(0). 9370 - If OpenCL and 9371 address space is 9372 local, omit 9373 vmcnt(0). 9374 - However, since LLVM 9375 currently has no 9376 address space on 9377 the fence need to 9378 conservatively 9379 always generate. If 9380 fence had an 9381 address space then 9382 set to address 9383 space of OpenCL 9384 fence flag, or to 9385 generic if both 9386 local and global 9387 flags are 9388 specified. 9389 - s_waitcnt vmcnt(0) 9390 must happen after 9391 any preceding 9392 global/generic load 9393 atomic/ 9394 atomicrmw 9395 with an equal or 9396 wider sync scope 9397 and memory ordering 9398 stronger than 9399 unordered (this is 9400 termed the 9401 fence-paired-atomic). 9402 - s_waitcnt lgkmcnt(0) 9403 must happen after 9404 any preceding 9405 local/generic load 9406 atomic/atomicrmw 9407 with an equal or 9408 wider sync scope 9409 and memory ordering 9410 stronger than 9411 unordered (this is 9412 termed the 9413 fence-paired-atomic). 9414 - Must happen before 9415 the following 9416 buffer_inv and 9417 any following 9418 global/generic 9419 load/load 9420 atomic/store/store 9421 atomic/atomicrmw. 9422 - Ensures any 9423 following global 9424 data read is no 9425 older than the 9426 value read by the 9427 fence-paired-atomic. 9428 9429 3. buffer_inv sc0=1 9430 9431 - If not TgSplit execution 9432 mode, omit. 9433 - Ensures that 9434 following 9435 loads will not see 9436 stale data. 9437 9438 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 9439 vmcnt(0) 9440 9441 - If TgSplit execution mode, 9442 omit lgkmcnt(0). 9443 - If OpenCL and 9444 address space is 9445 not generic, omit 9446 lgkmcnt(0). 9447 - However, since LLVM 9448 currently has no 9449 address space on 9450 the fence need to 9451 conservatively 9452 always generate 9453 (see comment for 9454 previous fence). 9455 - Could be split into 9456 separate s_waitcnt 9457 vmcnt(0) and 9458 s_waitcnt 9459 lgkmcnt(0) to allow 9460 them to be 9461 independently moved 9462 according to the 9463 following rules. 9464 - s_waitcnt vmcnt(0) 9465 must happen after 9466 any preceding 9467 global/generic load 9468 atomic/atomicrmw 9469 with an equal or 9470 wider sync scope 9471 and memory ordering 9472 stronger than 9473 unordered (this is 9474 termed the 9475 fence-paired-atomic). 9476 - s_waitcnt lgkmcnt(0) 9477 must happen after 9478 any preceding 9479 local/generic load 9480 atomic/atomicrmw 9481 with an equal or 9482 wider sync scope 9483 and memory ordering 9484 stronger than 9485 unordered (this is 9486 termed the 9487 fence-paired-atomic). 9488 - Must happen before 9489 the following 9490 buffer_inv. 9491 - Ensures that the 9492 fence-paired atomic 9493 has completed 9494 before invalidating 9495 the 9496 cache. Therefore 9497 any following 9498 locations read must 9499 be no older than 9500 the value read by 9501 the 9502 fence-paired-atomic. 9503 9504 2. buffer_inv sc1=1 9505 9506 - Must happen before any 9507 following global/generic 9508 load/load 9509 atomic/store/store 9510 atomic/atomicrmw. 9511 - Ensures that 9512 following loads 9513 will not see stale 9514 global data. 9515 9516 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) & 9517 vmcnt(0) 9518 9519 - If TgSplit execution mode, 9520 omit lgkmcnt(0). 9521 - If OpenCL and 9522 address space is 9523 not generic, omit 9524 lgkmcnt(0). 9525 - However, since LLVM 9526 currently has no 9527 address space on 9528 the fence need to 9529 conservatively 9530 always generate 9531 (see comment for 9532 previous fence). 9533 - Could be split into 9534 separate s_waitcnt 9535 vmcnt(0) and 9536 s_waitcnt 9537 lgkmcnt(0) to allow 9538 them to be 9539 independently moved 9540 according to the 9541 following rules. 9542 - s_waitcnt vmcnt(0) 9543 must happen after 9544 any preceding 9545 global/generic load 9546 atomic/atomicrmw 9547 with an equal or 9548 wider sync scope 9549 and memory ordering 9550 stronger than 9551 unordered (this is 9552 termed the 9553 fence-paired-atomic). 9554 - s_waitcnt lgkmcnt(0) 9555 must happen after 9556 any preceding 9557 local/generic load 9558 atomic/atomicrmw 9559 with an equal or 9560 wider sync scope 9561 and memory ordering 9562 stronger than 9563 unordered (this is 9564 termed the 9565 fence-paired-atomic). 9566 - Must happen before 9567 the following 9568 buffer_inv. 9569 - Ensures that the 9570 fence-paired atomic 9571 has completed 9572 before invalidating 9573 the 9574 cache. Therefore 9575 any following 9576 locations read must 9577 be no older than 9578 the value read by 9579 the 9580 fence-paired-atomic. 9581 9582 2. buffer_inv sc0=1 sc1=1 9583 9584 - Must happen before any 9585 following global/generic 9586 load/load 9587 atomic/store/store 9588 atomic/atomicrmw. 9589 - Ensures that 9590 following loads 9591 will not see stale 9592 global data. 9593 9594 **Release Atomic** 9595 ------------------------------------------------------------------------------------ 9596 store atomic release - singlethread - global 1. buffer/global/flat_store 9597 - wavefront - generic 9598 store atomic release - singlethread - local *If TgSplit execution mode, 9599 - wavefront local address space cannot 9600 be used.* 9601 9602 1. ds_store 9603 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9604 - generic 9605 - Use lgkmcnt(0) if not 9606 TgSplit execution mode 9607 and vmcnt(0) if TgSplit 9608 execution mode. 9609 - If OpenCL, omit lgkmcnt(0). 9610 - s_waitcnt vmcnt(0) 9611 must happen after 9612 any preceding 9613 global/generic load/store/ 9614 load atomic/store atomic/ 9615 atomicrmw. 9616 - s_waitcnt lgkmcnt(0) 9617 must happen after 9618 any preceding 9619 local/generic 9620 load/store/load 9621 atomic/store 9622 atomic/atomicrmw. 9623 - Must happen before 9624 the following 9625 store. 9626 - Ensures that all 9627 memory operations 9628 have 9629 completed before 9630 performing the 9631 store that is being 9632 released. 9633 9634 2. buffer/global/flat_store sc0=1 9635 store atomic release - workgroup - local *If TgSplit execution mode, 9636 local address space cannot 9637 be used.* 9638 9639 1. ds_store 9640 store atomic release - agent - global 1. buffer_wbl2 sc1=1 9641 - generic 9642 - Must happen before 9643 following s_waitcnt. 9644 - Performs L2 writeback to 9645 ensure previous 9646 global/generic 9647 store/atomicrmw are 9648 visible at agent scope. 9649 9650 2. s_waitcnt lgkmcnt(0) & 9651 vmcnt(0) 9652 9653 - If TgSplit execution mode, 9654 omit lgkmcnt(0). 9655 - If OpenCL and 9656 address space is 9657 not generic, omit 9658 lgkmcnt(0). 9659 - Could be split into 9660 separate s_waitcnt 9661 vmcnt(0) and 9662 s_waitcnt 9663 lgkmcnt(0) to allow 9664 them to be 9665 independently moved 9666 according to the 9667 following rules. 9668 - s_waitcnt vmcnt(0) 9669 must happen after 9670 any preceding 9671 global/generic 9672 load/store/load 9673 atomic/store 9674 atomic/atomicrmw. 9675 - s_waitcnt lgkmcnt(0) 9676 must happen after 9677 any preceding 9678 local/generic 9679 load/store/load 9680 atomic/store 9681 atomic/atomicrmw. 9682 - Must happen before 9683 the following 9684 store. 9685 - Ensures that all 9686 memory operations 9687 to memory have 9688 completed before 9689 performing the 9690 store that is being 9691 released. 9692 9693 3. buffer/global/flat_store sc1=1 9694 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9695 - generic 9696 - Must happen before 9697 following s_waitcnt. 9698 - Performs L2 writeback to 9699 ensure previous 9700 global/generic 9701 store/atomicrmw are 9702 visible at system scope. 9703 9704 2. s_waitcnt lgkmcnt(0) & 9705 vmcnt(0) 9706 9707 - If TgSplit execution mode, 9708 omit lgkmcnt(0). 9709 - If OpenCL and 9710 address space is 9711 not generic, omit 9712 lgkmcnt(0). 9713 - Could be split into 9714 separate s_waitcnt 9715 vmcnt(0) and 9716 s_waitcnt 9717 lgkmcnt(0) to allow 9718 them to be 9719 independently moved 9720 according to the 9721 following rules. 9722 - s_waitcnt vmcnt(0) 9723 must happen after any 9724 preceding 9725 global/generic 9726 load/store/load 9727 atomic/store 9728 atomic/atomicrmw. 9729 - s_waitcnt lgkmcnt(0) 9730 must happen after any 9731 preceding 9732 local/generic 9733 load/store/load 9734 atomic/store 9735 atomic/atomicrmw. 9736 - Must happen before 9737 the following 9738 store. 9739 - Ensures that all 9740 memory operations 9741 to memory and the L2 9742 writeback have 9743 completed before 9744 performing the 9745 store that is being 9746 released. 9747 9748 3. buffer/global/flat_store 9749 sc0=1 sc1=1 9750 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic 9751 - wavefront - generic 9752 atomicrmw release - singlethread - local *If TgSplit execution mode, 9753 - wavefront local address space cannot 9754 be used.* 9755 9756 1. ds_atomic 9757 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 9758 - generic 9759 - Use lgkmcnt(0) if not 9760 TgSplit execution mode 9761 and vmcnt(0) if TgSplit 9762 execution mode. 9763 - If OpenCL, omit 9764 lgkmcnt(0). 9765 - s_waitcnt vmcnt(0) 9766 must happen after 9767 any preceding 9768 global/generic load/store/ 9769 load atomic/store atomic/ 9770 atomicrmw. 9771 - s_waitcnt lgkmcnt(0) 9772 must happen after 9773 any preceding 9774 local/generic 9775 load/store/load 9776 atomic/store 9777 atomic/atomicrmw. 9778 - Must happen before 9779 the following 9780 atomicrmw. 9781 - Ensures that all 9782 memory operations 9783 have 9784 completed before 9785 performing the 9786 atomicrmw that is 9787 being released. 9788 9789 2. buffer/global/flat_atomic sc0=1 9790 atomicrmw release - workgroup - local *If TgSplit execution mode, 9791 local address space cannot 9792 be used.* 9793 9794 1. ds_atomic 9795 atomicrmw release - agent - global 1. buffer_wbl2 sc1=1 9796 - generic 9797 - Must happen before 9798 following s_waitcnt. 9799 - Performs L2 writeback to 9800 ensure previous 9801 global/generic 9802 store/atomicrmw are 9803 visible at agent scope. 9804 9805 2. s_waitcnt lgkmcnt(0) & 9806 vmcnt(0) 9807 9808 - If TgSplit execution mode, 9809 omit lgkmcnt(0). 9810 - If OpenCL, omit 9811 lgkmcnt(0). 9812 - Could be split into 9813 separate s_waitcnt 9814 vmcnt(0) and 9815 s_waitcnt 9816 lgkmcnt(0) to allow 9817 them to be 9818 independently moved 9819 according to the 9820 following rules. 9821 - s_waitcnt vmcnt(0) 9822 must happen after 9823 any preceding 9824 global/generic 9825 load/store/load 9826 atomic/store 9827 atomic/atomicrmw. 9828 - s_waitcnt lgkmcnt(0) 9829 must happen after 9830 any preceding 9831 local/generic 9832 load/store/load 9833 atomic/store 9834 atomic/atomicrmw. 9835 - Must happen before 9836 the following 9837 atomicrmw. 9838 - Ensures that all 9839 memory operations 9840 to global and local 9841 have completed 9842 before performing 9843 the atomicrmw that 9844 is being released. 9845 9846 3. buffer/global/flat_atomic sc1=1 9847 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=1 9848 - generic 9849 - Must happen before 9850 following s_waitcnt. 9851 - Performs L2 writeback to 9852 ensure previous 9853 global/generic 9854 store/atomicrmw are 9855 visible at system scope. 9856 9857 2. s_waitcnt lgkmcnt(0) & 9858 vmcnt(0) 9859 9860 - If TgSplit execution mode, 9861 omit lgkmcnt(0). 9862 - If OpenCL, omit 9863 lgkmcnt(0). 9864 - Could be split into 9865 separate s_waitcnt 9866 vmcnt(0) and 9867 s_waitcnt 9868 lgkmcnt(0) to allow 9869 them to be 9870 independently moved 9871 according to the 9872 following rules. 9873 - s_waitcnt vmcnt(0) 9874 must happen after 9875 any preceding 9876 global/generic 9877 load/store/load 9878 atomic/store 9879 atomic/atomicrmw. 9880 - s_waitcnt lgkmcnt(0) 9881 must happen after 9882 any preceding 9883 local/generic 9884 load/store/load 9885 atomic/store 9886 atomic/atomicrmw. 9887 - Must happen before 9888 the following 9889 atomicrmw. 9890 - Ensures that all 9891 memory operations 9892 to memory and the L2 9893 writeback have 9894 completed before 9895 performing the 9896 store that is being 9897 released. 9898 9899 3. buffer/global/flat_atomic 9900 sc0=1 sc1=1 9901 fence release - singlethread *none* *none* 9902 - wavefront 9903 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 9904 9905 - Use lgkmcnt(0) if not 9906 TgSplit execution mode 9907 and vmcnt(0) if TgSplit 9908 execution mode. 9909 - If OpenCL and 9910 address space is 9911 not generic, omit 9912 lgkmcnt(0). 9913 - If OpenCL and 9914 address space is 9915 local, omit 9916 vmcnt(0). 9917 - However, since LLVM 9918 currently has no 9919 address space on 9920 the fence need to 9921 conservatively 9922 always generate. If 9923 fence had an 9924 address space then 9925 set to address 9926 space of OpenCL 9927 fence flag, or to 9928 generic if both 9929 local and global 9930 flags are 9931 specified. 9932 - s_waitcnt vmcnt(0) 9933 must happen after 9934 any preceding 9935 global/generic 9936 load/store/ 9937 load atomic/store atomic/ 9938 atomicrmw. 9939 - s_waitcnt lgkmcnt(0) 9940 must happen after 9941 any preceding 9942 local/generic 9943 load/load 9944 atomic/store/store 9945 atomic/atomicrmw. 9946 - Must happen before 9947 any following store 9948 atomic/atomicrmw 9949 with an equal or 9950 wider sync scope 9951 and memory ordering 9952 stronger than 9953 unordered (this is 9954 termed the 9955 fence-paired-atomic). 9956 - Ensures that all 9957 memory operations 9958 have 9959 completed before 9960 performing the 9961 following 9962 fence-paired-atomic. 9963 9964 fence release - agent *none* 1. buffer_wbl2 sc1=1 9965 9966 - If OpenCL and 9967 address space is 9968 local, omit. 9969 - Must happen before 9970 following s_waitcnt. 9971 - Performs L2 writeback to 9972 ensure previous 9973 global/generic 9974 store/atomicrmw are 9975 visible at agent scope. 9976 9977 2. s_waitcnt lgkmcnt(0) & 9978 vmcnt(0) 9979 9980 - If TgSplit execution mode, 9981 omit lgkmcnt(0). 9982 - If OpenCL and 9983 address space is 9984 not generic, omit 9985 lgkmcnt(0). 9986 - If OpenCL and 9987 address space is 9988 local, omit 9989 vmcnt(0). 9990 - However, since LLVM 9991 currently has no 9992 address space on 9993 the fence need to 9994 conservatively 9995 always generate. If 9996 fence had an 9997 address space then 9998 set to address 9999 space of OpenCL 10000 fence flag, or to 10001 generic if both 10002 local and global 10003 flags are 10004 specified. 10005 - Could be split into 10006 separate s_waitcnt 10007 vmcnt(0) and 10008 s_waitcnt 10009 lgkmcnt(0) to allow 10010 them to be 10011 independently moved 10012 according to the 10013 following rules. 10014 - s_waitcnt vmcnt(0) 10015 must happen after 10016 any preceding 10017 global/generic 10018 load/store/load 10019 atomic/store 10020 atomic/atomicrmw. 10021 - s_waitcnt lgkmcnt(0) 10022 must happen after 10023 any preceding 10024 local/generic 10025 load/store/load 10026 atomic/store 10027 atomic/atomicrmw. 10028 - Must happen before 10029 any following store 10030 atomic/atomicrmw 10031 with an equal or 10032 wider sync scope 10033 and memory ordering 10034 stronger than 10035 unordered (this is 10036 termed the 10037 fence-paired-atomic). 10038 - Ensures that all 10039 memory operations 10040 have 10041 completed before 10042 performing the 10043 following 10044 fence-paired-atomic. 10045 10046 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10047 10048 - Must happen before 10049 following s_waitcnt. 10050 - Performs L2 writeback to 10051 ensure previous 10052 global/generic 10053 store/atomicrmw are 10054 visible at system scope. 10055 10056 2. s_waitcnt lgkmcnt(0) & 10057 vmcnt(0) 10058 10059 - If TgSplit execution mode, 10060 omit lgkmcnt(0). 10061 - If OpenCL and 10062 address space is 10063 not generic, omit 10064 lgkmcnt(0). 10065 - If OpenCL and 10066 address space is 10067 local, omit 10068 vmcnt(0). 10069 - However, since LLVM 10070 currently has no 10071 address space on 10072 the fence need to 10073 conservatively 10074 always generate. If 10075 fence had an 10076 address space then 10077 set to address 10078 space of OpenCL 10079 fence flag, or to 10080 generic if both 10081 local and global 10082 flags are 10083 specified. 10084 - Could be split into 10085 separate s_waitcnt 10086 vmcnt(0) and 10087 s_waitcnt 10088 lgkmcnt(0) to allow 10089 them to be 10090 independently moved 10091 according to the 10092 following rules. 10093 - s_waitcnt vmcnt(0) 10094 must happen after 10095 any preceding 10096 global/generic 10097 load/store/load 10098 atomic/store 10099 atomic/atomicrmw. 10100 - s_waitcnt lgkmcnt(0) 10101 must happen after 10102 any preceding 10103 local/generic 10104 load/store/load 10105 atomic/store 10106 atomic/atomicrmw. 10107 - Must happen before 10108 any following store 10109 atomic/atomicrmw 10110 with an equal or 10111 wider sync scope 10112 and memory ordering 10113 stronger than 10114 unordered (this is 10115 termed the 10116 fence-paired-atomic). 10117 - Ensures that all 10118 memory operations 10119 have 10120 completed before 10121 performing the 10122 following 10123 fence-paired-atomic. 10124 10125 **Acquire-Release Atomic** 10126 ------------------------------------------------------------------------------------ 10127 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic 10128 - wavefront - generic 10129 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode, 10130 - wavefront local address space cannot 10131 be used.* 10132 10133 1. ds_atomic 10134 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10135 10136 - Use lgkmcnt(0) if not 10137 TgSplit execution mode 10138 and vmcnt(0) if TgSplit 10139 execution mode. 10140 - If OpenCL, omit 10141 lgkmcnt(0). 10142 - Must happen after 10143 any preceding 10144 local/generic 10145 load/store/load 10146 atomic/store 10147 atomic/atomicrmw. 10148 - s_waitcnt vmcnt(0) 10149 must happen after 10150 any preceding 10151 global/generic load/store/ 10152 load atomic/store atomic/ 10153 atomicrmw. 10154 - s_waitcnt lgkmcnt(0) 10155 must happen after 10156 any preceding 10157 local/generic 10158 load/store/load 10159 atomic/store 10160 atomic/atomicrmw. 10161 - Must happen before 10162 the following 10163 atomicrmw. 10164 - Ensures that all 10165 memory operations 10166 have 10167 completed before 10168 performing the 10169 atomicrmw that is 10170 being released. 10171 10172 2. buffer/global_atomic 10173 3. s_waitcnt vmcnt(0) 10174 10175 - If not TgSplit execution 10176 mode, omit. 10177 - Must happen before 10178 the following 10179 buffer_inv. 10180 - Ensures any 10181 following global 10182 data read is no 10183 older than the 10184 atomicrmw value 10185 being acquired. 10186 10187 4. buffer_inv sc0=1 10188 10189 - If not TgSplit execution 10190 mode, omit. 10191 - Ensures that 10192 following 10193 loads will not see 10194 stale data. 10195 10196 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode, 10197 local address space cannot 10198 be used.* 10199 10200 1. ds_atomic 10201 2. s_waitcnt lgkmcnt(0) 10202 10203 - If OpenCL, omit. 10204 - Must happen before 10205 any following 10206 global/generic 10207 load/load 10208 atomic/store/store 10209 atomic/atomicrmw. 10210 - Ensures any 10211 following global 10212 data read is no 10213 older than the local load 10214 atomic value being 10215 acquired. 10216 10217 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0) 10218 10219 - Use lgkmcnt(0) if not 10220 TgSplit execution mode 10221 and vmcnt(0) if TgSplit 10222 execution mode. 10223 - If OpenCL, omit 10224 lgkmcnt(0). 10225 - s_waitcnt vmcnt(0) 10226 must happen after 10227 any preceding 10228 global/generic load/store/ 10229 load atomic/store atomic/ 10230 atomicrmw. 10231 - s_waitcnt lgkmcnt(0) 10232 must happen after 10233 any preceding 10234 local/generic 10235 load/store/load 10236 atomic/store 10237 atomic/atomicrmw. 10238 - Must happen before 10239 the following 10240 atomicrmw. 10241 - Ensures that all 10242 memory operations 10243 have 10244 completed before 10245 performing the 10246 atomicrmw that is 10247 being released. 10248 10249 2. flat_atomic 10250 3. s_waitcnt lgkmcnt(0) & 10251 vmcnt(0) 10252 10253 - If not TgSplit execution 10254 mode, omit vmcnt(0). 10255 - If OpenCL, omit 10256 lgkmcnt(0). 10257 - Must happen before 10258 the following 10259 buffer_inv and 10260 any following 10261 global/generic 10262 load/load 10263 atomic/store/store 10264 atomic/atomicrmw. 10265 - Ensures any 10266 following global 10267 data read is no 10268 older than a local load 10269 atomic value being 10270 acquired. 10271 10272 3. buffer_inv sc0=1 10273 10274 - If not TgSplit execution 10275 mode, omit. 10276 - Ensures that 10277 following 10278 loads will not see 10279 stale data. 10280 10281 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=1 10282 10283 - Must happen before 10284 following s_waitcnt. 10285 - Performs L2 writeback to 10286 ensure previous 10287 global/generic 10288 store/atomicrmw are 10289 visible at agent scope. 10290 10291 2. s_waitcnt lgkmcnt(0) & 10292 vmcnt(0) 10293 10294 - If TgSplit execution mode, 10295 omit lgkmcnt(0). 10296 - If OpenCL, omit 10297 lgkmcnt(0). 10298 - Could be split into 10299 separate s_waitcnt 10300 vmcnt(0) and 10301 s_waitcnt 10302 lgkmcnt(0) to allow 10303 them to be 10304 independently moved 10305 according to the 10306 following rules. 10307 - s_waitcnt vmcnt(0) 10308 must happen after 10309 any preceding 10310 global/generic 10311 load/store/load 10312 atomic/store 10313 atomic/atomicrmw. 10314 - s_waitcnt lgkmcnt(0) 10315 must happen after 10316 any preceding 10317 local/generic 10318 load/store/load 10319 atomic/store 10320 atomic/atomicrmw. 10321 - Must happen before 10322 the following 10323 atomicrmw. 10324 - Ensures that all 10325 memory operations 10326 to global have 10327 completed before 10328 performing the 10329 atomicrmw that is 10330 being released. 10331 10332 3. buffer/global_atomic 10333 4. s_waitcnt vmcnt(0) 10334 10335 - Must happen before 10336 following 10337 buffer_inv. 10338 - Ensures the 10339 atomicrmw has 10340 completed before 10341 invalidating the 10342 cache. 10343 10344 5. buffer_inv sc1=1 10345 10346 - Must happen before 10347 any following 10348 global/generic 10349 load/load 10350 atomic/atomicrmw. 10351 - Ensures that 10352 following loads 10353 will not see stale 10354 global data. 10355 10356 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=1 10357 10358 - Must happen before 10359 following s_waitcnt. 10360 - Performs L2 writeback to 10361 ensure previous 10362 global/generic 10363 store/atomicrmw are 10364 visible at system scope. 10365 10366 2. s_waitcnt lgkmcnt(0) & 10367 vmcnt(0) 10368 10369 - If TgSplit execution mode, 10370 omit lgkmcnt(0). 10371 - If OpenCL, omit 10372 lgkmcnt(0). 10373 - Could be split into 10374 separate s_waitcnt 10375 vmcnt(0) and 10376 s_waitcnt 10377 lgkmcnt(0) to allow 10378 them to be 10379 independently moved 10380 according to the 10381 following rules. 10382 - s_waitcnt vmcnt(0) 10383 must happen after 10384 any preceding 10385 global/generic 10386 load/store/load 10387 atomic/store 10388 atomic/atomicrmw. 10389 - s_waitcnt lgkmcnt(0) 10390 must happen after 10391 any preceding 10392 local/generic 10393 load/store/load 10394 atomic/store 10395 atomic/atomicrmw. 10396 - Must happen before 10397 the following 10398 atomicrmw. 10399 - Ensures that all 10400 memory operations 10401 to global and L2 writeback 10402 have completed before 10403 performing the 10404 atomicrmw that is 10405 being released. 10406 10407 3. buffer/global_atomic 10408 sc1=1 10409 4. s_waitcnt vmcnt(0) 10410 10411 - Must happen before 10412 following 10413 buffer_inv. 10414 - Ensures the 10415 atomicrmw has 10416 completed before 10417 invalidating the 10418 caches. 10419 10420 5. buffer_inv sc0=1 sc1=1 10421 10422 - Must happen before 10423 any following 10424 global/generic 10425 load/load 10426 atomic/atomicrmw. 10427 - Ensures that 10428 following loads 10429 will not see stale 10430 MTYPE NC global data. 10431 MTYPE RW and CC memory will 10432 never be stale due to the 10433 memory probes. 10434 10435 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=1 10436 10437 - Must happen before 10438 following s_waitcnt. 10439 - Performs L2 writeback to 10440 ensure previous 10441 global/generic 10442 store/atomicrmw are 10443 visible at agent scope. 10444 10445 2. s_waitcnt lgkmcnt(0) & 10446 vmcnt(0) 10447 10448 - If TgSplit execution mode, 10449 omit lgkmcnt(0). 10450 - If OpenCL, omit 10451 lgkmcnt(0). 10452 - Could be split into 10453 separate s_waitcnt 10454 vmcnt(0) and 10455 s_waitcnt 10456 lgkmcnt(0) to allow 10457 them to be 10458 independently moved 10459 according to the 10460 following rules. 10461 - s_waitcnt vmcnt(0) 10462 must happen after 10463 any preceding 10464 global/generic 10465 load/store/load 10466 atomic/store 10467 atomic/atomicrmw. 10468 - s_waitcnt lgkmcnt(0) 10469 must happen after 10470 any preceding 10471 local/generic 10472 load/store/load 10473 atomic/store 10474 atomic/atomicrmw. 10475 - Must happen before 10476 the following 10477 atomicrmw. 10478 - Ensures that all 10479 memory operations 10480 to global have 10481 completed before 10482 performing the 10483 atomicrmw that is 10484 being released. 10485 10486 3. flat_atomic 10487 4. s_waitcnt vmcnt(0) & 10488 lgkmcnt(0) 10489 10490 - If TgSplit execution mode, 10491 omit lgkmcnt(0). 10492 - If OpenCL, omit 10493 lgkmcnt(0). 10494 - Must happen before 10495 following 10496 buffer_inv. 10497 - Ensures the 10498 atomicrmw has 10499 completed before 10500 invalidating the 10501 cache. 10502 10503 5. buffer_inv sc1=1 10504 10505 - Must happen before 10506 any following 10507 global/generic 10508 load/load 10509 atomic/atomicrmw. 10510 - Ensures that 10511 following loads 10512 will not see stale 10513 global data. 10514 10515 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=1 10516 10517 - Must happen before 10518 following s_waitcnt. 10519 - Performs L2 writeback to 10520 ensure previous 10521 global/generic 10522 store/atomicrmw are 10523 visible at system scope. 10524 10525 2. s_waitcnt lgkmcnt(0) & 10526 vmcnt(0) 10527 10528 - If TgSplit execution mode, 10529 omit lgkmcnt(0). 10530 - If OpenCL, omit 10531 lgkmcnt(0). 10532 - Could be split into 10533 separate s_waitcnt 10534 vmcnt(0) and 10535 s_waitcnt 10536 lgkmcnt(0) to allow 10537 them to be 10538 independently moved 10539 according to the 10540 following rules. 10541 - s_waitcnt vmcnt(0) 10542 must happen after 10543 any preceding 10544 global/generic 10545 load/store/load 10546 atomic/store 10547 atomic/atomicrmw. 10548 - s_waitcnt lgkmcnt(0) 10549 must happen after 10550 any preceding 10551 local/generic 10552 load/store/load 10553 atomic/store 10554 atomic/atomicrmw. 10555 - Must happen before 10556 the following 10557 atomicrmw. 10558 - Ensures that all 10559 memory operations 10560 to global and L2 writeback 10561 have completed before 10562 performing the 10563 atomicrmw that is 10564 being released. 10565 10566 3. flat_atomic sc1=1 10567 4. s_waitcnt vmcnt(0) & 10568 lgkmcnt(0) 10569 10570 - If TgSplit execution mode, 10571 omit lgkmcnt(0). 10572 - If OpenCL, omit 10573 lgkmcnt(0). 10574 - Must happen before 10575 following 10576 buffer_inv. 10577 - Ensures the 10578 atomicrmw has 10579 completed before 10580 invalidating the 10581 caches. 10582 10583 5. buffer_inv sc0=1 sc1=1 10584 10585 - Must happen before 10586 any following 10587 global/generic 10588 load/load 10589 atomic/atomicrmw. 10590 - Ensures that 10591 following loads 10592 will not see stale 10593 MTYPE NC global data. 10594 MTYPE RW and CC memory will 10595 never be stale due to the 10596 memory probes. 10597 10598 fence acq_rel - singlethread *none* *none* 10599 - wavefront 10600 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0) 10601 10602 - Use lgkmcnt(0) if not 10603 TgSplit execution mode 10604 and vmcnt(0) if TgSplit 10605 execution mode. 10606 - If OpenCL and 10607 address space is 10608 not generic, omit 10609 lgkmcnt(0). 10610 - If OpenCL and 10611 address space is 10612 local, omit 10613 vmcnt(0). 10614 - However, 10615 since LLVM 10616 currently has no 10617 address space on 10618 the fence need to 10619 conservatively 10620 always generate 10621 (see comment for 10622 previous fence). 10623 - s_waitcnt vmcnt(0) 10624 must happen after 10625 any preceding 10626 global/generic 10627 load/store/ 10628 load atomic/store atomic/ 10629 atomicrmw. 10630 - s_waitcnt lgkmcnt(0) 10631 must happen after 10632 any preceding 10633 local/generic 10634 load/load 10635 atomic/store/store 10636 atomic/atomicrmw. 10637 - Must happen before 10638 any following 10639 global/generic 10640 load/load 10641 atomic/store/store 10642 atomic/atomicrmw. 10643 - Ensures that all 10644 memory operations 10645 have 10646 completed before 10647 performing any 10648 following global 10649 memory operations. 10650 - Ensures that the 10651 preceding 10652 local/generic load 10653 atomic/atomicrmw 10654 with an equal or 10655 wider sync scope 10656 and memory ordering 10657 stronger than 10658 unordered (this is 10659 termed the 10660 acquire-fence-paired-atomic) 10661 has completed 10662 before following 10663 global memory 10664 operations. This 10665 satisfies the 10666 requirements of 10667 acquire. 10668 - Ensures that all 10669 previous memory 10670 operations have 10671 completed before a 10672 following 10673 local/generic store 10674 atomic/atomicrmw 10675 with an equal or 10676 wider sync scope 10677 and memory ordering 10678 stronger than 10679 unordered (this is 10680 termed the 10681 release-fence-paired-atomic). 10682 This satisfies the 10683 requirements of 10684 release. 10685 - Must happen before 10686 the following 10687 buffer_inv. 10688 - Ensures that the 10689 acquire-fence-paired 10690 atomic has completed 10691 before invalidating 10692 the 10693 cache. Therefore 10694 any following 10695 locations read must 10696 be no older than 10697 the value read by 10698 the 10699 acquire-fence-paired-atomic. 10700 10701 3. buffer_inv sc0=1 10702 10703 - If not TgSplit execution 10704 mode, omit. 10705 - Ensures that 10706 following 10707 loads will not see 10708 stale data. 10709 10710 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=1 10711 10712 - If OpenCL and 10713 address space is 10714 local, omit. 10715 - Must happen before 10716 following s_waitcnt. 10717 - Performs L2 writeback to 10718 ensure previous 10719 global/generic 10720 store/atomicrmw are 10721 visible at agent scope. 10722 10723 2. s_waitcnt lgkmcnt(0) & 10724 vmcnt(0) 10725 10726 - If TgSplit execution mode, 10727 omit lgkmcnt(0). 10728 - If OpenCL and 10729 address space is 10730 not generic, omit 10731 lgkmcnt(0). 10732 - However, since LLVM 10733 currently has no 10734 address space on 10735 the fence need to 10736 conservatively 10737 always generate 10738 (see comment for 10739 previous fence). 10740 - Could be split into 10741 separate s_waitcnt 10742 vmcnt(0) and 10743 s_waitcnt 10744 lgkmcnt(0) to allow 10745 them to be 10746 independently moved 10747 according to the 10748 following rules. 10749 - s_waitcnt vmcnt(0) 10750 must happen after 10751 any preceding 10752 global/generic 10753 load/store/load 10754 atomic/store 10755 atomic/atomicrmw. 10756 - s_waitcnt lgkmcnt(0) 10757 must happen after 10758 any preceding 10759 local/generic 10760 load/store/load 10761 atomic/store 10762 atomic/atomicrmw. 10763 - Must happen before 10764 the following 10765 buffer_inv. 10766 - Ensures that the 10767 preceding 10768 global/local/generic 10769 load 10770 atomic/atomicrmw 10771 with an equal or 10772 wider sync scope 10773 and memory ordering 10774 stronger than 10775 unordered (this is 10776 termed the 10777 acquire-fence-paired-atomic) 10778 has completed 10779 before invalidating 10780 the cache. This 10781 satisfies the 10782 requirements of 10783 acquire. 10784 - Ensures that all 10785 previous memory 10786 operations have 10787 completed before a 10788 following 10789 global/local/generic 10790 store 10791 atomic/atomicrmw 10792 with an equal or 10793 wider sync scope 10794 and memory ordering 10795 stronger than 10796 unordered (this is 10797 termed the 10798 release-fence-paired-atomic). 10799 This satisfies the 10800 requirements of 10801 release. 10802 10803 3. buffer_inv sc1=1 10804 10805 - Must happen before 10806 any following 10807 global/generic 10808 load/load 10809 atomic/store/store 10810 atomic/atomicrmw. 10811 - Ensures that 10812 following loads 10813 will not see stale 10814 global data. This 10815 satisfies the 10816 requirements of 10817 acquire. 10818 10819 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=1 10820 10821 - If OpenCL and 10822 address space is 10823 local, omit. 10824 - Must happen before 10825 following s_waitcnt. 10826 - Performs L2 writeback to 10827 ensure previous 10828 global/generic 10829 store/atomicrmw are 10830 visible at system scope. 10831 10832 1. s_waitcnt lgkmcnt(0) & 10833 vmcnt(0) 10834 10835 - If TgSplit execution mode, 10836 omit lgkmcnt(0). 10837 - If OpenCL and 10838 address space is 10839 not generic, omit 10840 lgkmcnt(0). 10841 - However, since LLVM 10842 currently has no 10843 address space on 10844 the fence need to 10845 conservatively 10846 always generate 10847 (see comment for 10848 previous fence). 10849 - Could be split into 10850 separate s_waitcnt 10851 vmcnt(0) and 10852 s_waitcnt 10853 lgkmcnt(0) to allow 10854 them to be 10855 independently moved 10856 according to the 10857 following rules. 10858 - s_waitcnt vmcnt(0) 10859 must happen after 10860 any preceding 10861 global/generic 10862 load/store/load 10863 atomic/store 10864 atomic/atomicrmw. 10865 - s_waitcnt lgkmcnt(0) 10866 must happen after 10867 any preceding 10868 local/generic 10869 load/store/load 10870 atomic/store 10871 atomic/atomicrmw. 10872 - Must happen before 10873 the following 10874 buffer_inv. 10875 - Ensures that the 10876 preceding 10877 global/local/generic 10878 load 10879 atomic/atomicrmw 10880 with an equal or 10881 wider sync scope 10882 and memory ordering 10883 stronger than 10884 unordered (this is 10885 termed the 10886 acquire-fence-paired-atomic) 10887 has completed 10888 before invalidating 10889 the cache. This 10890 satisfies the 10891 requirements of 10892 acquire. 10893 - Ensures that all 10894 previous memory 10895 operations have 10896 completed before a 10897 following 10898 global/local/generic 10899 store 10900 atomic/atomicrmw 10901 with an equal or 10902 wider sync scope 10903 and memory ordering 10904 stronger than 10905 unordered (this is 10906 termed the 10907 release-fence-paired-atomic). 10908 This satisfies the 10909 requirements of 10910 release. 10911 10912 2. buffer_inv sc0=1 sc1=1 10913 10914 - Must happen before 10915 any following 10916 global/generic 10917 load/load 10918 atomic/store/store 10919 atomic/atomicrmw. 10920 - Ensures that 10921 following loads 10922 will not see stale 10923 MTYPE NC global data. 10924 MTYPE RW and CC memory will 10925 never be stale due to the 10926 memory probes. 10927 10928 **Sequential Consistent Atomic** 10929 ------------------------------------------------------------------------------------ 10930 load atomic seq_cst - singlethread - global *Same as corresponding 10931 - wavefront - local load atomic acquire, 10932 - generic except must generate 10933 all instructions even 10934 for OpenCL.* 10935 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0) 10936 - generic 10937 - Use lgkmcnt(0) if not 10938 TgSplit execution mode 10939 and vmcnt(0) if TgSplit 10940 execution mode. 10941 - s_waitcnt lgkmcnt(0) must 10942 happen after 10943 preceding 10944 local/generic load 10945 atomic/store 10946 atomic/atomicrmw 10947 with memory 10948 ordering of seq_cst 10949 and with equal or 10950 wider sync scope. 10951 (Note that seq_cst 10952 fences have their 10953 own s_waitcnt 10954 lgkmcnt(0) and so do 10955 not need to be 10956 considered.) 10957 - s_waitcnt vmcnt(0) 10958 must happen after 10959 preceding 10960 global/generic load 10961 atomic/store 10962 atomic/atomicrmw 10963 with memory 10964 ordering of seq_cst 10965 and with equal or 10966 wider sync scope. 10967 (Note that seq_cst 10968 fences have their 10969 own s_waitcnt 10970 vmcnt(0) and so do 10971 not need to be 10972 considered.) 10973 - Ensures any 10974 preceding 10975 sequential 10976 consistent global/local 10977 memory instructions 10978 have completed 10979 before executing 10980 this sequentially 10981 consistent 10982 instruction. This 10983 prevents reordering 10984 a seq_cst store 10985 followed by a 10986 seq_cst load. (Note 10987 that seq_cst is 10988 stronger than 10989 acquire/release as 10990 the reordering of 10991 load acquire 10992 followed by a store 10993 release is 10994 prevented by the 10995 s_waitcnt of 10996 the release, but 10997 there is nothing 10998 preventing a store 10999 release followed by 11000 load acquire from 11001 completing out of 11002 order. The s_waitcnt 11003 could be placed after 11004 seq_store or before 11005 the seq_load. We 11006 choose the load to 11007 make the s_waitcnt be 11008 as late as possible 11009 so that the store 11010 may have already 11011 completed.) 11012 11013 2. *Following 11014 instructions same as 11015 corresponding load 11016 atomic acquire, 11017 except must generate 11018 all instructions even 11019 for OpenCL.* 11020 load atomic seq_cst - workgroup - local *If TgSplit execution mode, 11021 local address space cannot 11022 be used.* 11023 11024 *Same as corresponding 11025 load atomic acquire, 11026 except must generate 11027 all instructions even 11028 for OpenCL.* 11029 11030 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 11031 - system - generic vmcnt(0) 11032 11033 - If TgSplit execution mode, 11034 omit lgkmcnt(0). 11035 - Could be split into 11036 separate s_waitcnt 11037 vmcnt(0) 11038 and s_waitcnt 11039 lgkmcnt(0) to allow 11040 them to be 11041 independently moved 11042 according to the 11043 following rules. 11044 - s_waitcnt lgkmcnt(0) 11045 must happen after 11046 preceding 11047 global/generic load 11048 atomic/store 11049 atomic/atomicrmw 11050 with memory 11051 ordering of seq_cst 11052 and with equal or 11053 wider sync scope. 11054 (Note that seq_cst 11055 fences have their 11056 own s_waitcnt 11057 lgkmcnt(0) and so do 11058 not need to be 11059 considered.) 11060 - s_waitcnt vmcnt(0) 11061 must happen after 11062 preceding 11063 global/generic load 11064 atomic/store 11065 atomic/atomicrmw 11066 with memory 11067 ordering of seq_cst 11068 and with equal or 11069 wider sync scope. 11070 (Note that seq_cst 11071 fences have their 11072 own s_waitcnt 11073 vmcnt(0) and so do 11074 not need to be 11075 considered.) 11076 - Ensures any 11077 preceding 11078 sequential 11079 consistent global 11080 memory instructions 11081 have completed 11082 before executing 11083 this sequentially 11084 consistent 11085 instruction. This 11086 prevents reordering 11087 a seq_cst store 11088 followed by a 11089 seq_cst load. (Note 11090 that seq_cst is 11091 stronger than 11092 acquire/release as 11093 the reordering of 11094 load acquire 11095 followed by a store 11096 release is 11097 prevented by the 11098 s_waitcnt of 11099 the release, but 11100 there is nothing 11101 preventing a store 11102 release followed by 11103 load acquire from 11104 completing out of 11105 order. The s_waitcnt 11106 could be placed after 11107 seq_store or before 11108 the seq_load. We 11109 choose the load to 11110 make the s_waitcnt be 11111 as late as possible 11112 so that the store 11113 may have already 11114 completed.) 11115 11116 2. *Following 11117 instructions same as 11118 corresponding load 11119 atomic acquire, 11120 except must generate 11121 all instructions even 11122 for OpenCL.* 11123 store atomic seq_cst - singlethread - global *Same as corresponding 11124 - wavefront - local store atomic release, 11125 - workgroup - generic except must generate 11126 - agent all instructions even 11127 - system for OpenCL.* 11128 atomicrmw seq_cst - singlethread - global *Same as corresponding 11129 - wavefront - local atomicrmw acq_rel, 11130 - workgroup - generic except must generate 11131 - agent all instructions even 11132 - system for OpenCL.* 11133 fence seq_cst - singlethread *none* *Same as corresponding 11134 - wavefront fence acq_rel, 11135 - workgroup except must generate 11136 - agent all instructions even 11137 - system for OpenCL.* 11138 ============ ============ ============== ========== ================================ 11139 11140.. _amdgpu-amdhsa-memory-model-gfx10: 11141 11142Memory Model GFX10 11143++++++++++++++++++ 11144 11145For GFX10: 11146 11147* Each agent has multiple shader arrays (SA). 11148* Each SA has multiple work-group processors (WGP). 11149* Each WGP has multiple compute units (CU). 11150* Each CU has multiple SIMDs that execute wavefronts. 11151* The wavefronts for a single work-group are executed in the same 11152 WGP. In CU wavefront execution mode the wavefronts may be executed by 11153 different SIMDs in the same CU. In WGP wavefront execution mode the 11154 wavefronts may be executed by different SIMDs in different CUs in the same 11155 WGP. 11156* Each WGP has a single LDS memory shared by the wavefronts of the work-groups 11157 executing on it. 11158* All LDS operations of a WGP are performed as wavefront wide operations in a 11159 global order and involve no caching. Completion is reported to a wavefront in 11160 execution order. 11161* The LDS memory has multiple request queues shared by the SIMDs of a 11162 WGP. Therefore, the LDS operations performed by different wavefronts of a 11163 work-group can be reordered relative to each other, which can result in 11164 reordering the visibility of vector memory operations with respect to LDS 11165 operations of other wavefronts in the same work-group. A ``s_waitcnt 11166 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and 11167 vector memory operations between wavefronts of a work-group, but not between 11168 operations performed by the same wavefront. 11169* The vector memory operations are performed as wavefront wide operations. 11170 Completion of load/store/sample operations are reported to a wavefront in 11171 execution order of other load/store/sample operations performed by that 11172 wavefront. 11173* The vector memory operations access a vector L0 cache. There is a single L0 11174 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no 11175 special action is required for coherence between the lanes of a single 11176 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between 11177 wavefronts executing in the same work-group as they may be executing on SIMDs 11178 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also 11179 required for coherence between wavefronts executing in different work-groups 11180 as they may be executing on different WGPs. 11181* The scalar memory operations access a scalar L0 cache shared by all wavefronts 11182 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar 11183 operations are used in a restricted way so do not impact the memory model. See 11184 :ref:`amdgpu-amdhsa-memory-spaces`. 11185* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on 11186 the same SA. Therefore, no special action is required for coherence between 11187 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is 11188 required for coherence between wavefronts executing in different work-groups 11189 as they may be executing on different SAs that access different L1s. 11190* The L1 caches have independent quadrants to service disjoint ranges of virtual 11191 addresses. 11192* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the 11193 vector and scalar memory operations performed by different wavefronts, whether 11194 executing in the same or different work-groups (which may be executing on 11195 different CUs accessing different L0s), can be reordered relative to each 11196 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure 11197 synchronization between vector memory operations of different wavefronts. It 11198 ensures a previous vector memory operation has completed before executing a 11199 subsequent vector memory or LDS operation and so can be used to meet the 11200 requirements of acquire, release and sequential consistency. 11201* The L1 caches use an L2 cache shared by all SAs on the same agent. 11202* The L2 cache has independent channels to service disjoint ranges of virtual 11203 addresses. 11204* Each L1 quadrant of a single SA accesses a different L2 channel. Each L1 11205 quadrant has a separate request queue per L2 channel. Therefore, the vector 11206 and scalar memory operations performed by wavefronts executing in different 11207 work-groups (which may be executing on different SAs) of an agent can be 11208 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is 11209 required to ensure synchronization between vector memory operations of 11210 different SAs. It ensures a previous vector memory operation has completed 11211 before executing a subsequent vector memory and so can be used to meet the 11212 requirements of acquire, release and sequential consistency. 11213* The L2 cache can be kept coherent with other agents on some targets, or ranges 11214 of virtual addresses can be set up to bypass it to ensure system coherence. 11215* On GFX10.3 a memory attached last level (MALL) cache exists for GPU memory. 11216 The MALL cache is fully coherent with GPU memory and has no impact on system 11217 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache. 11218 11219Scalar memory operations are only used to access memory that is proven to not 11220change during the execution of the kernel dispatch. This includes constant 11221address space and global address space for program scope ``const`` variables. 11222Therefore, the kernel machine code does not have to maintain the scalar cache to 11223ensure it is coherent with the vector caches. The scalar and vector caches are 11224invalidated between kernel dispatches by CP since constant address space data 11225may change between kernel dispatch executions. See 11226:ref:`amdgpu-amdhsa-memory-spaces`. 11227 11228The one exception is if scalar writes are used to spill SGPR registers. In this 11229case the AMDGPU backend ensures the memory location used to spill is never 11230accessed by vector memory operations at the same time. If scalar writes are used 11231then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function 11232return since the locations may be used for vector memory instructions by a 11233future wavefront that uses the same scratch area, or a function call that 11234creates a frame at the same address, respectively. There is no need for a 11235``s_dcache_inv`` as all scalar writes are write-before-read in the same thread. 11236 11237For kernarg backing memory: 11238 11239* CP invalidates the L0 and L1 caches at the start of each kernel dispatch. 11240* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid 11241 needing to invalidate the L2 cache. 11242* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and 11243 so the L2 cache will be coherent with the CPU and other agents. 11244 11245Scratch backing memory (which is used for the private address space) is accessed 11246with MTYPE NC (non-coherent). Since the private address space is only accessed 11247by a single thread, and is always write-before-read, there is never a need to 11248invalidate these entries from the L0 or L1 caches. 11249 11250Wavefronts are executed in native mode with in-order reporting of loads and 11251sample instructions. In this mode vmcnt reports completion of load, atomic with 11252return and sample instructions in order, and the vscnt reports the completion of 11253store and atomic without return in order. See ``MEM_ORDERED`` field in 11254:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 11255 11256Wavefronts can be executed in WGP or CU wavefront execution mode: 11257 11258* In WGP wavefront execution mode the wavefronts of a work-group are executed 11259 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per 11260 CU L0 caches is required for work-group synchronization. Also accesses to L1 11261 at work-group scope need to be explicitly ordered as the accesses from 11262 different CUs are not ordered. 11263* In CU wavefront execution mode the wavefronts of a work-group are executed on 11264 the SIMDs of a single CU of the WGP. Therefore, all global memory access by 11265 the work-group access the same L0 which in turn ensures L1 accesses are 11266 ordered and so do not require explicit management of the caches for 11267 work-group synchronization. 11268 11269See ``WGP_MODE`` field in 11270:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table` and 11271:ref:`amdgpu-target-features`. 11272 11273The code sequences used to implement the memory model for GFX10 are defined in 11274table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-table`. 11275 11276 .. table:: AMDHSA Memory Model Code Sequences GFX10 11277 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-table 11278 11279 ============ ============ ============== ========== ================================ 11280 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code 11281 Ordering Sync Scope Address GFX10 11282 Space 11283 ============ ============ ============== ========== ================================ 11284 **Non-Atomic** 11285 ------------------------------------------------------------------------------------ 11286 load *none* *none* - global - !volatile & !nontemporal 11287 - generic 11288 - private 1. buffer/global/flat_load 11289 - constant 11290 - !volatile & nontemporal 11291 11292 1. buffer/global/flat_load 11293 slc=1 11294 11295 - volatile 11296 11297 1. buffer/global/flat_load 11298 glc=1 dlc=1 11299 2. s_waitcnt vmcnt(0) 11300 11301 - Must happen before 11302 any following volatile 11303 global/generic 11304 load/store. 11305 - Ensures that 11306 volatile 11307 operations to 11308 different 11309 addresses will not 11310 be reordered by 11311 hardware. 11312 11313 load *none* *none* - local 1. ds_load 11314 store *none* *none* - global - !volatile & !nontemporal 11315 - generic 11316 - private 1. buffer/global/flat_store 11317 - constant 11318 - !volatile & nontemporal 11319 11320 1. buffer/global/flat_store 11321 glc=1 slc=1 11322 11323 - volatile 11324 11325 1. buffer/global/flat_store 11326 2. s_waitcnt vscnt(0) 11327 11328 - Must happen before 11329 any following volatile 11330 global/generic 11331 load/store. 11332 - Ensures that 11333 volatile 11334 operations to 11335 different 11336 addresses will not 11337 be reordered by 11338 hardware. 11339 11340 store *none* *none* - local 1. ds_store 11341 **Unordered Atomic** 11342 ------------------------------------------------------------------------------------ 11343 load atomic unordered *any* *any* *Same as non-atomic*. 11344 store atomic unordered *any* *any* *Same as non-atomic*. 11345 atomicrmw unordered *any* *any* *Same as monotonic atomic*. 11346 **Monotonic Atomic** 11347 ------------------------------------------------------------------------------------ 11348 load atomic monotonic - singlethread - global 1. buffer/global/flat_load 11349 - wavefront - generic 11350 load atomic monotonic - workgroup - global 1. buffer/global/flat_load 11351 - generic glc=1 11352 11353 - If CU wavefront execution 11354 mode, omit glc=1. 11355 11356 load atomic monotonic - singlethread - local 1. ds_load 11357 - wavefront 11358 - workgroup 11359 load atomic monotonic - agent - global 1. buffer/global/flat_load 11360 - system - generic glc=1 dlc=1 11361 store atomic monotonic - singlethread - global 1. buffer/global/flat_store 11362 - wavefront - generic 11363 - workgroup 11364 - agent 11365 - system 11366 store atomic monotonic - singlethread - local 1. ds_store 11367 - wavefront 11368 - workgroup 11369 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic 11370 - wavefront - generic 11371 - workgroup 11372 - agent 11373 - system 11374 atomicrmw monotonic - singlethread - local 1. ds_atomic 11375 - wavefront 11376 - workgroup 11377 **Acquire Atomic** 11378 ------------------------------------------------------------------------------------ 11379 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load 11380 - wavefront - local 11381 - generic 11382 load atomic acquire - workgroup - global 1. buffer/global_load glc=1 11383 11384 - If CU wavefront execution 11385 mode, omit glc=1. 11386 11387 2. s_waitcnt vmcnt(0) 11388 11389 - If CU wavefront execution 11390 mode, omit. 11391 - Must happen before 11392 the following buffer_gl0_inv 11393 and before any following 11394 global/generic 11395 load/load 11396 atomic/store/store 11397 atomic/atomicrmw. 11398 11399 3. buffer_gl0_inv 11400 11401 - If CU wavefront execution 11402 mode, omit. 11403 - Ensures that 11404 following 11405 loads will not see 11406 stale data. 11407 11408 load atomic acquire - workgroup - local 1. ds_load 11409 2. s_waitcnt lgkmcnt(0) 11410 11411 - If OpenCL, omit. 11412 - Must happen before 11413 the following buffer_gl0_inv 11414 and before any following 11415 global/generic load/load 11416 atomic/store/store 11417 atomic/atomicrmw. 11418 - Ensures any 11419 following global 11420 data read is no 11421 older than the local load 11422 atomic value being 11423 acquired. 11424 11425 3. buffer_gl0_inv 11426 11427 - If CU wavefront execution 11428 mode, omit. 11429 - If OpenCL, omit. 11430 - Ensures that 11431 following 11432 loads will not see 11433 stale data. 11434 11435 load atomic acquire - workgroup - generic 1. flat_load glc=1 11436 11437 - If CU wavefront execution 11438 mode, omit glc=1. 11439 11440 2. s_waitcnt lgkmcnt(0) & 11441 vmcnt(0) 11442 11443 - If CU wavefront execution 11444 mode, omit vmcnt(0). 11445 - If OpenCL, omit 11446 lgkmcnt(0). 11447 - Must happen before 11448 the following 11449 buffer_gl0_inv and any 11450 following global/generic 11451 load/load 11452 atomic/store/store 11453 atomic/atomicrmw. 11454 - Ensures any 11455 following global 11456 data read is no 11457 older than a local load 11458 atomic value being 11459 acquired. 11460 11461 3. buffer_gl0_inv 11462 11463 - If CU wavefront execution 11464 mode, omit. 11465 - Ensures that 11466 following 11467 loads will not see 11468 stale data. 11469 11470 load atomic acquire - agent - global 1. buffer/global_load 11471 - system glc=1 dlc=1 11472 2. s_waitcnt vmcnt(0) 11473 11474 - Must happen before 11475 following 11476 buffer_gl*_inv. 11477 - Ensures the load 11478 has completed 11479 before invalidating 11480 the caches. 11481 11482 3. buffer_gl0_inv; 11483 buffer_gl1_inv 11484 11485 - Must happen before 11486 any following 11487 global/generic 11488 load/load 11489 atomic/atomicrmw. 11490 - Ensures that 11491 following 11492 loads will not see 11493 stale global data. 11494 11495 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=1 11496 - system 2. s_waitcnt vmcnt(0) & 11497 lgkmcnt(0) 11498 11499 - If OpenCL omit 11500 lgkmcnt(0). 11501 - Must happen before 11502 following 11503 buffer_gl*_invl. 11504 - Ensures the flat_load 11505 has completed 11506 before invalidating 11507 the caches. 11508 11509 3. buffer_gl0_inv; 11510 buffer_gl1_inv 11511 11512 - Must happen before 11513 any following 11514 global/generic 11515 load/load 11516 atomic/atomicrmw. 11517 - Ensures that 11518 following loads 11519 will not see stale 11520 global data. 11521 11522 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic 11523 - wavefront - local 11524 - generic 11525 atomicrmw acquire - workgroup - global 1. buffer/global_atomic 11526 2. s_waitcnt vm/vscnt(0) 11527 11528 - If CU wavefront execution 11529 mode, omit. 11530 - Use vmcnt(0) if atomic with 11531 return and vscnt(0) if 11532 atomic with no-return. 11533 - Must happen before 11534 the following buffer_gl0_inv 11535 and before any following 11536 global/generic 11537 load/load 11538 atomic/store/store 11539 atomic/atomicrmw. 11540 11541 3. buffer_gl0_inv 11542 11543 - If CU wavefront execution 11544 mode, omit. 11545 - Ensures that 11546 following 11547 loads will not see 11548 stale data. 11549 11550 atomicrmw acquire - workgroup - local 1. ds_atomic 11551 2. s_waitcnt lgkmcnt(0) 11552 11553 - If OpenCL, omit. 11554 - Must happen before 11555 the following 11556 buffer_gl0_inv. 11557 - Ensures any 11558 following global 11559 data read is no 11560 older than the local 11561 atomicrmw value 11562 being acquired. 11563 11564 3. buffer_gl0_inv 11565 11566 - If OpenCL omit. 11567 - Ensures that 11568 following 11569 loads will not see 11570 stale data. 11571 11572 atomicrmw acquire - workgroup - generic 1. flat_atomic 11573 2. s_waitcnt lgkmcnt(0) & 11574 vm/vscnt(0) 11575 11576 - If CU wavefront execution 11577 mode, omit vm/vscnt(0). 11578 - If OpenCL, omit lgkmcnt(0). 11579 - Use vmcnt(0) if atomic with 11580 return and vscnt(0) if 11581 atomic with no-return. 11582 - Must happen before 11583 the following 11584 buffer_gl0_inv. 11585 - Ensures any 11586 following global 11587 data read is no 11588 older than a local 11589 atomicrmw value 11590 being acquired. 11591 11592 3. buffer_gl0_inv 11593 11594 - If CU wavefront execution 11595 mode, omit. 11596 - Ensures that 11597 following 11598 loads will not see 11599 stale data. 11600 11601 atomicrmw acquire - agent - global 1. buffer/global_atomic 11602 - system 2. s_waitcnt vm/vscnt(0) 11603 11604 - Use vmcnt(0) if atomic with 11605 return and vscnt(0) if 11606 atomic with no-return. 11607 - Must happen before 11608 following 11609 buffer_gl*_inv. 11610 - Ensures the 11611 atomicrmw has 11612 completed before 11613 invalidating the 11614 caches. 11615 11616 3. buffer_gl0_inv; 11617 buffer_gl1_inv 11618 11619 - Must happen before 11620 any following 11621 global/generic 11622 load/load 11623 atomic/atomicrmw. 11624 - Ensures that 11625 following loads 11626 will not see stale 11627 global data. 11628 11629 atomicrmw acquire - agent - generic 1. flat_atomic 11630 - system 2. s_waitcnt vm/vscnt(0) & 11631 lgkmcnt(0) 11632 11633 - If OpenCL, omit 11634 lgkmcnt(0). 11635 - Use vmcnt(0) if atomic with 11636 return and vscnt(0) if 11637 atomic with no-return. 11638 - Must happen before 11639 following 11640 buffer_gl*_inv. 11641 - Ensures the 11642 atomicrmw has 11643 completed before 11644 invalidating the 11645 caches. 11646 11647 3. buffer_gl0_inv; 11648 buffer_gl1_inv 11649 11650 - Must happen before 11651 any following 11652 global/generic 11653 load/load 11654 atomic/atomicrmw. 11655 - Ensures that 11656 following loads 11657 will not see stale 11658 global data. 11659 11660 fence acquire - singlethread *none* *none* 11661 - wavefront 11662 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 11663 vmcnt(0) & vscnt(0) 11664 11665 - If CU wavefront execution 11666 mode, omit vmcnt(0) and 11667 vscnt(0). 11668 - If OpenCL and 11669 address space is 11670 not generic, omit 11671 lgkmcnt(0). 11672 - If OpenCL and 11673 address space is 11674 local, omit 11675 vmcnt(0) and vscnt(0). 11676 - However, since LLVM 11677 currently has no 11678 address space on 11679 the fence need to 11680 conservatively 11681 always generate. If 11682 fence had an 11683 address space then 11684 set to address 11685 space of OpenCL 11686 fence flag, or to 11687 generic if both 11688 local and global 11689 flags are 11690 specified. 11691 - Could be split into 11692 separate s_waitcnt 11693 vmcnt(0), s_waitcnt 11694 vscnt(0) and s_waitcnt 11695 lgkmcnt(0) to allow 11696 them to be 11697 independently moved 11698 according to the 11699 following rules. 11700 - s_waitcnt vmcnt(0) 11701 must happen after 11702 any preceding 11703 global/generic load 11704 atomic/ 11705 atomicrmw-with-return-value 11706 with an equal or 11707 wider sync scope 11708 and memory ordering 11709 stronger than 11710 unordered (this is 11711 termed the 11712 fence-paired-atomic). 11713 - s_waitcnt vscnt(0) 11714 must happen after 11715 any preceding 11716 global/generic 11717 atomicrmw-no-return-value 11718 with an equal or 11719 wider sync scope 11720 and memory ordering 11721 stronger than 11722 unordered (this is 11723 termed the 11724 fence-paired-atomic). 11725 - s_waitcnt lgkmcnt(0) 11726 must happen after 11727 any preceding 11728 local/generic load 11729 atomic/atomicrmw 11730 with an equal or 11731 wider sync scope 11732 and memory ordering 11733 stronger than 11734 unordered (this is 11735 termed the 11736 fence-paired-atomic). 11737 - Must happen before 11738 the following 11739 buffer_gl0_inv. 11740 - Ensures that the 11741 fence-paired atomic 11742 has completed 11743 before invalidating 11744 the 11745 cache. Therefore 11746 any following 11747 locations read must 11748 be no older than 11749 the value read by 11750 the 11751 fence-paired-atomic. 11752 11753 3. buffer_gl0_inv 11754 11755 - If CU wavefront execution 11756 mode, omit. 11757 - Ensures that 11758 following 11759 loads will not see 11760 stale data. 11761 11762 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) & 11763 - system vmcnt(0) & vscnt(0) 11764 11765 - If OpenCL and 11766 address space is 11767 not generic, omit 11768 lgkmcnt(0). 11769 - If OpenCL and 11770 address space is 11771 local, omit 11772 vmcnt(0) and vscnt(0). 11773 - However, since LLVM 11774 currently has no 11775 address space on 11776 the fence need to 11777 conservatively 11778 always generate 11779 (see comment for 11780 previous fence). 11781 - Could be split into 11782 separate s_waitcnt 11783 vmcnt(0), s_waitcnt 11784 vscnt(0) and s_waitcnt 11785 lgkmcnt(0) to allow 11786 them to be 11787 independently moved 11788 according to the 11789 following rules. 11790 - s_waitcnt vmcnt(0) 11791 must happen after 11792 any preceding 11793 global/generic load 11794 atomic/ 11795 atomicrmw-with-return-value 11796 with an equal or 11797 wider sync scope 11798 and memory ordering 11799 stronger than 11800 unordered (this is 11801 termed the 11802 fence-paired-atomic). 11803 - s_waitcnt vscnt(0) 11804 must happen after 11805 any preceding 11806 global/generic 11807 atomicrmw-no-return-value 11808 with an equal or 11809 wider sync scope 11810 and memory ordering 11811 stronger than 11812 unordered (this is 11813 termed the 11814 fence-paired-atomic). 11815 - s_waitcnt lgkmcnt(0) 11816 must happen after 11817 any preceding 11818 local/generic load 11819 atomic/atomicrmw 11820 with an equal or 11821 wider sync scope 11822 and memory ordering 11823 stronger than 11824 unordered (this is 11825 termed the 11826 fence-paired-atomic). 11827 - Must happen before 11828 the following 11829 buffer_gl*_inv. 11830 - Ensures that the 11831 fence-paired atomic 11832 has completed 11833 before invalidating 11834 the 11835 caches. Therefore 11836 any following 11837 locations read must 11838 be no older than 11839 the value read by 11840 the 11841 fence-paired-atomic. 11842 11843 2. buffer_gl0_inv; 11844 buffer_gl1_inv 11845 11846 - Must happen before any 11847 following global/generic 11848 load/load 11849 atomic/store/store 11850 atomic/atomicrmw. 11851 - Ensures that 11852 following loads 11853 will not see stale 11854 global data. 11855 11856 **Release Atomic** 11857 ------------------------------------------------------------------------------------ 11858 store atomic release - singlethread - global 1. buffer/global/ds/flat_store 11859 - wavefront - local 11860 - generic 11861 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11862 - generic vmcnt(0) & vscnt(0) 11863 11864 - If CU wavefront execution 11865 mode, omit vmcnt(0) and 11866 vscnt(0). 11867 - If OpenCL, omit 11868 lgkmcnt(0). 11869 - Could be split into 11870 separate s_waitcnt 11871 vmcnt(0), s_waitcnt 11872 vscnt(0) and s_waitcnt 11873 lgkmcnt(0) to allow 11874 them to be 11875 independently moved 11876 according to the 11877 following rules. 11878 - s_waitcnt vmcnt(0) 11879 must happen after 11880 any preceding 11881 global/generic load/load 11882 atomic/ 11883 atomicrmw-with-return-value. 11884 - s_waitcnt vscnt(0) 11885 must happen after 11886 any preceding 11887 global/generic 11888 store/store 11889 atomic/ 11890 atomicrmw-no-return-value. 11891 - s_waitcnt lgkmcnt(0) 11892 must happen after 11893 any preceding 11894 local/generic 11895 load/store/load 11896 atomic/store 11897 atomic/atomicrmw. 11898 - Must happen before 11899 the following 11900 store. 11901 - Ensures that all 11902 memory operations 11903 have 11904 completed before 11905 performing the 11906 store that is being 11907 released. 11908 11909 2. buffer/global/flat_store 11910 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 11911 11912 - If CU wavefront execution 11913 mode, omit. 11914 - If OpenCL, omit. 11915 - Could be split into 11916 separate s_waitcnt 11917 vmcnt(0) and s_waitcnt 11918 vscnt(0) to allow 11919 them to be 11920 independently moved 11921 according to the 11922 following rules. 11923 - s_waitcnt vmcnt(0) 11924 must happen after 11925 any preceding 11926 global/generic load/load 11927 atomic/ 11928 atomicrmw-with-return-value. 11929 - s_waitcnt vscnt(0) 11930 must happen after 11931 any preceding 11932 global/generic 11933 store/store atomic/ 11934 atomicrmw-no-return-value. 11935 - Must happen before 11936 the following 11937 store. 11938 - Ensures that all 11939 global memory 11940 operations have 11941 completed before 11942 performing the 11943 store that is being 11944 released. 11945 11946 2. ds_store 11947 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) & 11948 - system - generic vmcnt(0) & vscnt(0) 11949 11950 - If OpenCL and 11951 address space is 11952 not generic, omit 11953 lgkmcnt(0). 11954 - Could be split into 11955 separate s_waitcnt 11956 vmcnt(0), s_waitcnt vscnt(0) 11957 and s_waitcnt 11958 lgkmcnt(0) to allow 11959 them to be 11960 independently moved 11961 according to the 11962 following rules. 11963 - s_waitcnt vmcnt(0) 11964 must happen after 11965 any preceding 11966 global/generic 11967 load/load 11968 atomic/ 11969 atomicrmw-with-return-value. 11970 - s_waitcnt vscnt(0) 11971 must happen after 11972 any preceding 11973 global/generic 11974 store/store atomic/ 11975 atomicrmw-no-return-value. 11976 - s_waitcnt lgkmcnt(0) 11977 must happen after 11978 any preceding 11979 local/generic 11980 load/store/load 11981 atomic/store 11982 atomic/atomicrmw. 11983 - Must happen before 11984 the following 11985 store. 11986 - Ensures that all 11987 memory operations 11988 have 11989 completed before 11990 performing the 11991 store that is being 11992 released. 11993 11994 2. buffer/global/flat_store 11995 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic 11996 - wavefront - local 11997 - generic 11998 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) & 11999 - generic vmcnt(0) & vscnt(0) 12000 12001 - If CU wavefront execution 12002 mode, omit vmcnt(0) and 12003 vscnt(0). 12004 - If OpenCL, omit lgkmcnt(0). 12005 - Could be split into 12006 separate s_waitcnt 12007 vmcnt(0), s_waitcnt 12008 vscnt(0) and s_waitcnt 12009 lgkmcnt(0) to allow 12010 them to be 12011 independently moved 12012 according to the 12013 following rules. 12014 - s_waitcnt vmcnt(0) 12015 must happen after 12016 any preceding 12017 global/generic load/load 12018 atomic/ 12019 atomicrmw-with-return-value. 12020 - s_waitcnt vscnt(0) 12021 must happen after 12022 any preceding 12023 global/generic 12024 store/store 12025 atomic/ 12026 atomicrmw-no-return-value. 12027 - s_waitcnt lgkmcnt(0) 12028 must happen after 12029 any preceding 12030 local/generic 12031 load/store/load 12032 atomic/store 12033 atomic/atomicrmw. 12034 - Must happen before 12035 the following 12036 atomicrmw. 12037 - Ensures that all 12038 memory operations 12039 have 12040 completed before 12041 performing the 12042 atomicrmw that is 12043 being released. 12044 12045 2. buffer/global/flat_atomic 12046 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12047 12048 - If CU wavefront execution 12049 mode, omit. 12050 - If OpenCL, omit. 12051 - Could be split into 12052 separate s_waitcnt 12053 vmcnt(0) and s_waitcnt 12054 vscnt(0) to allow 12055 them to be 12056 independently moved 12057 according to the 12058 following rules. 12059 - s_waitcnt vmcnt(0) 12060 must happen after 12061 any preceding 12062 global/generic load/load 12063 atomic/ 12064 atomicrmw-with-return-value. 12065 - s_waitcnt vscnt(0) 12066 must happen after 12067 any preceding 12068 global/generic 12069 store/store atomic/ 12070 atomicrmw-no-return-value. 12071 - Must happen before 12072 the following 12073 store. 12074 - Ensures that all 12075 global memory 12076 operations have 12077 completed before 12078 performing the 12079 store that is being 12080 released. 12081 12082 2. ds_atomic 12083 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) & 12084 - system - generic vmcnt(0) & vscnt(0) 12085 12086 - If OpenCL, omit 12087 lgkmcnt(0). 12088 - Could be split into 12089 separate s_waitcnt 12090 vmcnt(0), s_waitcnt 12091 vscnt(0) and s_waitcnt 12092 lgkmcnt(0) to allow 12093 them to be 12094 independently moved 12095 according to the 12096 following rules. 12097 - s_waitcnt vmcnt(0) 12098 must happen after 12099 any preceding 12100 global/generic 12101 load/load atomic/ 12102 atomicrmw-with-return-value. 12103 - s_waitcnt vscnt(0) 12104 must happen after 12105 any preceding 12106 global/generic 12107 store/store atomic/ 12108 atomicrmw-no-return-value. 12109 - s_waitcnt lgkmcnt(0) 12110 must happen after 12111 any preceding 12112 local/generic 12113 load/store/load 12114 atomic/store 12115 atomic/atomicrmw. 12116 - Must happen before 12117 the following 12118 atomicrmw. 12119 - Ensures that all 12120 memory operations 12121 to global and local 12122 have completed 12123 before performing 12124 the atomicrmw that 12125 is being released. 12126 12127 2. buffer/global/flat_atomic 12128 fence release - singlethread *none* *none* 12129 - wavefront 12130 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12131 vmcnt(0) & vscnt(0) 12132 12133 - If CU wavefront execution 12134 mode, omit vmcnt(0) and 12135 vscnt(0). 12136 - If OpenCL and 12137 address space is 12138 not generic, omit 12139 lgkmcnt(0). 12140 - If OpenCL and 12141 address space is 12142 local, omit 12143 vmcnt(0) and vscnt(0). 12144 - However, since LLVM 12145 currently has no 12146 address space on 12147 the fence need to 12148 conservatively 12149 always generate. If 12150 fence had an 12151 address space then 12152 set to address 12153 space of OpenCL 12154 fence flag, or to 12155 generic if both 12156 local and global 12157 flags are 12158 specified. 12159 - Could be split into 12160 separate s_waitcnt 12161 vmcnt(0), s_waitcnt 12162 vscnt(0) and s_waitcnt 12163 lgkmcnt(0) to allow 12164 them to be 12165 independently moved 12166 according to the 12167 following rules. 12168 - s_waitcnt vmcnt(0) 12169 must happen after 12170 any preceding 12171 global/generic 12172 load/load 12173 atomic/ 12174 atomicrmw-with-return-value. 12175 - s_waitcnt vscnt(0) 12176 must happen after 12177 any preceding 12178 global/generic 12179 store/store atomic/ 12180 atomicrmw-no-return-value. 12181 - s_waitcnt lgkmcnt(0) 12182 must happen after 12183 any preceding 12184 local/generic 12185 load/store/load 12186 atomic/store atomic/ 12187 atomicrmw. 12188 - Must happen before 12189 any following store 12190 atomic/atomicrmw 12191 with an equal or 12192 wider sync scope 12193 and memory ordering 12194 stronger than 12195 unordered (this is 12196 termed the 12197 fence-paired-atomic). 12198 - Ensures that all 12199 memory operations 12200 have 12201 completed before 12202 performing the 12203 following 12204 fence-paired-atomic. 12205 12206 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) & 12207 - system vmcnt(0) & vscnt(0) 12208 12209 - If OpenCL and 12210 address space is 12211 not generic, omit 12212 lgkmcnt(0). 12213 - If OpenCL and 12214 address space is 12215 local, omit 12216 vmcnt(0) and vscnt(0). 12217 - However, since LLVM 12218 currently has no 12219 address space on 12220 the fence need to 12221 conservatively 12222 always generate. If 12223 fence had an 12224 address space then 12225 set to address 12226 space of OpenCL 12227 fence flag, or to 12228 generic if both 12229 local and global 12230 flags are 12231 specified. 12232 - Could be split into 12233 separate s_waitcnt 12234 vmcnt(0), s_waitcnt 12235 vscnt(0) and s_waitcnt 12236 lgkmcnt(0) to allow 12237 them to be 12238 independently moved 12239 according to the 12240 following rules. 12241 - s_waitcnt vmcnt(0) 12242 must happen after 12243 any preceding 12244 global/generic 12245 load/load atomic/ 12246 atomicrmw-with-return-value. 12247 - s_waitcnt vscnt(0) 12248 must happen after 12249 any preceding 12250 global/generic 12251 store/store atomic/ 12252 atomicrmw-no-return-value. 12253 - s_waitcnt lgkmcnt(0) 12254 must happen after 12255 any preceding 12256 local/generic 12257 load/store/load 12258 atomic/store 12259 atomic/atomicrmw. 12260 - Must happen before 12261 any following store 12262 atomic/atomicrmw 12263 with an equal or 12264 wider sync scope 12265 and memory ordering 12266 stronger than 12267 unordered (this is 12268 termed the 12269 fence-paired-atomic). 12270 - Ensures that all 12271 memory operations 12272 have 12273 completed before 12274 performing the 12275 following 12276 fence-paired-atomic. 12277 12278 **Acquire-Release Atomic** 12279 ------------------------------------------------------------------------------------ 12280 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic 12281 - wavefront - local 12282 - generic 12283 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12284 vmcnt(0) & vscnt(0) 12285 12286 - If CU wavefront execution 12287 mode, omit vmcnt(0) and 12288 vscnt(0). 12289 - If OpenCL, omit 12290 lgkmcnt(0). 12291 - Must happen after 12292 any preceding 12293 local/generic 12294 load/store/load 12295 atomic/store 12296 atomic/atomicrmw. 12297 - Could be split into 12298 separate s_waitcnt 12299 vmcnt(0), s_waitcnt 12300 vscnt(0), and s_waitcnt 12301 lgkmcnt(0) to allow 12302 them to be 12303 independently moved 12304 according to the 12305 following rules. 12306 - s_waitcnt vmcnt(0) 12307 must happen after 12308 any preceding 12309 global/generic load/load 12310 atomic/ 12311 atomicrmw-with-return-value. 12312 - s_waitcnt vscnt(0) 12313 must happen after 12314 any preceding 12315 global/generic 12316 store/store 12317 atomic/ 12318 atomicrmw-no-return-value. 12319 - s_waitcnt lgkmcnt(0) 12320 must happen after 12321 any preceding 12322 local/generic 12323 load/store/load 12324 atomic/store 12325 atomic/atomicrmw. 12326 - Must happen before 12327 the following 12328 atomicrmw. 12329 - Ensures that all 12330 memory operations 12331 have 12332 completed before 12333 performing the 12334 atomicrmw that is 12335 being released. 12336 12337 2. buffer/global_atomic 12338 3. s_waitcnt vm/vscnt(0) 12339 12340 - If CU wavefront execution 12341 mode, omit. 12342 - Use vmcnt(0) if atomic with 12343 return and vscnt(0) if 12344 atomic with no-return. 12345 - Must happen before 12346 the following 12347 buffer_gl0_inv. 12348 - Ensures any 12349 following global 12350 data read is no 12351 older than the 12352 atomicrmw value 12353 being acquired. 12354 12355 4. buffer_gl0_inv 12356 12357 - If CU wavefront execution 12358 mode, omit. 12359 - Ensures that 12360 following 12361 loads will not see 12362 stale data. 12363 12364 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0) 12365 12366 - If CU wavefront execution 12367 mode, omit. 12368 - If OpenCL, omit. 12369 - Could be split into 12370 separate s_waitcnt 12371 vmcnt(0) and s_waitcnt 12372 vscnt(0) to allow 12373 them to be 12374 independently moved 12375 according to the 12376 following rules. 12377 - s_waitcnt vmcnt(0) 12378 must happen after 12379 any preceding 12380 global/generic load/load 12381 atomic/ 12382 atomicrmw-with-return-value. 12383 - s_waitcnt vscnt(0) 12384 must happen after 12385 any preceding 12386 global/generic 12387 store/store atomic/ 12388 atomicrmw-no-return-value. 12389 - Must happen before 12390 the following 12391 store. 12392 - Ensures that all 12393 global memory 12394 operations have 12395 completed before 12396 performing the 12397 store that is being 12398 released. 12399 12400 2. ds_atomic 12401 3. s_waitcnt lgkmcnt(0) 12402 12403 - If OpenCL, omit. 12404 - Must happen before 12405 the following 12406 buffer_gl0_inv. 12407 - Ensures any 12408 following global 12409 data read is no 12410 older than the local load 12411 atomic value being 12412 acquired. 12413 12414 4. buffer_gl0_inv 12415 12416 - If CU wavefront execution 12417 mode, omit. 12418 - If OpenCL omit. 12419 - Ensures that 12420 following 12421 loads will not see 12422 stale data. 12423 12424 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) & 12425 vmcnt(0) & vscnt(0) 12426 12427 - If CU wavefront execution 12428 mode, omit vmcnt(0) and 12429 vscnt(0). 12430 - If OpenCL, omit lgkmcnt(0). 12431 - Could be split into 12432 separate s_waitcnt 12433 vmcnt(0), s_waitcnt 12434 vscnt(0) and s_waitcnt 12435 lgkmcnt(0) to allow 12436 them to be 12437 independently moved 12438 according to the 12439 following rules. 12440 - s_waitcnt vmcnt(0) 12441 must happen after 12442 any preceding 12443 global/generic load/load 12444 atomic/ 12445 atomicrmw-with-return-value. 12446 - s_waitcnt vscnt(0) 12447 must happen after 12448 any preceding 12449 global/generic 12450 store/store 12451 atomic/ 12452 atomicrmw-no-return-value. 12453 - s_waitcnt lgkmcnt(0) 12454 must happen after 12455 any preceding 12456 local/generic 12457 load/store/load 12458 atomic/store 12459 atomic/atomicrmw. 12460 - Must happen before 12461 the following 12462 atomicrmw. 12463 - Ensures that all 12464 memory operations 12465 have 12466 completed before 12467 performing the 12468 atomicrmw that is 12469 being released. 12470 12471 2. flat_atomic 12472 3. s_waitcnt lgkmcnt(0) & 12473 vmcnt(0) & vscnt(0) 12474 12475 - If CU wavefront execution 12476 mode, omit vmcnt(0) and 12477 vscnt(0). 12478 - If OpenCL, omit lgkmcnt(0). 12479 - Must happen before 12480 the following 12481 buffer_gl0_inv. 12482 - Ensures any 12483 following global 12484 data read is no 12485 older than the load 12486 atomic value being 12487 acquired. 12488 12489 3. buffer_gl0_inv 12490 12491 - If CU wavefront execution 12492 mode, omit. 12493 - Ensures that 12494 following 12495 loads will not see 12496 stale data. 12497 12498 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) & 12499 - system vmcnt(0) & vscnt(0) 12500 12501 - If OpenCL, omit 12502 lgkmcnt(0). 12503 - Could be split into 12504 separate s_waitcnt 12505 vmcnt(0), s_waitcnt 12506 vscnt(0) and s_waitcnt 12507 lgkmcnt(0) to allow 12508 them to be 12509 independently moved 12510 according to the 12511 following rules. 12512 - s_waitcnt vmcnt(0) 12513 must happen after 12514 any preceding 12515 global/generic 12516 load/load atomic/ 12517 atomicrmw-with-return-value. 12518 - s_waitcnt vscnt(0) 12519 must happen after 12520 any preceding 12521 global/generic 12522 store/store atomic/ 12523 atomicrmw-no-return-value. 12524 - s_waitcnt lgkmcnt(0) 12525 must happen after 12526 any preceding 12527 local/generic 12528 load/store/load 12529 atomic/store 12530 atomic/atomicrmw. 12531 - Must happen before 12532 the following 12533 atomicrmw. 12534 - Ensures that all 12535 memory operations 12536 to global have 12537 completed before 12538 performing the 12539 atomicrmw that is 12540 being released. 12541 12542 2. buffer/global_atomic 12543 3. s_waitcnt vm/vscnt(0) 12544 12545 - Use vmcnt(0) if atomic with 12546 return and vscnt(0) if 12547 atomic with no-return. 12548 - Must happen before 12549 following 12550 buffer_gl*_inv. 12551 - Ensures the 12552 atomicrmw has 12553 completed before 12554 invalidating the 12555 caches. 12556 12557 4. buffer_gl0_inv; 12558 buffer_gl1_inv 12559 12560 - Must happen before 12561 any following 12562 global/generic 12563 load/load 12564 atomic/atomicrmw. 12565 - Ensures that 12566 following loads 12567 will not see stale 12568 global data. 12569 12570 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) & 12571 - system vmcnt(0) & vscnt(0) 12572 12573 - If OpenCL, omit 12574 lgkmcnt(0). 12575 - Could be split into 12576 separate s_waitcnt 12577 vmcnt(0), s_waitcnt 12578 vscnt(0), and s_waitcnt 12579 lgkmcnt(0) to allow 12580 them to be 12581 independently moved 12582 according to the 12583 following rules. 12584 - s_waitcnt vmcnt(0) 12585 must happen after 12586 any preceding 12587 global/generic 12588 load/load atomic 12589 atomicrmw-with-return-value. 12590 - s_waitcnt vscnt(0) 12591 must happen after 12592 any preceding 12593 global/generic 12594 store/store atomic/ 12595 atomicrmw-no-return-value. 12596 - s_waitcnt lgkmcnt(0) 12597 must happen after 12598 any preceding 12599 local/generic 12600 load/store/load 12601 atomic/store 12602 atomic/atomicrmw. 12603 - Must happen before 12604 the following 12605 atomicrmw. 12606 - Ensures that all 12607 memory operations 12608 have 12609 completed before 12610 performing the 12611 atomicrmw that is 12612 being released. 12613 12614 2. flat_atomic 12615 3. s_waitcnt vm/vscnt(0) & 12616 lgkmcnt(0) 12617 12618 - If OpenCL, omit 12619 lgkmcnt(0). 12620 - Use vmcnt(0) if atomic with 12621 return and vscnt(0) if 12622 atomic with no-return. 12623 - Must happen before 12624 following 12625 buffer_gl*_inv. 12626 - Ensures the 12627 atomicrmw has 12628 completed before 12629 invalidating the 12630 caches. 12631 12632 4. buffer_gl0_inv; 12633 buffer_gl1_inv 12634 12635 - Must happen before 12636 any following 12637 global/generic 12638 load/load 12639 atomic/atomicrmw. 12640 - Ensures that 12641 following loads 12642 will not see stale 12643 global data. 12644 12645 fence acq_rel - singlethread *none* *none* 12646 - wavefront 12647 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) & 12648 vmcnt(0) & vscnt(0) 12649 12650 - If CU wavefront execution 12651 mode, omit vmcnt(0) and 12652 vscnt(0). 12653 - If OpenCL and 12654 address space is 12655 not generic, omit 12656 lgkmcnt(0). 12657 - If OpenCL and 12658 address space is 12659 local, omit 12660 vmcnt(0) and vscnt(0). 12661 - However, 12662 since LLVM 12663 currently has no 12664 address space on 12665 the fence need to 12666 conservatively 12667 always generate 12668 (see comment for 12669 previous fence). 12670 - Could be split into 12671 separate s_waitcnt 12672 vmcnt(0), s_waitcnt 12673 vscnt(0) and s_waitcnt 12674 lgkmcnt(0) to allow 12675 them to be 12676 independently moved 12677 according to the 12678 following rules. 12679 - s_waitcnt vmcnt(0) 12680 must happen after 12681 any preceding 12682 global/generic 12683 load/load 12684 atomic/ 12685 atomicrmw-with-return-value. 12686 - s_waitcnt vscnt(0) 12687 must happen after 12688 any preceding 12689 global/generic 12690 store/store atomic/ 12691 atomicrmw-no-return-value. 12692 - s_waitcnt lgkmcnt(0) 12693 must happen after 12694 any preceding 12695 local/generic 12696 load/store/load 12697 atomic/store atomic/ 12698 atomicrmw. 12699 - Must happen before 12700 any following 12701 global/generic 12702 load/load 12703 atomic/store/store 12704 atomic/atomicrmw. 12705 - Ensures that all 12706 memory operations 12707 have 12708 completed before 12709 performing any 12710 following global 12711 memory operations. 12712 - Ensures that the 12713 preceding 12714 local/generic load 12715 atomic/atomicrmw 12716 with an equal or 12717 wider sync scope 12718 and memory ordering 12719 stronger than 12720 unordered (this is 12721 termed the 12722 acquire-fence-paired-atomic) 12723 has completed 12724 before following 12725 global memory 12726 operations. This 12727 satisfies the 12728 requirements of 12729 acquire. 12730 - Ensures that all 12731 previous memory 12732 operations have 12733 completed before a 12734 following 12735 local/generic store 12736 atomic/atomicrmw 12737 with an equal or 12738 wider sync scope 12739 and memory ordering 12740 stronger than 12741 unordered (this is 12742 termed the 12743 release-fence-paired-atomic). 12744 This satisfies the 12745 requirements of 12746 release. 12747 - Must happen before 12748 the following 12749 buffer_gl0_inv. 12750 - Ensures that the 12751 acquire-fence-paired 12752 atomic has completed 12753 before invalidating 12754 the 12755 cache. Therefore 12756 any following 12757 locations read must 12758 be no older than 12759 the value read by 12760 the 12761 acquire-fence-paired-atomic. 12762 12763 3. buffer_gl0_inv 12764 12765 - If CU wavefront execution 12766 mode, omit. 12767 - Ensures that 12768 following 12769 loads will not see 12770 stale data. 12771 12772 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) & 12773 - system vmcnt(0) & vscnt(0) 12774 12775 - If OpenCL and 12776 address space is 12777 not generic, omit 12778 lgkmcnt(0). 12779 - If OpenCL and 12780 address space is 12781 local, omit 12782 vmcnt(0) and vscnt(0). 12783 - However, since LLVM 12784 currently has no 12785 address space on 12786 the fence need to 12787 conservatively 12788 always generate 12789 (see comment for 12790 previous fence). 12791 - Could be split into 12792 separate s_waitcnt 12793 vmcnt(0), s_waitcnt 12794 vscnt(0) and s_waitcnt 12795 lgkmcnt(0) to allow 12796 them to be 12797 independently moved 12798 according to the 12799 following rules. 12800 - s_waitcnt vmcnt(0) 12801 must happen after 12802 any preceding 12803 global/generic 12804 load/load 12805 atomic/ 12806 atomicrmw-with-return-value. 12807 - s_waitcnt vscnt(0) 12808 must happen after 12809 any preceding 12810 global/generic 12811 store/store atomic/ 12812 atomicrmw-no-return-value. 12813 - s_waitcnt lgkmcnt(0) 12814 must happen after 12815 any preceding 12816 local/generic 12817 load/store/load 12818 atomic/store 12819 atomic/atomicrmw. 12820 - Must happen before 12821 the following 12822 buffer_gl*_inv. 12823 - Ensures that the 12824 preceding 12825 global/local/generic 12826 load 12827 atomic/atomicrmw 12828 with an equal or 12829 wider sync scope 12830 and memory ordering 12831 stronger than 12832 unordered (this is 12833 termed the 12834 acquire-fence-paired-atomic) 12835 has completed 12836 before invalidating 12837 the caches. This 12838 satisfies the 12839 requirements of 12840 acquire. 12841 - Ensures that all 12842 previous memory 12843 operations have 12844 completed before a 12845 following 12846 global/local/generic 12847 store 12848 atomic/atomicrmw 12849 with an equal or 12850 wider sync scope 12851 and memory ordering 12852 stronger than 12853 unordered (this is 12854 termed the 12855 release-fence-paired-atomic). 12856 This satisfies the 12857 requirements of 12858 release. 12859 12860 2. buffer_gl0_inv; 12861 buffer_gl1_inv 12862 12863 - Must happen before 12864 any following 12865 global/generic 12866 load/load 12867 atomic/store/store 12868 atomic/atomicrmw. 12869 - Ensures that 12870 following loads 12871 will not see stale 12872 global data. This 12873 satisfies the 12874 requirements of 12875 acquire. 12876 12877 **Sequential Consistent Atomic** 12878 ------------------------------------------------------------------------------------ 12879 load atomic seq_cst - singlethread - global *Same as corresponding 12880 - wavefront - local load atomic acquire, 12881 - generic except must generate 12882 all instructions even 12883 for OpenCL.* 12884 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) & 12885 - generic vmcnt(0) & vscnt(0) 12886 12887 - If CU wavefront execution 12888 mode, omit vmcnt(0) and 12889 vscnt(0). 12890 - Could be split into 12891 separate s_waitcnt 12892 vmcnt(0), s_waitcnt 12893 vscnt(0), and s_waitcnt 12894 lgkmcnt(0) to allow 12895 them to be 12896 independently moved 12897 according to the 12898 following rules. 12899 - s_waitcnt lgkmcnt(0) must 12900 happen after 12901 preceding 12902 local/generic load 12903 atomic/store 12904 atomic/atomicrmw 12905 with memory 12906 ordering of seq_cst 12907 and with equal or 12908 wider sync scope. 12909 (Note that seq_cst 12910 fences have their 12911 own s_waitcnt 12912 lgkmcnt(0) and so do 12913 not need to be 12914 considered.) 12915 - s_waitcnt vmcnt(0) 12916 must happen after 12917 preceding 12918 global/generic load 12919 atomic/ 12920 atomicrmw-with-return-value 12921 with memory 12922 ordering of seq_cst 12923 and with equal or 12924 wider sync scope. 12925 (Note that seq_cst 12926 fences have their 12927 own s_waitcnt 12928 vmcnt(0) and so do 12929 not need to be 12930 considered.) 12931 - s_waitcnt vscnt(0) 12932 Must happen after 12933 preceding 12934 global/generic store 12935 atomic/ 12936 atomicrmw-no-return-value 12937 with memory 12938 ordering of seq_cst 12939 and with equal or 12940 wider sync scope. 12941 (Note that seq_cst 12942 fences have their 12943 own s_waitcnt 12944 vscnt(0) and so do 12945 not need to be 12946 considered.) 12947 - Ensures any 12948 preceding 12949 sequential 12950 consistent global/local 12951 memory instructions 12952 have completed 12953 before executing 12954 this sequentially 12955 consistent 12956 instruction. This 12957 prevents reordering 12958 a seq_cst store 12959 followed by a 12960 seq_cst load. (Note 12961 that seq_cst is 12962 stronger than 12963 acquire/release as 12964 the reordering of 12965 load acquire 12966 followed by a store 12967 release is 12968 prevented by the 12969 s_waitcnt of 12970 the release, but 12971 there is nothing 12972 preventing a store 12973 release followed by 12974 load acquire from 12975 completing out of 12976 order. The s_waitcnt 12977 could be placed after 12978 seq_store or before 12979 the seq_load. We 12980 choose the load to 12981 make the s_waitcnt be 12982 as late as possible 12983 so that the store 12984 may have already 12985 completed.) 12986 12987 2. *Following 12988 instructions same as 12989 corresponding load 12990 atomic acquire, 12991 except must generate 12992 all instructions even 12993 for OpenCL.* 12994 load atomic seq_cst - workgroup - local 12995 12996 1. s_waitcnt vmcnt(0) & vscnt(0) 12997 12998 - If CU wavefront execution 12999 mode, omit. 13000 - Could be split into 13001 separate s_waitcnt 13002 vmcnt(0) and s_waitcnt 13003 vscnt(0) to allow 13004 them to be 13005 independently moved 13006 according to the 13007 following rules. 13008 - s_waitcnt vmcnt(0) 13009 Must happen after 13010 preceding 13011 global/generic load 13012 atomic/ 13013 atomicrmw-with-return-value 13014 with memory 13015 ordering of seq_cst 13016 and with equal or 13017 wider sync scope. 13018 (Note that seq_cst 13019 fences have their 13020 own s_waitcnt 13021 vmcnt(0) and so do 13022 not need to be 13023 considered.) 13024 - s_waitcnt vscnt(0) 13025 Must happen after 13026 preceding 13027 global/generic store 13028 atomic/ 13029 atomicrmw-no-return-value 13030 with memory 13031 ordering of seq_cst 13032 and with equal or 13033 wider sync scope. 13034 (Note that seq_cst 13035 fences have their 13036 own s_waitcnt 13037 vscnt(0) and so do 13038 not need to be 13039 considered.) 13040 - Ensures any 13041 preceding 13042 sequential 13043 consistent global 13044 memory instructions 13045 have completed 13046 before executing 13047 this sequentially 13048 consistent 13049 instruction. This 13050 prevents reordering 13051 a seq_cst store 13052 followed by a 13053 seq_cst load. (Note 13054 that seq_cst is 13055 stronger than 13056 acquire/release as 13057 the reordering of 13058 load acquire 13059 followed by a store 13060 release is 13061 prevented by the 13062 s_waitcnt of 13063 the release, but 13064 there is nothing 13065 preventing a store 13066 release followed by 13067 load acquire from 13068 completing out of 13069 order. The s_waitcnt 13070 could be placed after 13071 seq_store or before 13072 the seq_load. We 13073 choose the load to 13074 make the s_waitcnt be 13075 as late as possible 13076 so that the store 13077 may have already 13078 completed.) 13079 13080 2. *Following 13081 instructions same as 13082 corresponding load 13083 atomic acquire, 13084 except must generate 13085 all instructions even 13086 for OpenCL.* 13087 13088 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) & 13089 - system - generic vmcnt(0) & vscnt(0) 13090 13091 - Could be split into 13092 separate s_waitcnt 13093 vmcnt(0), s_waitcnt 13094 vscnt(0) and s_waitcnt 13095 lgkmcnt(0) to allow 13096 them to be 13097 independently moved 13098 according to the 13099 following rules. 13100 - s_waitcnt lgkmcnt(0) 13101 must happen after 13102 preceding 13103 local load 13104 atomic/store 13105 atomic/atomicrmw 13106 with memory 13107 ordering of seq_cst 13108 and with equal or 13109 wider sync scope. 13110 (Note that seq_cst 13111 fences have their 13112 own s_waitcnt 13113 lgkmcnt(0) and so do 13114 not need to be 13115 considered.) 13116 - s_waitcnt vmcnt(0) 13117 must happen after 13118 preceding 13119 global/generic load 13120 atomic/ 13121 atomicrmw-with-return-value 13122 with memory 13123 ordering of seq_cst 13124 and with equal or 13125 wider sync scope. 13126 (Note that seq_cst 13127 fences have their 13128 own s_waitcnt 13129 vmcnt(0) and so do 13130 not need to be 13131 considered.) 13132 - s_waitcnt vscnt(0) 13133 Must happen after 13134 preceding 13135 global/generic store 13136 atomic/ 13137 atomicrmw-no-return-value 13138 with memory 13139 ordering of seq_cst 13140 and with equal or 13141 wider sync scope. 13142 (Note that seq_cst 13143 fences have their 13144 own s_waitcnt 13145 vscnt(0) and so do 13146 not need to be 13147 considered.) 13148 - Ensures any 13149 preceding 13150 sequential 13151 consistent global 13152 memory instructions 13153 have completed 13154 before executing 13155 this sequentially 13156 consistent 13157 instruction. This 13158 prevents reordering 13159 a seq_cst store 13160 followed by a 13161 seq_cst load. (Note 13162 that seq_cst is 13163 stronger than 13164 acquire/release as 13165 the reordering of 13166 load acquire 13167 followed by a store 13168 release is 13169 prevented by the 13170 s_waitcnt of 13171 the release, but 13172 there is nothing 13173 preventing a store 13174 release followed by 13175 load acquire from 13176 completing out of 13177 order. The s_waitcnt 13178 could be placed after 13179 seq_store or before 13180 the seq_load. We 13181 choose the load to 13182 make the s_waitcnt be 13183 as late as possible 13184 so that the store 13185 may have already 13186 completed.) 13187 13188 2. *Following 13189 instructions same as 13190 corresponding load 13191 atomic acquire, 13192 except must generate 13193 all instructions even 13194 for OpenCL.* 13195 store atomic seq_cst - singlethread - global *Same as corresponding 13196 - wavefront - local store atomic release, 13197 - workgroup - generic except must generate 13198 - agent all instructions even 13199 - system for OpenCL.* 13200 atomicrmw seq_cst - singlethread - global *Same as corresponding 13201 - wavefront - local atomicrmw acq_rel, 13202 - workgroup - generic except must generate 13203 - agent all instructions even 13204 - system for OpenCL.* 13205 fence seq_cst - singlethread *none* *Same as corresponding 13206 - wavefront fence acq_rel, 13207 - workgroup except must generate 13208 - agent all instructions even 13209 - system for OpenCL.* 13210 ============ ============ ============== ========== ================================ 13211 13212.. _amdgpu-amdhsa-trap-handler-abi: 13213 13214Trap Handler ABI 13215~~~~~~~~~~~~~~~~ 13216 13217For code objects generated by the AMDGPU backend for HSA [HSA]_ compatible 13218runtimes (see :ref:`amdgpu-os`), the runtime installs a trap handler that 13219supports the ``s_trap`` instruction. For usage see: 13220 13221- :ref:`amdgpu-trap-handler-for-amdhsa-os-v2-table` 13222- :ref:`amdgpu-trap-handler-for-amdhsa-os-v3-table` 13223- :ref:`amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table` 13224 13225 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V2 13226 :name: amdgpu-trap-handler-for-amdhsa-os-v2-table 13227 13228 =================== =============== =============== ======================================= 13229 Usage Code Sequence Trap Handler Description 13230 Inputs 13231 =================== =============== =============== ======================================= 13232 reserved ``s_trap 0x00`` Reserved by hardware. 13233 ``debugtrap(arg)`` ``s_trap 0x01`` ``SGPR0-1``: Reserved for Finalizer HSA ``debugtrap`` 13234 ``queue_ptr`` intrinsic (not implemented). 13235 ``VGPR0``: 13236 ``arg`` 13237 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13238 ``queue_ptr`` the trap instruction. The associated 13239 queue is signalled to put it into the 13240 error state. When the queue is put in 13241 the error state, the waves executing 13242 dispatches on the queue will be 13243 terminated. 13244 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13245 as a no-operation. The trap handler 13246 is entered and immediately returns to 13247 continue execution of the wavefront. 13248 - If the debugger is enabled, causes 13249 the debug trap to be reported by the 13250 debugger and the wavefront is put in 13251 the halt state with the PC at the 13252 instruction. The debugger must 13253 increment the PC and resume the wave. 13254 reserved ``s_trap 0x04`` Reserved. 13255 reserved ``s_trap 0x05`` Reserved. 13256 reserved ``s_trap 0x06`` Reserved. 13257 reserved ``s_trap 0x07`` Reserved. 13258 reserved ``s_trap 0x08`` Reserved. 13259 reserved ``s_trap 0xfe`` Reserved. 13260 reserved ``s_trap 0xff`` Reserved. 13261 =================== =============== =============== ======================================= 13262 13263.. 13264 13265 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V3 13266 :name: amdgpu-trap-handler-for-amdhsa-os-v3-table 13267 13268 =================== =============== =============== ======================================= 13269 Usage Code Sequence Trap Handler Description 13270 Inputs 13271 =================== =============== =============== ======================================= 13272 reserved ``s_trap 0x00`` Reserved by hardware. 13273 debugger breakpoint ``s_trap 0x01`` *none* Reserved for debugger to use for 13274 breakpoints. Causes wave to be halted 13275 with the PC at the trap instruction. 13276 The debugger is responsible to resume 13277 the wave, including the instruction 13278 that the breakpoint overwrote. 13279 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: Causes wave to be halted with the PC at 13280 ``queue_ptr`` the trap instruction. The associated 13281 queue is signalled to put it into the 13282 error state. When the queue is put in 13283 the error state, the waves executing 13284 dispatches on the queue will be 13285 terminated. 13286 ``llvm.debugtrap`` ``s_trap 0x03`` *none* - If debugger not enabled then behaves 13287 as a no-operation. The trap handler 13288 is entered and immediately returns to 13289 continue execution of the wavefront. 13290 - If the debugger is enabled, causes 13291 the debug trap to be reported by the 13292 debugger and the wavefront is put in 13293 the halt state with the PC at the 13294 instruction. The debugger must 13295 increment the PC and resume the wave. 13296 reserved ``s_trap 0x04`` Reserved. 13297 reserved ``s_trap 0x05`` Reserved. 13298 reserved ``s_trap 0x06`` Reserved. 13299 reserved ``s_trap 0x07`` Reserved. 13300 reserved ``s_trap 0x08`` Reserved. 13301 reserved ``s_trap 0xfe`` Reserved. 13302 reserved ``s_trap 0xff`` Reserved. 13303 =================== =============== =============== ======================================= 13304 13305.. 13306 13307 .. table:: AMDGPU Trap Handler for AMDHSA OS Code Object V4 and Above 13308 :name: amdgpu-trap-handler-for-amdhsa-os-v4-onwards-table 13309 13310 =================== =============== ================ ================= ======================================= 13311 Usage Code Sequence GFX6-GFX8 Inputs GFX9-GFX10 Inputs Description 13312 =================== =============== ================ ================= ======================================= 13313 reserved ``s_trap 0x00`` Reserved by hardware. 13314 debugger breakpoint ``s_trap 0x01`` *none* *none* Reserved for debugger to use for 13315 breakpoints. Causes wave to be halted 13316 with the PC at the trap instruction. 13317 The debugger is responsible to resume 13318 the wave, including the instruction 13319 that the breakpoint overwrote. 13320 ``llvm.trap`` ``s_trap 0x02`` ``SGPR0-1``: *none* Causes wave to be halted with the PC at 13321 ``queue_ptr`` the trap instruction. The associated 13322 queue is signalled to put it into the 13323 error state. When the queue is put in 13324 the error state, the waves executing 13325 dispatches on the queue will be 13326 terminated. 13327 ``llvm.debugtrap`` ``s_trap 0x03`` *none* *none* - If debugger not enabled then behaves 13328 as a no-operation. The trap handler 13329 is entered and immediately returns to 13330 continue execution of the wavefront. 13331 - If the debugger is enabled, causes 13332 the debug trap to be reported by the 13333 debugger and the wavefront is put in 13334 the halt state with the PC at the 13335 instruction. The debugger must 13336 increment the PC and resume the wave. 13337 reserved ``s_trap 0x04`` Reserved. 13338 reserved ``s_trap 0x05`` Reserved. 13339 reserved ``s_trap 0x06`` Reserved. 13340 reserved ``s_trap 0x07`` Reserved. 13341 reserved ``s_trap 0x08`` Reserved. 13342 reserved ``s_trap 0xfe`` Reserved. 13343 reserved ``s_trap 0xff`` Reserved. 13344 =================== =============== ================ ================= ======================================= 13345 13346.. _amdgpu-amdhsa-function-call-convention: 13347 13348Call Convention 13349~~~~~~~~~~~~~~~ 13350 13351.. note:: 13352 13353 This section is currently incomplete and has inaccuracies. It is WIP that will 13354 be updated as information is determined. 13355 13356See :ref:`amdgpu-dwarf-address-space-identifier` for information on swizzled 13357addresses. Unswizzled addresses are normal linear addresses. 13358 13359.. _amdgpu-amdhsa-function-call-convention-kernel-functions: 13360 13361Kernel Functions 13362++++++++++++++++ 13363 13364This section describes the call convention ABI for the outer kernel function. 13365 13366See :ref:`amdgpu-amdhsa-initial-kernel-execution-state` for the kernel call 13367convention. 13368 13369The following is not part of the AMDGPU kernel calling convention but describes 13370how the AMDGPU implements function calls: 13371 133721. Clang decides the kernarg layout to match the *HSA Programmer's Language 13373 Reference* [HSA]_. 13374 13375 - All structs are passed directly. 13376 - Lambda values are passed *TBA*. 13377 13378 .. TODO:: 13379 13380 - Does this really follow HSA rules? Or are structs >16 bytes passed 13381 by-value struct? 13382 - What is ABI for lambda values? 13383 133844. The kernel performs certain setup in its prolog, as described in 13385 :ref:`amdgpu-amdhsa-kernel-prolog`. 13386 13387.. _amdgpu-amdhsa-function-call-convention-non-kernel-functions: 13388 13389Non-Kernel Functions 13390++++++++++++++++++++ 13391 13392This section describes the call convention ABI for functions other than the 13393outer kernel function. 13394 13395If a kernel has function calls then scratch is always allocated and used for 13396the call stack which grows from low address to high address using the swizzled 13397scratch address space. 13398 13399On entry to a function: 13400 134011. SGPR0-3 contain a V# with the following properties (see 13402 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`): 13403 13404 * Base address pointing to the beginning of the wavefront scratch backing 13405 memory. 13406 * Swizzled with dword element size and stride of wavefront size elements. 13407 134082. The FLAT_SCRATCH register pair is setup. See 13409 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`. 134103. GFX6-GFX8: M0 register set to the size of LDS in bytes. See 13411 :ref:`amdgpu-amdhsa-kernel-prolog-m0`. 134124. The EXEC register is set to the lanes active on entry to the function. 134135. MODE register: *TBD* 134146. VGPR0-31 and SGPR4-29 are used to pass function input arguments as described 13415 below. 134167. SGPR30-31 return address (RA). The code address that the function must 13417 return to when it completes. The value is undefined if the function is *no 13418 return*. 134198. SGPR32 is used for the stack pointer (SP). It is an unswizzled scratch 13420 offset relative to the beginning of the wavefront scratch backing memory. 13421 13422 The unswizzled SP can be used with buffer instructions as an unswizzled SGPR 13423 offset with the scratch V# in SGPR0-3 to access the stack in a swizzled 13424 manner. 13425 13426 The unswizzled SP value can be converted into the swizzled SP value by: 13427 13428 | swizzled SP = unswizzled SP / wavefront size 13429 13430 This may be used to obtain the private address space address of stack 13431 objects and to convert this address to a flat address by adding the flat 13432 scratch aperture base address. 13433 13434 The swizzled SP value is always 4 bytes aligned for the ``r600`` 13435 architecture and 16 byte aligned for the ``amdgcn`` architecture. 13436 13437 .. note:: 13438 13439 The ``amdgcn`` value is selected to avoid dynamic stack alignment for the 13440 OpenCL language which has the largest base type defined as 16 bytes. 13441 13442 On entry, the swizzled SP value is the address of the first function 13443 argument passed on the stack. Other stack passed arguments are positive 13444 offsets from the entry swizzled SP value. 13445 13446 The function may use positive offsets beyond the last stack passed argument 13447 for stack allocated local variables and register spill slots. If necessary, 13448 the function may align these to greater alignment than 16 bytes. After these 13449 the function may dynamically allocate space for such things as runtime sized 13450 ``alloca`` local allocations. 13451 13452 If the function calls another function, it will place any stack allocated 13453 arguments after the last local allocation and adjust SGPR32 to the address 13454 after the last local allocation. 13455 134569. All other registers are unspecified. 1345710. Any necessary ``s_waitcnt`` has been performed to ensure memory is available 13458 to the function. 13459 13460On exit from a function: 13461 134621. VGPR0-31 and SGPR4-29 are used to pass function result arguments as 13463 described below. Any registers used are considered clobbered registers. 134642. The following registers are preserved and have the same value as on entry: 13465 13466 * FLAT_SCRATCH 13467 * EXEC 13468 * GFX6-GFX8: M0 13469 * All SGPR registers except the clobbered registers of SGPR4-31. 13470 * VGPR40-47 13471 * VGPR56-63 13472 * VGPR72-79 13473 * VGPR88-95 13474 * VGPR104-111 13475 * VGPR120-127 13476 * VGPR136-143 13477 * VGPR152-159 13478 * VGPR168-175 13479 * VGPR184-191 13480 * VGPR200-207 13481 * VGPR216-223 13482 * VGPR232-239 13483 * VGPR248-255 13484 13485 .. note:: 13486 13487 Except the argument registers, the VGPRs clobbered and the preserved 13488 registers are intermixed at regular intervals in order to keep a 13489 similar ratio independent of the number of allocated VGPRs. 13490 13491 * GFX90A: All AGPR registers except the clobbered registers AGPR0-31. 13492 * Lanes of all VGPRs that are inactive at the call site. 13493 13494 For the AMDGPU backend, an inter-procedural register allocation (IPRA) 13495 optimization may mark some of clobbered SGPR and VGPR registers as 13496 preserved if it can be determined that the called function does not change 13497 their value. 13498 134992. The PC is set to the RA provided on entry. 135003. MODE register: *TBD*. 135014. All other registers are clobbered. 135025. Any necessary ``s_waitcnt`` has been performed to ensure memory accessed by 13503 function is available to the caller. 13504 13505.. TODO:: 13506 13507 - How are function results returned? The address of structured types is passed 13508 by reference, but what about other types? 13509 13510The function input arguments are made up of the formal arguments explicitly 13511declared by the source language function plus the implicit input arguments used 13512by the implementation. 13513 13514The source language input arguments are: 13515 135161. Any source language implicit ``this`` or ``self`` argument comes first as a 13517 pointer type. 135182. Followed by the function formal arguments in left to right source order. 13519 13520The source language result arguments are: 13521 135221. The function result argument. 13523 13524The source language input or result struct type arguments that are less than or 13525equal to 16 bytes, are decomposed recursively into their base type fields, and 13526each field is passed as if a separate argument. For input arguments, if the 13527called function requires the struct to be in memory, for example because its 13528address is taken, then the function body is responsible for allocating a stack 13529location and copying the field arguments into it. Clang terms this *direct 13530struct*. 13531 13532The source language input struct type arguments that are greater than 16 bytes, 13533are passed by reference. The caller is responsible for allocating a stack 13534location to make a copy of the struct value and pass the address as the input 13535argument. The called function is responsible to perform the dereference when 13536accessing the input argument. Clang terms this *by-value struct*. 13537 13538A source language result struct type argument that is greater than 16 bytes, is 13539returned by reference. The caller is responsible for allocating a stack location 13540to hold the result value and passes the address as the last input argument 13541(before the implicit input arguments). In this case there are no result 13542arguments. The called function is responsible to perform the dereference when 13543storing the result value. Clang terms this *structured return (sret)*. 13544 13545*TODO: correct the ``sret`` definition.* 13546 13547.. TODO:: 13548 13549 Is this definition correct? Or is ``sret`` only used if passing in registers, and 13550 pass as non-decomposed struct as stack argument? Or something else? Is the 13551 memory location in the caller stack frame, or a stack memory argument and so 13552 no address is passed as the caller can directly write to the argument stack 13553 location? But then the stack location is still live after return. If an 13554 argument stack location is it the first stack argument or the last one? 13555 13556Lambda argument types are treated as struct types with an implementation defined 13557set of fields. 13558 13559.. TODO:: 13560 13561 Need to specify the ABI for lambda types for AMDGPU. 13562 13563For AMDGPU backend all source language arguments (including the decomposed 13564struct type arguments) are passed in VGPRs unless marked ``inreg`` in which case 13565they are passed in SGPRs. 13566 13567The AMDGPU backend walks the function call graph from the leaves to determine 13568which implicit input arguments are used, propagating to each caller of the 13569function. The used implicit arguments are appended to the function arguments 13570after the source language arguments in the following order: 13571 13572.. TODO:: 13573 13574 Is recursion or external functions supported? 13575 135761. Work-Item ID (1 VGPR) 13577 13578 The X, Y and Z work-item ID are packed into a single VGRP with the following 13579 layout. Only fields actually used by the function are set. The other bits 13580 are undefined. 13581 13582 The values come from the initial kernel execution state. See 13583 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`. 13584 13585 .. table:: Work-item implicit argument layout 13586 :name: amdgpu-amdhsa-workitem-implicit-argument-layout-table 13587 13588 ======= ======= ============== 13589 Bits Size Field Name 13590 ======= ======= ============== 13591 9:0 10 bits X Work-Item ID 13592 19:10 10 bits Y Work-Item ID 13593 29:20 10 bits Z Work-Item ID 13594 31:30 2 bits Unused 13595 ======= ======= ============== 13596 135972. Dispatch Ptr (2 SGPRs) 13598 13599 The value comes from the initial kernel execution state. See 13600 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13601 136023. Queue Ptr (2 SGPRs) 13603 13604 The value comes from the initial kernel execution state. See 13605 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13606 136074. Kernarg Segment Ptr (2 SGPRs) 13608 13609 The value comes from the initial kernel execution state. See 13610 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13611 136125. Dispatch id (2 SGPRs) 13613 13614 The value comes from the initial kernel execution state. See 13615 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13616 136176. Work-Group ID X (1 SGPR) 13618 13619 The value comes from the initial kernel execution state. See 13620 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13621 136227. Work-Group ID Y (1 SGPR) 13623 13624 The value comes from the initial kernel execution state. See 13625 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13626 136278. Work-Group ID Z (1 SGPR) 13628 13629 The value comes from the initial kernel execution state. See 13630 :ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`. 13631 136329. Implicit Argument Ptr (2 SGPRs) 13633 13634 The value is computed by adding an offset to Kernarg Segment Ptr to get the 13635 global address space pointer to the first kernarg implicit argument. 13636 13637The input and result arguments are assigned in order in the following manner: 13638 13639.. note:: 13640 13641 There are likely some errors and omissions in the following description that 13642 need correction. 13643 13644 .. TODO:: 13645 13646 Check the Clang source code to decipher how function arguments and return 13647 results are handled. Also see the AMDGPU specific values used. 13648 13649* VGPR arguments are assigned to consecutive VGPRs starting at VGPR0 up to 13650 VGPR31. 13651 13652 If there are more arguments than will fit in these registers, the remaining 13653 arguments are allocated on the stack in order on naturally aligned 13654 addresses. 13655 13656 .. TODO:: 13657 13658 How are overly aligned structures allocated on the stack? 13659 13660* SGPR arguments are assigned to consecutive SGPRs starting at SGPR0 up to 13661 SGPR29. 13662 13663 If there are more arguments than will fit in these registers, the remaining 13664 arguments are allocated on the stack in order on naturally aligned 13665 addresses. 13666 13667Note that decomposed struct type arguments may have some fields passed in 13668registers and some in memory. 13669 13670.. TODO:: 13671 13672 So, a struct which can pass some fields as decomposed register arguments, will 13673 pass the rest as decomposed stack elements? But an argument that will not start 13674 in registers will not be decomposed and will be passed as a non-decomposed 13675 stack value? 13676 13677The following is not part of the AMDGPU function calling convention but 13678describes how the AMDGPU implements function calls: 13679 136801. SGPR33 is used as a frame pointer (FP) if necessary. Like the SP it is an 13681 unswizzled scratch address. It is only needed if runtime sized ``alloca`` 13682 are used, or for the reasons defined in ``SIFrameLowering``. 136832. Runtime stack alignment is supported. SGPR34 is used as a base pointer (BP) 13684 to access the incoming stack arguments in the function. The BP is needed 13685 only when the function requires the runtime stack alignment. 13686 136873. Allocating SGPR arguments on the stack are not supported. 13688 136894. No CFI is currently generated. See 13690 :ref:`amdgpu-dwarf-call-frame-information`. 13691 13692 .. note:: 13693 13694 CFI will be generated that defines the CFA as the unswizzled address 13695 relative to the wave scratch base in the unswizzled private address space 13696 of the lowest address stack allocated local variable. 13697 13698 ``DW_AT_frame_base`` will be defined as the swizzled address in the 13699 swizzled private address space by dividing the CFA by the wavefront size 13700 (since CFA is always at least dword aligned which matches the scratch 13701 swizzle element size). 13702 13703 If no dynamic stack alignment was performed, the stack allocated arguments 13704 are accessed as negative offsets relative to ``DW_AT_frame_base``, and the 13705 local variables and register spill slots are accessed as positive offsets 13706 relative to ``DW_AT_frame_base``. 13707 137085. Function argument passing is implemented by copying the input physical 13709 registers to virtual registers on entry. The register allocator can spill if 13710 necessary. These are copied back to physical registers at call sites. The 13711 net effect is that each function call can have these values in entirely 13712 distinct locations. The IPRA can help avoid shuffling argument registers. 137136. Call sites are implemented by setting up the arguments at positive offsets 13714 from SP. Then SP is incremented to account for the known frame size before 13715 the call and decremented after the call. 13716 13717 .. note:: 13718 13719 The CFI will reflect the changed calculation needed to compute the CFA 13720 from SP. 13721 137227. 4 byte spill slots are used in the stack frame. One slot is allocated for an 13723 emergency spill slot. Buffer instructions are used for stack accesses and 13724 not the ``flat_scratch`` instruction. 13725 13726 .. TODO:: 13727 13728 Explain when the emergency spill slot is used. 13729 13730.. TODO:: 13731 13732 Possible broken issues: 13733 13734 - Stack arguments must be aligned to required alignment. 13735 - Stack is aligned to max(16, max formal argument alignment) 13736 - Direct argument < 64 bits should check register budget. 13737 - Register budget calculation should respect ``inreg`` for SGPR. 13738 - SGPR overflow is not handled. 13739 - struct with 1 member unpeeling is not checking size of member. 13740 - ``sret`` is after ``this`` pointer. 13741 - Caller is not implementing stack realignment: need an extra pointer. 13742 - Should say AMDGPU passes FP rather than SP. 13743 - Should CFI define CFA as address of locals or arguments. Difference is 13744 apparent when have implemented dynamic alignment. 13745 - If ``SCRATCH`` instruction could allow negative offsets, then can make FP be 13746 highest address of stack frame and use negative offset for locals. Would 13747 allow SP to be the same as FP and could support signal-handler-like as now 13748 have a real SP for the top of the stack. 13749 - How is ``sret`` passed on the stack? In argument stack area? Can it overlay 13750 arguments? 13751 13752AMDPAL 13753------ 13754 13755This section provides code conventions used when the target triple OS is 13756``amdpal`` (see :ref:`amdgpu-target-triples`). 13757 13758.. _amdgpu-amdpal-code-object-metadata-section: 13759 13760Code Object Metadata 13761~~~~~~~~~~~~~~~~~~~~ 13762 13763.. note:: 13764 13765 The metadata is currently in development and is subject to major 13766 changes. Only the current version is supported. *When this document 13767 was generated the version was 2.6.* 13768 13769Code object metadata is specified by the ``NT_AMDGPU_METADATA`` note 13770record (see :ref:`amdgpu-note-records-v3-onwards`). 13771 13772The metadata is represented as Message Pack formatted binary data (see 13773[MsgPack]_). The top level is a Message Pack map that includes the keys 13774defined in table :ref:`amdgpu-amdpal-code-object-metadata-map-table` 13775and referenced tables. 13776 13777Additional information can be added to the maps. To avoid conflicts, any 13778key names should be prefixed by "*vendor-name*." where ``vendor-name`` 13779can be the name of the vendor and specific vendor tool that generates the 13780information. The prefix is abbreviated to simply "." when it appears 13781within a map that has been added by the same *vendor-name*. 13782 13783 .. table:: AMDPAL Code Object Metadata Map 13784 :name: amdgpu-amdpal-code-object-metadata-map-table 13785 13786 =================== ============== ========= ====================================================================== 13787 String Key Value Type Required? Description 13788 =================== ============== ========= ====================================================================== 13789 "amdpal.version" sequence of Required PAL code object metadata (major, minor) version. The current values 13790 2 integers are defined by *Util::Abi::PipelineMetadata(Major|Minor)Version*. 13791 "amdpal.pipelines" sequence of Required Per-pipeline metadata. See 13792 map :ref:`amdgpu-amdpal-code-object-pipeline-metadata-map-table` for the 13793 definition of the keys included in that map. 13794 =================== ============== ========= ====================================================================== 13795 13796.. 13797 13798 .. table:: AMDPAL Code Object Pipeline Metadata Map 13799 :name: amdgpu-amdpal-code-object-pipeline-metadata-map-table 13800 13801 ====================================== ============== ========= =================================================== 13802 String Key Value Type Required? Description 13803 ====================================== ============== ========= =================================================== 13804 ".name" string Source name of the pipeline. 13805 ".type" string Pipeline type, e.g. VsPs. Values include: 13806 13807 - "VsPs" 13808 - "Gs" 13809 - "Cs" 13810 - "Ngg" 13811 - "Tess" 13812 - "GsTess" 13813 - "NggTess" 13814 13815 ".internal_pipeline_hash" sequence of Required Internal compiler hash for this pipeline. Lower 13816 2 integers 64 bits is the "stable" portion of the hash, used 13817 for e.g. shader replacement lookup. Upper 64 bits 13818 is the "unique" portion of the hash, used for 13819 e.g. pipeline cache lookup. The value is 13820 implementation defined, and can not be relied on 13821 between different builds of the compiler. 13822 ".shaders" map Per-API shader metadata. See 13823 :ref:`amdgpu-amdpal-code-object-shader-map-table` 13824 for the definition of the keys included in that 13825 map. 13826 ".hardware_stages" map Per-hardware stage metadata. See 13827 :ref:`amdgpu-amdpal-code-object-hardware-stage-map-table` 13828 for the definition of the keys included in that 13829 map. 13830 ".shader_functions" map Per-shader function metadata. See 13831 :ref:`amdgpu-amdpal-code-object-shader-function-map-table` 13832 for the definition of the keys included in that 13833 map. 13834 ".registers" map Required Hardware register configuration. See 13835 :ref:`amdgpu-amdpal-code-object-register-map-table` 13836 for the definition of the keys included in that 13837 map. 13838 ".user_data_limit" integer Number of user data entries accessed by this 13839 pipeline. 13840 ".spill_threshold" integer The user data spill threshold. 0xFFFF for 13841 NoUserDataSpilling. 13842 ".uses_viewport_array_index" boolean Indicates whether or not the pipeline uses the 13843 viewport array index feature. Pipelines which use 13844 this feature can render into all 16 viewports, 13845 whereas pipelines which do not use it are 13846 restricted to viewport #0. 13847 ".es_gs_lds_size" integer Size in bytes of LDS space used internally for 13848 handling data-passing between the ES and GS 13849 shader stages. This can be zero if the data is 13850 passed using off-chip buffers. This value should 13851 be used to program all user-SGPRs which have been 13852 marked with "UserDataMapping::EsGsLdsSize" 13853 (typically only the GS and VS HW stages will ever 13854 have a user-SGPR so marked). 13855 ".nggSubgroupSize" integer Explicit maximum subgroup size for NGG shaders 13856 (maximum number of threads in a subgroup). 13857 ".num_interpolants" integer Graphics only. Number of PS interpolants. 13858 ".mesh_scratch_memory_size" integer Max mesh shader scratch memory used. 13859 ".api" string Name of the client graphics API. 13860 ".api_create_info" binary Graphics API shader create info binary blob. Can 13861 be defined by the driver using the compiler if 13862 they want to be able to correlate API-specific 13863 information used during creation at a later time. 13864 ====================================== ============== ========= =================================================== 13865 13866.. 13867 13868 .. table:: AMDPAL Code Object Shader Map 13869 :name: amdgpu-amdpal-code-object-shader-map-table 13870 13871 13872 +-------------+--------------+-------------------------------------------------------------------+ 13873 |String Key |Value Type |Description | 13874 +=============+==============+===================================================================+ 13875 |- ".compute" |map |See :ref:`amdgpu-amdpal-code-object-api-shader-metadata-map-table` | 13876 |- ".vertex" | |for the definition of the keys included in that map. | 13877 |- ".hull" | | | 13878 |- ".domain" | | | 13879 |- ".geometry"| | | 13880 |- ".pixel" | | | 13881 +-------------+--------------+-------------------------------------------------------------------+ 13882 13883.. 13884 13885 .. table:: AMDPAL Code Object API Shader Metadata Map 13886 :name: amdgpu-amdpal-code-object-api-shader-metadata-map-table 13887 13888 ==================== ============== ========= ===================================================================== 13889 String Key Value Type Required? Description 13890 ==================== ============== ========= ===================================================================== 13891 ".api_shader_hash" sequence of Required Input shader hash, typically passed in from the client. The value 13892 2 integers is implementation defined, and can not be relied on between 13893 different builds of the compiler. 13894 ".hardware_mapping" sequence of Required Flags indicating the HW stages this API shader maps to. Values 13895 string include: 13896 13897 - ".ls" 13898 - ".hs" 13899 - ".es" 13900 - ".gs" 13901 - ".vs" 13902 - ".ps" 13903 - ".cs" 13904 13905 ==================== ============== ========= ===================================================================== 13906 13907.. 13908 13909 .. table:: AMDPAL Code Object Hardware Stage Map 13910 :name: amdgpu-amdpal-code-object-hardware-stage-map-table 13911 13912 +-------------+--------------+-----------------------------------------------------------------------+ 13913 |String Key |Value Type |Description | 13914 +=============+==============+=======================================================================+ 13915 |- ".ls" |map |See :ref:`amdgpu-amdpal-code-object-hardware-stage-metadata-map-table` | 13916 |- ".hs" | |for the definition of the keys included in that map. | 13917 |- ".es" | | | 13918 |- ".gs" | | | 13919 |- ".vs" | | | 13920 |- ".ps" | | | 13921 |- ".cs" | | | 13922 +-------------+--------------+-----------------------------------------------------------------------+ 13923 13924.. 13925 13926 .. table:: AMDPAL Code Object Hardware Stage Metadata Map 13927 :name: amdgpu-amdpal-code-object-hardware-stage-metadata-map-table 13928 13929 ========================== ============== ========= =============================================================== 13930 String Key Value Type Required? Description 13931 ========================== ============== ========= =============================================================== 13932 ".entry_point" string The ELF symbol pointing to this pipeline's stage entry point. 13933 ".scratch_memory_size" integer Scratch memory size in bytes. 13934 ".lds_size" integer Local Data Share size in bytes. 13935 ".perf_data_buffer_size" integer Performance data buffer size in bytes. 13936 ".vgpr_count" integer Number of VGPRs used. 13937 ".agpr_count" integer Number of AGPRs used. 13938 ".sgpr_count" integer Number of SGPRs used. 13939 ".vgpr_limit" integer If non-zero, indicates the shader was compiled with a 13940 directive to instruct the compiler to limit the VGPR usage to 13941 be less than or equal to the specified value (only set if 13942 different from HW default). 13943 ".sgpr_limit" integer SGPR count upper limit (only set if different from HW 13944 default). 13945 ".threadgroup_dimensions" sequence of Thread-group X/Y/Z dimensions (Compute only). 13946 3 integers 13947 ".wavefront_size" integer Wavefront size (only set if different from HW default). 13948 ".uses_uavs" boolean The shader reads or writes UAVs. 13949 ".uses_rovs" boolean The shader reads or writes ROVs. 13950 ".writes_uavs" boolean The shader writes to one or more UAVs. 13951 ".writes_depth" boolean The shader writes out a depth value. 13952 ".uses_append_consume" boolean The shader uses append and/or consume operations, either 13953 memory or GDS. 13954 ".uses_prim_id" boolean The shader uses PrimID. 13955 ========================== ============== ========= =============================================================== 13956 13957.. 13958 13959 .. table:: AMDPAL Code Object Shader Function Map 13960 :name: amdgpu-amdpal-code-object-shader-function-map-table 13961 13962 =============== ============== ==================================================================== 13963 String Key Value Type Description 13964 =============== ============== ==================================================================== 13965 *symbol name* map *symbol name* is the ELF symbol name of the shader function code 13966 entry address. The value is the function's metadata. See 13967 :ref:`amdgpu-amdpal-code-object-shader-function-metadata-map-table`. 13968 =============== ============== ==================================================================== 13969 13970.. 13971 13972 .. table:: AMDPAL Code Object Shader Function Metadata Map 13973 :name: amdgpu-amdpal-code-object-shader-function-metadata-map-table 13974 13975 ============================= ============== ================================================================= 13976 String Key Value Type Description 13977 ============================= ============== ================================================================= 13978 ".api_shader_hash" sequence of Input shader hash, typically passed in from the client. The value 13979 2 integers is implementation defined, and can not be relied on between 13980 different builds of the compiler. 13981 ".scratch_memory_size" integer Size in bytes of scratch memory used by the shader. 13982 ".lds_size" integer Size in bytes of LDS memory. 13983 ".vgpr_count" integer Number of VGPRs used by the shader. 13984 ".sgpr_count" integer Number of SGPRs used by the shader. 13985 ".stack_frame_size_in_bytes" integer Amount of stack size used by the shader. 13986 ".shader_subtype" string Shader subtype/kind. Values include: 13987 13988 - "Unknown" 13989 13990 ============================= ============== ================================================================= 13991 13992.. 13993 13994 .. table:: AMDPAL Code Object Register Map 13995 :name: amdgpu-amdpal-code-object-register-map-table 13996 13997 ========================== ============== ==================================================================== 13998 32-bit Integer Key Value Type Description 13999 ========================== ============== ==================================================================== 14000 ``reg offset`` 32-bit integer ``reg offset`` is the dword offset into the GFXIP register space of 14001 a GRBM register (i.e., driver accessible GPU register number, not 14002 shader GPR register number). The driver is required to program each 14003 specified register to the corresponding specified value when 14004 executing this pipeline. Typically, the ``reg offsets`` are the 14005 ``uint16_t`` offsets to each register as defined by the hardware 14006 chip headers. The register is set to the provided value. However, a 14007 ``reg offset`` that specifies a user data register (e.g., 14008 COMPUTE_USER_DATA_0) needs special treatment. See 14009 :ref:`amdgpu-amdpal-code-object-user-data-section` section for more 14010 information. 14011 ========================== ============== ==================================================================== 14012 14013.. _amdgpu-amdpal-code-object-user-data-section: 14014 14015User Data 14016+++++++++ 14017 14018Each hardware stage has a set of 32-bit physical SPI *user data registers* 14019(either 16 or 32 based on graphics IP and the stage) which can be 14020written from a command buffer and then loaded into SGPRs when waves are 14021launched via a subsequent dispatch or draw operation. This is the way 14022most arguments are passed from the application/runtime to a hardware 14023shader. 14024 14025PAL abstracts this functionality by exposing a set of 128 *user data 14026entries* per pipeline a client can use to pass arguments from a command 14027buffer to one or more shaders in that pipeline. The ELF code object must 14028specify a mapping from virtualized *user data entries* to physical *user 14029data registers*, and PAL is responsible for implementing that mapping, 14030including spilling overflow *user data entries* to memory if needed. 14031 14032Since the *user data registers* are GRBM-accessible SPI registers, this 14033mapping is actually embedded in the ``.registers`` metadata entry. For 14034most registers, the value in that map is a literal 32-bit value that 14035should be written to the register by the driver. However, when the 14036register is a *user data register* (any USER_DATA register e.g., 14037SPI_SHADER_USER_DATA_PS_5), the value is instead an encoding that tells 14038the driver to write either a *user data entry* value or one of several 14039driver-internal values to the register. This encoding is described in 14040the following table: 14041 14042.. note:: 14043 14044 Currently, *user data registers* 0 and 1 (e.g., SPI_SHADER_USER_DATA_PS_0, 14045 and SPI_SHADER_USER_DATA_PS_1) are reserved. *User data register* 0 must 14046 always be programmed to the address of the GlobalTable, and *user data 14047 register* 1 must always be programmed to the address of the PerShaderTable. 14048 14049.. 14050 14051 .. table:: AMDPAL User Data Mapping 14052 :name: amdgpu-amdpal-code-object-metadata-user-data-mapping-table 14053 14054 ========== ================= =============================================================================== 14055 Value Name Description 14056 ========== ================= =============================================================================== 14057 0..127 *User Data Entry* 32-bit value of user_data_entry[N] as specified via *CmdSetUserData()* 14058 0x10000000 GlobalTable 32-bit pointer to GPU memory containing the global internal table (should 14059 always point to *user data register* 0). 14060 0x10000001 PerShaderTable 32-bit pointer to GPU memory containing the per-shader internal table. See 14061 :ref:`amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section` 14062 for more detail (should always point to *user data register* 1). 14063 0x10000002 SpillTable 32-bit pointer to GPU memory containing the user data spill table. See 14064 :ref:`amdgpu-amdpal-code-object-metadata-user-data-spill-table-section` for 14065 more detail. 14066 0x10000003 BaseVertex Vertex offset (32-bit unsigned integer). Not needed if the pipeline doesn't 14067 reference the draw index in the vertex shader. Only supported by the first 14068 stage in a graphics pipeline. 14069 0x10000004 BaseInstance Instance offset (32-bit unsigned integer). Only supported by the first stage in 14070 a graphics pipeline. 14071 0x10000005 DrawIndex Draw index (32-bit unsigned integer). Only supported by the first stage in a 14072 graphics pipeline. 14073 0x10000006 Workgroup Thread group count (32-bit unsigned integer). Low half of a 64-bit address of 14074 a buffer containing the grid dimensions for a Compute dispatch operation. The 14075 high half of the address is stored in the next sequential user-SGPR. Only 14076 supported by compute pipelines. 14077 0x1000000A EsGsLdsSize Indicates that PAL will program this user-SGPR to contain the amount of LDS 14078 space used for the ES/GS pseudo-ring-buffer for passing data between shader 14079 stages. 14080 0x1000000B ViewId View id (32-bit unsigned integer) identifies a view of graphic 14081 pipeline instancing. 14082 0x1000000C StreamOutTable 32-bit pointer to GPU memory containing the stream out target SRD table. This 14083 can only appear for one shader stage per pipeline. 14084 0x1000000D PerShaderPerfData 32-bit pointer to GPU memory containing the per-shader performance data buffer. 14085 0x1000000F VertexBufferTable 32-bit pointer to GPU memory containing the vertex buffer SRD table. This can 14086 only appear for one shader stage per pipeline. 14087 0x10000010 UavExportTable 32-bit pointer to GPU memory containing the UAV export SRD table. This can 14088 only appear for one shader stage per pipeline (PS). These replace color targets 14089 and are completely separate from any UAVs used by the shader. This is optional, 14090 and only used by the PS when UAV exports are used to replace color-target 14091 exports to optimize specific shaders. 14092 0x10000011 NggCullingData 64-bit pointer to GPU memory containing the hardware register data needed by 14093 some NGG pipelines to perform culling. This value contains the address of the 14094 first of two consecutive registers which provide the full GPU address. 14095 0x10000015 FetchShaderPtr 64-bit pointer to GPU memory containing the fetch shader subroutine. 14096 ========== ================= =============================================================================== 14097 14098.. _amdgpu-amdpal-code-object-metadata-user-data-per-shader-table-section: 14099 14100Per-Shader Table 14101################ 14102 14103Low 32 bits of the GPU address for an optional buffer in the ``.data`` 14104section of the ELF. The high 32 bits of the address match the high 32 bits 14105of the shader's program counter. 14106 14107The buffer can be anything the shader compiler needs it for, and 14108allows each shader to have its own region of the ``.data`` section. 14109Typically, this could be a table of buffer SRD's and the data pointed to 14110by the buffer SRD's, but it could be a flat-address region of memory as 14111well. Its layout and usage are defined by the shader compiler. 14112 14113Each shader's table in the ``.data`` section is referenced by the symbol 14114``_amdgpu_``\ *xs*\ ``_shdr_intrl_data`` where *xs* corresponds with the 14115hardware shader stage the data is for. E.g., 14116``_amdgpu_cs_shdr_intrl_data`` for the compute shader hardware stage. 14117 14118.. _amdgpu-amdpal-code-object-metadata-user-data-spill-table-section: 14119 14120Spill Table 14121########### 14122 14123It is possible for a hardware shader to need access to more *user data 14124entries* than there are slots available in user data registers for one 14125or more hardware shader stages. In that case, the PAL runtime expects 14126the necessary *user data entries* to be spilled to GPU memory and use 14127one user data register to point to the spilled user data memory. The 14128value of the *user data entry* must then represent the location where 14129a shader expects to read the low 32-bits of the table's GPU virtual 14130address. The *spill table* itself represents a set of 32-bit values 14131managed by the PAL runtime in GPU-accessible memory that can be made 14132indirectly accessible to a hardware shader. 14133 14134Unspecified OS 14135-------------- 14136 14137This section provides code conventions used when the target triple OS is 14138empty (see :ref:`amdgpu-target-triples`). 14139 14140Trap Handler ABI 14141~~~~~~~~~~~~~~~~ 14142 14143For code objects generated by AMDGPU backend for non-amdhsa OS, the runtime does 14144not install a trap handler. The ``llvm.trap`` and ``llvm.debugtrap`` 14145instructions are handled as follows: 14146 14147 .. table:: AMDGPU Trap Handler for Non-AMDHSA OS 14148 :name: amdgpu-trap-handler-for-non-amdhsa-os-table 14149 14150 =============== =============== =========================================== 14151 Usage Code Sequence Description 14152 =============== =============== =========================================== 14153 llvm.trap s_endpgm Causes wavefront to be terminated. 14154 llvm.debugtrap *none* Compiler warning given that there is no 14155 trap handler installed. 14156 =============== =============== =========================================== 14157 14158Source Languages 14159================ 14160 14161.. _amdgpu-opencl: 14162 14163OpenCL 14164------ 14165 14166When the language is OpenCL the following differences occur: 14167 141681. The OpenCL memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 141692. The AMDGPU backend appends additional arguments to the kernel's explicit 14170 arguments for the AMDHSA OS (see 14171 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`). 141723. Additional metadata is generated 14173 (see :ref:`amdgpu-amdhsa-code-object-metadata`). 14174 14175 .. table:: OpenCL kernel implicit arguments appended for AMDHSA OS 14176 :name: opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table 14177 14178 ======== ==== ========= =========================================== 14179 Position Byte Byte Description 14180 Size Alignment 14181 ======== ==== ========= =========================================== 14182 1 8 8 OpenCL Global Offset X 14183 2 8 8 OpenCL Global Offset Y 14184 3 8 8 OpenCL Global Offset Z 14185 4 8 8 OpenCL address of printf buffer 14186 5 8 8 OpenCL address of virtual queue used by 14187 enqueue_kernel. 14188 6 8 8 OpenCL address of AqlWrap struct used by 14189 enqueue_kernel. 14190 7 8 8 Pointer argument used for Multi-gird 14191 synchronization. 14192 ======== ==== ========= =========================================== 14193 14194.. _amdgpu-hcc: 14195 14196HCC 14197--- 14198 14199When the language is HCC the following differences occur: 14200 142011. The HSA memory model is used (see :ref:`amdgpu-amdhsa-memory-model`). 14202 14203.. _amdgpu-assembler: 14204 14205Assembler 14206--------- 14207 14208AMDGPU backend has LLVM-MC based assembler which is currently in development. 14209It supports AMDGCN GFX6-GFX10. 14210 14211This section describes general syntax for instructions and operands. 14212 14213Instructions 14214~~~~~~~~~~~~ 14215 14216An instruction has the following :doc:`syntax<AMDGPUInstructionSyntax>`: 14217 14218 | ``<``\ *opcode*\ ``> <``\ *operand0*\ ``>, <``\ *operand1*\ ``>,... 14219 <``\ *modifier0*\ ``> <``\ *modifier1*\ ``>...`` 14220 14221:doc:`Operands<AMDGPUOperandSyntax>` are comma-separated while 14222:doc:`modifiers<AMDGPUModifierSyntax>` are space-separated. 14223 14224The order of operands and modifiers is fixed. 14225Most modifiers are optional and may be omitted. 14226 14227Links to detailed instruction syntax description may be found in the following 14228table. Note that features under development are not included 14229in this description. 14230 14231 ============= ============================================= ======================================= 14232 Architecture Core ISA ISA Variants and Extensions 14233 ============= ============================================= ======================================= 14234 GCN 2 :doc:`GFX7<AMDGPU/AMDGPUAsmGFX7>` \- 14235 GCN 3, GCN 4 :doc:`GFX8<AMDGPU/AMDGPUAsmGFX8>` \- 14236 GCN 5 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx900<AMDGPU/AMDGPUAsmGFX900>` 14237 14238 :doc:`gfx902<AMDGPU/AMDGPUAsmGFX900>` 14239 14240 :doc:`gfx904<AMDGPU/AMDGPUAsmGFX904>` 14241 14242 :doc:`gfx906<AMDGPU/AMDGPUAsmGFX906>` 14243 14244 :doc:`gfx909<AMDGPU/AMDGPUAsmGFX900>` 14245 14246 :doc:`gfx90c<AMDGPU/AMDGPUAsmGFX900>` 14247 14248 CDNA 1 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx908<AMDGPU/AMDGPUAsmGFX908>` 14249 14250 CDNA 2 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx90a<AMDGPU/AMDGPUAsmGFX90a>` 14251 14252 CDNA 3 :doc:`GFX9<AMDGPU/AMDGPUAsmGFX9>` :doc:`gfx940<AMDGPU/AMDGPUAsmGFX940>` 14253 14254 RDNA 1 :doc:`GFX10 RDNA1<AMDGPU/AMDGPUAsmGFX10>` :doc:`gfx1010<AMDGPU/AMDGPUAsmGFX10>` 14255 14256 :doc:`gfx1011<AMDGPU/AMDGPUAsmGFX1011>` 14257 14258 :doc:`gfx1012<AMDGPU/AMDGPUAsmGFX1011>` 14259 14260 :doc:`gfx1013<AMDGPU/AMDGPUAsmGFX1013>` 14261 14262 RDNA 2 :doc:`GFX10 RDNA2<AMDGPU/AMDGPUAsmGFX1030>` :doc:`gfx1030<AMDGPU/AMDGPUAsmGFX1030>` 14263 14264 :doc:`gfx1031<AMDGPU/AMDGPUAsmGFX1030>` 14265 14266 :doc:`gfx1032<AMDGPU/AMDGPUAsmGFX1030>` 14267 14268 :doc:`gfx1033<AMDGPU/AMDGPUAsmGFX1030>` 14269 14270 :doc:`gfx1034<AMDGPU/AMDGPUAsmGFX1030>` 14271 14272 :doc:`gfx1035<AMDGPU/AMDGPUAsmGFX1030>` 14273 14274 :doc:`gfx1036<AMDGPU/AMDGPUAsmGFX1030>` 14275 ============= ============================================= ======================================= 14276 14277For more information about instructions, their semantics and supported 14278combinations of operands, refer to one of instruction set architecture manuals 14279[AMD-GCN-GFX6]_, [AMD-GCN-GFX7]_, [AMD-GCN-GFX8]_, 14280[AMD-GCN-GFX900-GFX904-VEGA]_, [AMD-GCN-GFX906-VEGA7NM]_, 14281[AMD-GCN-GFX908-CDNA1]_, [AMD-GCN-GFX90A-CDNA2]_, [AMD-GCN-GFX10-RDNA1]_ and 14282[AMD-GCN-GFX10-RDNA2]_. 14283 14284Operands 14285~~~~~~~~ 14286 14287Detailed description of operands may be found :doc:`here<AMDGPUOperandSyntax>`. 14288 14289Modifiers 14290~~~~~~~~~ 14291 14292Detailed description of modifiers may be found 14293:doc:`here<AMDGPUModifierSyntax>`. 14294 14295Instruction Examples 14296~~~~~~~~~~~~~~~~~~~~ 14297 14298DS 14299++ 14300 14301.. code-block:: nasm 14302 14303 ds_add_u32 v2, v4 offset:16 14304 ds_write_src2_b64 v2 offset0:4 offset1:8 14305 ds_cmpst_f32 v2, v4, v6 14306 ds_min_rtn_f64 v[8:9], v2, v[4:5] 14307 14308For full list of supported instructions, refer to "LDS/GDS instructions" in ISA 14309Manual. 14310 14311FLAT 14312++++ 14313 14314.. code-block:: nasm 14315 14316 flat_load_dword v1, v[3:4] 14317 flat_store_dwordx3 v[3:4], v[5:7] 14318 flat_atomic_swap v1, v[3:4], v5 glc 14319 flat_atomic_cmpswap v1, v[3:4], v[5:6] glc slc 14320 flat_atomic_fmax_x2 v[1:2], v[3:4], v[5:6] glc 14321 14322For full list of supported instructions, refer to "FLAT instructions" in ISA 14323Manual. 14324 14325MUBUF 14326+++++ 14327 14328.. code-block:: nasm 14329 14330 buffer_load_dword v1, off, s[4:7], s1 14331 buffer_store_dwordx4 v[1:4], v2, ttmp[4:7], s1 offen offset:4 glc tfe 14332 buffer_store_format_xy v[1:2], off, s[4:7], s1 14333 buffer_wbinvl1 14334 buffer_atomic_inc v1, v2, s[8:11], s4 idxen offset:4 slc 14335 14336For full list of supported instructions, refer to "MUBUF Instructions" in ISA 14337Manual. 14338 14339SMRD/SMEM 14340+++++++++ 14341 14342.. code-block:: nasm 14343 14344 s_load_dword s1, s[2:3], 0xfc 14345 s_load_dwordx8 s[8:15], s[2:3], s4 14346 s_load_dwordx16 s[88:103], s[2:3], s4 14347 s_dcache_inv_vol 14348 s_memtime s[4:5] 14349 14350For full list of supported instructions, refer to "Scalar Memory Operations" in 14351ISA Manual. 14352 14353SOP1 14354++++ 14355 14356.. code-block:: nasm 14357 14358 s_mov_b32 s1, s2 14359 s_mov_b64 s[0:1], 0x80000000 14360 s_cmov_b32 s1, 200 14361 s_wqm_b64 s[2:3], s[4:5] 14362 s_bcnt0_i32_b64 s1, s[2:3] 14363 s_swappc_b64 s[2:3], s[4:5] 14364 s_cbranch_join s[4:5] 14365 14366For full list of supported instructions, refer to "SOP1 Instructions" in ISA 14367Manual. 14368 14369SOP2 14370++++ 14371 14372.. code-block:: nasm 14373 14374 s_add_u32 s1, s2, s3 14375 s_and_b64 s[2:3], s[4:5], s[6:7] 14376 s_cselect_b32 s1, s2, s3 14377 s_andn2_b32 s2, s4, s6 14378 s_lshr_b64 s[2:3], s[4:5], s6 14379 s_ashr_i32 s2, s4, s6 14380 s_bfm_b64 s[2:3], s4, s6 14381 s_bfe_i64 s[2:3], s[4:5], s6 14382 s_cbranch_g_fork s[4:5], s[6:7] 14383 14384For full list of supported instructions, refer to "SOP2 Instructions" in ISA 14385Manual. 14386 14387SOPC 14388++++ 14389 14390.. code-block:: nasm 14391 14392 s_cmp_eq_i32 s1, s2 14393 s_bitcmp1_b32 s1, s2 14394 s_bitcmp0_b64 s[2:3], s4 14395 s_setvskip s3, s5 14396 14397For full list of supported instructions, refer to "SOPC Instructions" in ISA 14398Manual. 14399 14400SOPP 14401++++ 14402 14403.. code-block:: nasm 14404 14405 s_barrier 14406 s_nop 2 14407 s_endpgm 14408 s_waitcnt 0 ; Wait for all counters to be 0 14409 s_waitcnt vmcnt(0) & expcnt(0) & lgkmcnt(0) ; Equivalent to above 14410 s_waitcnt vmcnt(1) ; Wait for vmcnt counter to be 1. 14411 s_sethalt 9 14412 s_sleep 10 14413 s_sendmsg 0x1 14414 s_sendmsg sendmsg(MSG_INTERRUPT) 14415 s_trap 1 14416 14417For full list of supported instructions, refer to "SOPP Instructions" in ISA 14418Manual. 14419 14420Unless otherwise mentioned, little verification is performed on the operands 14421of SOPP Instructions, so it is up to the programmer to be familiar with the 14422range or acceptable values. 14423 14424VALU 14425++++ 14426 14427For vector ALU instruction opcodes (VOP1, VOP2, VOP3, VOPC, VOP_DPP, VOP_SDWA), 14428the assembler will automatically use optimal encoding based on its operands. To 14429force specific encoding, one can add a suffix to the opcode of the instruction: 14430 14431* _e32 for 32-bit VOP1/VOP2/VOPC 14432* _e64 for 64-bit VOP3 14433* _dpp for VOP_DPP 14434* _sdwa for VOP_SDWA 14435 14436VOP1/VOP2/VOP3/VOPC examples: 14437 14438.. code-block:: nasm 14439 14440 v_mov_b32 v1, v2 14441 v_mov_b32_e32 v1, v2 14442 v_nop 14443 v_cvt_f64_i32_e32 v[1:2], v2 14444 v_floor_f32_e32 v1, v2 14445 v_bfrev_b32_e32 v1, v2 14446 v_add_f32_e32 v1, v2, v3 14447 v_mul_i32_i24_e64 v1, v2, 3 14448 v_mul_i32_i24_e32 v1, -3, v3 14449 v_mul_i32_i24_e32 v1, -100, v3 14450 v_addc_u32 v1, s[0:1], v2, v3, s[2:3] 14451 v_max_f16_e32 v1, v2, v3 14452 14453VOP_DPP examples: 14454 14455.. code-block:: nasm 14456 14457 v_mov_b32 v0, v0 quad_perm:[0,2,1,1] 14458 v_sin_f32 v0, v0 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14459 v_mov_b32 v0, v0 wave_shl:1 14460 v_mov_b32 v0, v0 row_mirror 14461 v_mov_b32 v0, v0 row_bcast:31 14462 v_mov_b32 v0, v0 quad_perm:[1,3,0,1] row_mask:0xa bank_mask:0x1 bound_ctrl:0 14463 v_add_f32 v0, v0, |v0| row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14464 v_max_f16 v1, v2, v3 row_shl:1 row_mask:0xa bank_mask:0x1 bound_ctrl:0 14465 14466VOP_SDWA examples: 14467 14468.. code-block:: nasm 14469 14470 v_mov_b32 v1, v2 dst_sel:BYTE_0 dst_unused:UNUSED_PRESERVE src0_sel:DWORD 14471 v_min_u32 v200, v200, v1 dst_sel:WORD_1 dst_unused:UNUSED_PAD src0_sel:BYTE_1 src1_sel:DWORD 14472 v_sin_f32 v0, v0 dst_unused:UNUSED_PAD src0_sel:WORD_1 14473 v_fract_f32 v0, |v0| dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1 14474 v_cmpx_le_u32 vcc, v1, v2 src0_sel:BYTE_2 src1_sel:WORD_0 14475 14476For full list of supported instructions, refer to "Vector ALU instructions". 14477 14478.. _amdgpu-amdhsa-assembler-predefined-symbols-v2: 14479 14480Code Object V2 Predefined Symbols 14481~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14482 14483.. warning:: 14484 Code object V2 is not the default code object version emitted by 14485 this version of LLVM. 14486 14487The AMDGPU assembler defines and updates some symbols automatically. These 14488symbols do not affect code generation. 14489 14490.option.machine_version_major 14491+++++++++++++++++++++++++++++ 14492 14493Set to the GFX major generation number of the target being assembled for. For 14494example, when assembling for a "GFX9" target this will be set to the integer 14495value "9". The possible GFX major generation numbers are presented in 14496:ref:`amdgpu-processors`. 14497 14498.option.machine_version_minor 14499+++++++++++++++++++++++++++++ 14500 14501Set to the GFX minor generation number of the target being assembled for. For 14502example, when assembling for a "GFX810" target this will be set to the integer 14503value "1". The possible GFX minor generation numbers are presented in 14504:ref:`amdgpu-processors`. 14505 14506.option.machine_version_stepping 14507++++++++++++++++++++++++++++++++ 14508 14509Set to the GFX stepping generation number of the target being assembled for. 14510For example, when assembling for a "GFX704" target this will be set to the 14511integer value "4". The possible GFX stepping generation numbers are presented 14512in :ref:`amdgpu-processors`. 14513 14514.kernel.vgpr_count 14515++++++++++++++++++ 14516 14517Set to zero each time a 14518:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14519encountered. At each instruction, if the current value of this symbol is less 14520than or equal to the maximum VGPR number explicitly referenced within that 14521instruction then the symbol value is updated to equal that VGPR number plus 14522one. 14523 14524.kernel.sgpr_count 14525++++++++++++++++++ 14526 14527Set to zero each time a 14528:ref:`amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel` directive is 14529encountered. At each instruction, if the current value of this symbol is less 14530than or equal to the maximum VGPR number explicitly referenced within that 14531instruction then the symbol value is updated to equal that SGPR number plus 14532one. 14533 14534.. _amdgpu-amdhsa-assembler-directives-v2: 14535 14536Code Object V2 Directives 14537~~~~~~~~~~~~~~~~~~~~~~~~~ 14538 14539.. warning:: 14540 Code object V2 is not the default code object version emitted by 14541 this version of LLVM. 14542 14543AMDGPU ABI defines auxiliary data in output code object. In assembly source, 14544one can specify them with assembler directives. 14545 14546.hsa_code_object_version major, minor 14547+++++++++++++++++++++++++++++++++++++ 14548 14549*major* and *minor* are integers that specify the version of the HSA code 14550object that will be generated by the assembler. 14551 14552.hsa_code_object_isa [major, minor, stepping, vendor, arch] 14553+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 14554 14555 14556*major*, *minor*, and *stepping* are all integers that describe the instruction 14557set architecture (ISA) version of the assembly program. 14558 14559*vendor* and *arch* are quoted strings. *vendor* should always be equal to 14560"AMD" and *arch* should always be equal to "AMDGPU". 14561 14562By default, the assembler will derive the ISA version, *vendor*, and *arch* 14563from the value of the -mcpu option that is passed to the assembler. 14564 14565.. _amdgpu-amdhsa-assembler-directive-amdgpu_hsa_kernel: 14566 14567.amdgpu_hsa_kernel (name) 14568+++++++++++++++++++++++++ 14569 14570This directives specifies that the symbol with given name is a kernel entry 14571point (label) and the object should contain corresponding symbol of type 14572STT_AMDGPU_HSA_KERNEL. 14573 14574.amd_kernel_code_t 14575++++++++++++++++++ 14576 14577This directive marks the beginning of a list of key / value pairs that are used 14578to specify the amd_kernel_code_t object that will be emitted by the assembler. 14579The list must be terminated by the *.end_amd_kernel_code_t* directive. For any 14580amd_kernel_code_t values that are unspecified a default value will be used. The 14581default value for all keys is 0, with the following exceptions: 14582 14583- *amd_code_version_major* defaults to 1. 14584- *amd_kernel_code_version_minor* defaults to 2. 14585- *amd_machine_kind* defaults to 1. 14586- *amd_machine_version_major*, *machine_version_minor*, and 14587 *amd_machine_version_stepping* are derived from the value of the -mcpu option 14588 that is passed to the assembler. 14589- *kernel_code_entry_byte_offset* defaults to 256. 14590- *wavefront_size* defaults 6 for all targets before GFX10. For GFX10 onwards 14591 defaults to 6 if target feature ``wavefrontsize64`` is enabled, otherwise 5. 14592 Note that wavefront size is specified as a power of two, so a value of **n** 14593 means a size of 2^ **n**. 14594- *call_convention* defaults to -1. 14595- *kernarg_segment_alignment*, *group_segment_alignment*, and 14596 *private_segment_alignment* default to 4. Note that alignments are specified 14597 as a power of 2, so a value of **n** means an alignment of 2^ **n**. 14598- *enable_tg_split* defaults to 1 if target feature ``tgsplit`` is enabled for 14599 GFX90A onwards. 14600- *enable_wgp_mode* defaults to 1 if target feature ``cumode`` is disabled for 14601 GFX10 onwards. 14602- *enable_mem_ordered* defaults to 1 for GFX10 onwards. 14603 14604The *.amd_kernel_code_t* directive must be placed immediately after the 14605function label and before any instructions. 14606 14607For a full list of amd_kernel_code_t keys, refer to AMDGPU ABI document, 14608comments in lib/Target/AMDGPU/AmdKernelCodeT.h and test/CodeGen/AMDGPU/hsa.s. 14609 14610.. _amdgpu-amdhsa-assembler-example-v2: 14611 14612Code Object V2 Example Source Code 14613~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14614 14615.. warning:: 14616 Code Object V2 is not the default code object version emitted by 14617 this version of LLVM. 14618 14619Here is an example of a minimal assembly source file, defining one HSA kernel: 14620 14621.. code:: 14622 :number-lines: 14623 14624 .hsa_code_object_version 1,0 14625 .hsa_code_object_isa 14626 14627 .hsatext 14628 .globl hello_world 14629 .p2align 8 14630 .amdgpu_hsa_kernel hello_world 14631 14632 hello_world: 14633 14634 .amd_kernel_code_t 14635 enable_sgpr_kernarg_segment_ptr = 1 14636 is_ptr64 = 1 14637 compute_pgm_rsrc1_vgprs = 0 14638 compute_pgm_rsrc1_sgprs = 0 14639 compute_pgm_rsrc2_user_sgpr = 2 14640 compute_pgm_rsrc1_wgp_mode = 0 14641 compute_pgm_rsrc1_mem_ordered = 0 14642 compute_pgm_rsrc1_fwd_progress = 1 14643 .end_amd_kernel_code_t 14644 14645 s_load_dwordx2 s[0:1], s[0:1] 0x0 14646 v_mov_b32 v0, 3.14159 14647 s_waitcnt lgkmcnt(0) 14648 v_mov_b32 v1, s0 14649 v_mov_b32 v2, s1 14650 flat_store_dword v[1:2], v0 14651 s_endpgm 14652 .Lfunc_end0: 14653 .size hello_world, .Lfunc_end0-hello_world 14654 14655.. _amdgpu-amdhsa-assembler-predefined-symbols-v3-onwards: 14656 14657Code Object V3 and Above Predefined Symbols 14658~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14659 14660The AMDGPU assembler defines and updates some symbols automatically. These 14661symbols do not affect code generation. 14662 14663.amdgcn.gfx_generation_number 14664+++++++++++++++++++++++++++++ 14665 14666Set to the GFX major generation number of the target being assembled for. For 14667example, when assembling for a "GFX9" target this will be set to the integer 14668value "9". The possible GFX major generation numbers are presented in 14669:ref:`amdgpu-processors`. 14670 14671.amdgcn.gfx_generation_minor 14672++++++++++++++++++++++++++++ 14673 14674Set to the GFX minor generation number of the target being assembled for. For 14675example, when assembling for a "GFX810" target this will be set to the integer 14676value "1". The possible GFX minor generation numbers are presented in 14677:ref:`amdgpu-processors`. 14678 14679.amdgcn.gfx_generation_stepping 14680+++++++++++++++++++++++++++++++ 14681 14682Set to the GFX stepping generation number of the target being assembled for. 14683For example, when assembling for a "GFX704" target this will be set to the 14684integer value "4". The possible GFX stepping generation numbers are presented 14685in :ref:`amdgpu-processors`. 14686 14687.. _amdgpu-amdhsa-assembler-symbol-next_free_vgpr: 14688 14689.amdgcn.next_free_vgpr 14690++++++++++++++++++++++ 14691 14692Set to zero before assembly begins. At each instruction, if the current value 14693of this symbol is less than or equal to the maximum VGPR number explicitly 14694referenced within that instruction then the symbol value is updated to equal 14695that VGPR number plus one. 14696 14697May be used to set the `.amdhsa_next_free_vgpr` directive in 14698:ref:`amdhsa-kernel-directives-table`. 14699 14700May be set at any time, e.g. manually set to zero at the start of each kernel. 14701 14702.. _amdgpu-amdhsa-assembler-symbol-next_free_sgpr: 14703 14704.amdgcn.next_free_sgpr 14705++++++++++++++++++++++ 14706 14707Set to zero before assembly begins. At each instruction, if the current value 14708of this symbol is less than or equal the maximum SGPR number explicitly 14709referenced within that instruction then the symbol value is updated to equal 14710that SGPR number plus one. 14711 14712May be used to set the `.amdhsa_next_free_spgr` directive in 14713:ref:`amdhsa-kernel-directives-table`. 14714 14715May be set at any time, e.g. manually set to zero at the start of each kernel. 14716 14717.. _amdgpu-amdhsa-assembler-directives-v3-onwards: 14718 14719Code Object V3 and Above Directives 14720~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14721 14722Directives which begin with ``.amdgcn`` are valid for all ``amdgcn`` 14723architecture processors, and are not OS-specific. Directives which begin with 14724``.amdhsa`` are specific to ``amdgcn`` architecture processors when the 14725``amdhsa`` OS is specified. See :ref:`amdgpu-target-triples` and 14726:ref:`amdgpu-processors`. 14727 14728.. _amdgpu-assembler-directive-amdgcn-target: 14729 14730.amdgcn_target <target-triple> "-" <target-id> 14731++++++++++++++++++++++++++++++++++++++++++++++ 14732 14733Optional directive which declares the ``<target-triple>-<target-id>`` supported 14734by the containing assembler source file. Used by the assembler to validate 14735command-line options such as ``-triple``, ``-mcpu``, and 14736``--offload-arch=<target-id>``. A non-canonical target ID is allowed. See 14737:ref:`amdgpu-target-triples` and :ref:`amdgpu-target-id`. 14738 14739.. note:: 14740 14741 The target ID syntax used for code object V2 to V3 for this directive differs 14742 from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`. 14743 14744.amdhsa_kernel <name> 14745+++++++++++++++++++++ 14746 14747Creates a correctly aligned AMDHSA kernel descriptor and a symbol, 14748``<name>.kd``, in the current location of the current section. Only valid when 14749the OS is ``amdhsa``. ``<name>`` must be a symbol that labels the first 14750instruction to execute, and does not need to be previously defined. 14751 14752Marks the beginning of a list of directives used to generate the bytes of a 14753kernel descriptor, as described in :ref:`amdgpu-amdhsa-kernel-descriptor`. 14754Directives which may appear in this list are described in 14755:ref:`amdhsa-kernel-directives-table`. Directives may appear in any order, must 14756be valid for the target being assembled for, and cannot be repeated. Directives 14757support the range of values specified by the field they reference in 14758:ref:`amdgpu-amdhsa-kernel-descriptor`. If a directive is not specified, it is 14759assumed to have its default value, unless it is marked as "Required", in which 14760case it is an error to omit the directive. This list of directives is 14761terminated by an ``.end_amdhsa_kernel`` directive. 14762 14763 .. table:: AMDHSA Kernel Assembler Directives 14764 :name: amdhsa-kernel-directives-table 14765 14766 ======================================================== =================== ============ =================== 14767 Directive Default Supported On Description 14768 ======================================================== =================== ============ =================== 14769 ``.amdhsa_group_segment_fixed_size`` 0 GFX6-GFX10 Controls GROUP_SEGMENT_FIXED_SIZE in 14770 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14771 ``.amdhsa_private_segment_fixed_size`` 0 GFX6-GFX10 Controls PRIVATE_SEGMENT_FIXED_SIZE in 14772 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14773 ``.amdhsa_kernarg_size`` 0 GFX6-GFX10 Controls KERNARG_SIZE in 14774 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14775 ``.amdhsa_user_sgpr_count`` 0 GFX6-GFX10 Controls USER_SGPR_COUNT in COMPUTE_PGM_RSRC2 14776 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table` 14777 ``.amdhsa_user_sgpr_private_segment_buffer`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_BUFFER in 14778 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14779 GFX940) 14780 ``.amdhsa_user_sgpr_dispatch_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_PTR in 14781 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14782 ``.amdhsa_user_sgpr_queue_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_QUEUE_PTR in 14783 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14784 ``.amdhsa_user_sgpr_kernarg_segment_ptr`` 0 GFX6-GFX10 Controls ENABLE_SGPR_KERNARG_SEGMENT_PTR in 14785 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14786 ``.amdhsa_user_sgpr_dispatch_id`` 0 GFX6-GFX10 Controls ENABLE_SGPR_DISPATCH_ID in 14787 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14788 ``.amdhsa_user_sgpr_flat_scratch_init`` 0 GFX6-GFX10 Controls ENABLE_SGPR_FLAT_SCRATCH_INIT in 14789 (except :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14790 GFX940) 14791 ``.amdhsa_user_sgpr_private_segment_size`` 0 GFX6-GFX10 Controls ENABLE_SGPR_PRIVATE_SEGMENT_SIZE in 14792 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14793 ``.amdhsa_wavefront_size32`` Target GFX10 Controls ENABLE_WAVEFRONT_SIZE32 in 14794 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14795 Specific 14796 (wavefrontsize64) 14797 ``.amdhsa_system_sgpr_private_segment_wavefront_offset`` 0 GFX6-GFX10 Controls ENABLE_PRIVATE_SEGMENT in 14798 (except :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14799 GFX940) 14800 ``.amdhsa_enable_private_segment`` 0 GFX940 Controls ENABLE_PRIVATE_SEGMENT in 14801 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14802 ``.amdhsa_system_sgpr_workgroup_id_x`` 1 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_X in 14803 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14804 ``.amdhsa_system_sgpr_workgroup_id_y`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Y in 14805 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14806 ``.amdhsa_system_sgpr_workgroup_id_z`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_ID_Z in 14807 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14808 ``.amdhsa_system_sgpr_workgroup_info`` 0 GFX6-GFX10 Controls ENABLE_SGPR_WORKGROUP_INFO in 14809 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14810 ``.amdhsa_system_vgpr_workitem_id`` 0 GFX6-GFX10 Controls ENABLE_VGPR_WORKITEM_ID in 14811 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14812 Possible values are defined in 14813 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`. 14814 ``.amdhsa_next_free_vgpr`` Required GFX6-GFX10 Maximum VGPR number explicitly referenced, plus one. 14815 Used to calculate GRANULATED_WORKITEM_VGPR_COUNT in 14816 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14817 ``.amdhsa_next_free_sgpr`` Required GFX6-GFX10 Maximum SGPR number explicitly referenced, plus one. 14818 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14819 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14820 ``.amdhsa_accum_offset`` Required GFX90A, Offset of a first AccVGPR in the unified register file. 14821 GFX940 Used to calculate ACCUM_OFFSET in 14822 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14823 ``.amdhsa_reserve_vcc`` 1 GFX6-GFX10 Whether the kernel may use the special VCC SGPR. 14824 Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14825 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14826 ``.amdhsa_reserve_flat_scratch`` 1 GFX7-GFX10 Whether the kernel may use flat instructions to access 14827 (except scratch memory. Used to calculate 14828 GFX940) GRANULATED_WAVEFRONT_SGPR_COUNT in 14829 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14830 ``.amdhsa_reserve_xnack_mask`` Target GFX8-GFX10 Whether the kernel may trigger XNACK replay. 14831 Feature Used to calculate GRANULATED_WAVEFRONT_SGPR_COUNT in 14832 Specific :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14833 (xnack) 14834 ``.amdhsa_float_round_mode_32`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_32 in 14835 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14836 Possible values are defined in 14837 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14838 ``.amdhsa_float_round_mode_16_64`` 0 GFX6-GFX10 Controls FLOAT_ROUND_MODE_16_64 in 14839 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14840 Possible values are defined in 14841 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`. 14842 ``.amdhsa_float_denorm_mode_32`` 0 GFX6-GFX10 Controls FLOAT_DENORM_MODE_32 in 14843 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14844 Possible values are defined in 14845 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14846 ``.amdhsa_float_denorm_mode_16_64`` 3 GFX6-GFX10 Controls FLOAT_DENORM_MODE_16_64 in 14847 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14848 Possible values are defined in 14849 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`. 14850 ``.amdhsa_dx10_clamp`` 1 GFX6-GFX10 Controls ENABLE_DX10_CLAMP in 14851 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14852 ``.amdhsa_ieee_mode`` 1 GFX6-GFX10 Controls ENABLE_IEEE_MODE in 14853 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14854 ``.amdhsa_fp16_overflow`` 0 GFX9-GFX10 Controls FP16_OVFL in 14855 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14856 ``.amdhsa_tg_split`` Target GFX90A, Controls TG_SPLIT in 14857 Feature GFX940 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`. 14858 Specific 14859 (tgsplit) 14860 ``.amdhsa_workgroup_processor_mode`` Target GFX10 Controls ENABLE_WGP_MODE in 14861 Feature :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`. 14862 Specific 14863 (cumode) 14864 ``.amdhsa_memory_ordered`` 1 GFX10 Controls MEM_ORDERED in 14865 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14866 ``.amdhsa_forward_progress`` 0 GFX10 Controls FWD_PROGRESS in 14867 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx10-table`. 14868 ``.amdhsa_shared_vgpr_count`` 0 GFX10 Controls SHARED_VGPR_COUNT in 14869 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-table`. 14870 ``.amdhsa_exception_fp_ieee_invalid_op`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INVALID_OPERATION in 14871 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14872 ``.amdhsa_exception_fp_denorm_src`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_FP_DENORMAL_SOURCE in 14873 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14874 ``.amdhsa_exception_fp_ieee_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_DIVISION_BY_ZERO in 14875 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14876 ``.amdhsa_exception_fp_ieee_overflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_OVERFLOW in 14877 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14878 ``.amdhsa_exception_fp_ieee_underflow`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_UNDERFLOW in 14879 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14880 ``.amdhsa_exception_fp_ieee_inexact`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_IEEE_754_FP_INEXACT in 14881 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14882 ``.amdhsa_exception_int_div_zero`` 0 GFX6-GFX10 Controls ENABLE_EXCEPTION_INT_DIVIDE_BY_ZERO in 14883 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx10-table`. 14884 ======================================================== =================== ============ =================== 14885 14886.amdgpu_metadata 14887++++++++++++++++ 14888 14889Optional directive which declares the contents of the ``NT_AMDGPU_METADATA`` 14890note record (see :ref:`amdgpu-elf-note-records-table-v3-onwards`). 14891 14892The contents must be in the [YAML]_ markup format, with the same structure and 14893semantics described in :ref:`amdgpu-amdhsa-code-object-metadata-v3`, 14894:ref:`amdgpu-amdhsa-code-object-metadata-v4` or 14895:ref:`amdgpu-amdhsa-code-object-metadata-v5`. 14896 14897This directive is terminated by an ``.end_amdgpu_metadata`` directive. 14898 14899.. _amdgpu-amdhsa-assembler-example-v3-onwards: 14900 14901Code Object V3 and Above Example Source Code 14902~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 14903 14904Here is an example of a minimal assembly source file, defining one HSA kernel: 14905 14906.. code:: 14907 :number-lines: 14908 14909 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14910 14911 .text 14912 .globl hello_world 14913 .p2align 8 14914 .type hello_world,@function 14915 hello_world: 14916 s_load_dwordx2 s[0:1], s[0:1] 0x0 14917 v_mov_b32 v0, 3.14159 14918 s_waitcnt lgkmcnt(0) 14919 v_mov_b32 v1, s0 14920 v_mov_b32 v2, s1 14921 flat_store_dword v[1:2], v0 14922 s_endpgm 14923 .Lfunc_end0: 14924 .size hello_world, .Lfunc_end0-hello_world 14925 14926 .rodata 14927 .p2align 6 14928 .amdhsa_kernel hello_world 14929 .amdhsa_user_sgpr_kernarg_segment_ptr 1 14930 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14931 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14932 .end_amdhsa_kernel 14933 14934 .amdgpu_metadata 14935 --- 14936 amdhsa.version: 14937 - 1 14938 - 0 14939 amdhsa.kernels: 14940 - .name: hello_world 14941 .symbol: hello_world.kd 14942 .kernarg_segment_size: 48 14943 .group_segment_fixed_size: 0 14944 .private_segment_fixed_size: 0 14945 .kernarg_segment_align: 4 14946 .wavefront_size: 64 14947 .sgpr_count: 2 14948 .vgpr_count: 3 14949 .max_flat_workgroup_size: 256 14950 .args: 14951 - .size: 8 14952 .offset: 0 14953 .value_kind: global_buffer 14954 .address_space: global 14955 .actual_access: write_only 14956 //... 14957 .end_amdgpu_metadata 14958 14959This kernel is equivalent to the following HIP program: 14960 14961.. code:: 14962 :number-lines: 14963 14964 __global__ void hello_world(float *p) { 14965 *p = 3.14159f; 14966 } 14967 14968If an assembly source file contains multiple kernels and/or functions, the 14969:ref:`amdgpu-amdhsa-assembler-symbol-next_free_vgpr` and 14970:ref:`amdgpu-amdhsa-assembler-symbol-next_free_sgpr` symbols may be reset using 14971the ``.set <symbol>, <expression>`` directive. For example, in the case of two 14972kernels, where ``function1`` is only called from ``kernel1`` it is sufficient 14973to group the function with the kernel that calls it and reset the symbols 14974between the two connected components: 14975 14976.. code:: 14977 :number-lines: 14978 14979 .amdgcn_target "amdgcn-amd-amdhsa--gfx900+xnack" // optional 14980 14981 // gpr tracking symbols are implicitly set to zero 14982 14983 .text 14984 .globl kern0 14985 .p2align 8 14986 .type kern0,@function 14987 kern0: 14988 // ... 14989 s_endpgm 14990 .Lkern0_end: 14991 .size kern0, .Lkern0_end-kern0 14992 14993 .rodata 14994 .p2align 6 14995 .amdhsa_kernel kern0 14996 // ... 14997 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 14998 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 14999 .end_amdhsa_kernel 15000 15001 // reset symbols to begin tracking usage in func1 and kern1 15002 .set .amdgcn.next_free_vgpr, 0 15003 .set .amdgcn.next_free_sgpr, 0 15004 15005 .text 15006 .hidden func1 15007 .global func1 15008 .p2align 2 15009 .type func1,@function 15010 func1: 15011 // ... 15012 s_setpc_b64 s[30:31] 15013 .Lfunc1_end: 15014 .size func1, .Lfunc1_end-func1 15015 15016 .globl kern1 15017 .p2align 8 15018 .type kern1,@function 15019 kern1: 15020 // ... 15021 s_getpc_b64 s[4:5] 15022 s_add_u32 s4, s4, func1@rel32@lo+4 15023 s_addc_u32 s5, s5, func1@rel32@lo+4 15024 s_swappc_b64 s[30:31], s[4:5] 15025 // ... 15026 s_endpgm 15027 .Lkern1_end: 15028 .size kern1, .Lkern1_end-kern1 15029 15030 .rodata 15031 .p2align 6 15032 .amdhsa_kernel kern1 15033 // ... 15034 .amdhsa_next_free_vgpr .amdgcn.next_free_vgpr 15035 .amdhsa_next_free_sgpr .amdgcn.next_free_sgpr 15036 .end_amdhsa_kernel 15037 15038These symbols cannot identify connected components in order to automatically 15039track the usage for each kernel. However, in some cases careful organization of 15040the kernels and functions in the source file means there is minimal additional 15041effort required to accurately calculate GPR usage. 15042 15043Additional Documentation 15044======================== 15045 15046.. [AMD-GCN-GFX6] `AMD Southern Islands Series ISA <http://developer.amd.com/wordpress/media/2012/12/AMD_Southern_Islands_Instruction_Set_Architecture.pdf>`__ 15047.. [AMD-GCN-GFX7] `AMD Sea Islands Series ISA <http://developer.amd.com/wordpress/media/2013/07/AMD_Sea_Islands_Instruction_Set_Architecture.pdf>`_ 15048.. [AMD-GCN-GFX8] `AMD GCN3 Instruction Set Architecture <http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/AMD_GCN3_Instruction_Set_Architecture_rev1.1.pdf>`__ 15049.. [AMD-GCN-GFX900-GFX904-VEGA] `AMD Vega Instruction Set Architecture <http://developer.amd.com/wordpress/media/2013/12/Vega_Shader_ISA_28July2017.pdf>`__ 15050.. [AMD-GCN-GFX906-VEGA7NM] `AMD Vega 7nm Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/11/Vega_7nm_Shader_ISA_26November2019.pdf>`__ 15051.. [AMD-GCN-GFX908-CDNA1] `AMD Instinct MI100 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA1_Shader_ISA_14December2020.pdf>`__ 15052.. [AMD-GCN-GFX90A-CDNA2] `AMD Instinct MI200 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/CDNA2_Shader_ISA_4February2022.pdf>`__ 15053.. [AMD-GCN-GFX10-RDNA1] `AMD RDNA 1.0 Instruction Set Architecture <https://gpuopen.com/wp-content/uploads/2019/08/RDNA_Shader_ISA_5August2019.pdf>`__ 15054.. [AMD-GCN-GFX10-RDNA2] `AMD RDNA 2 Instruction Set Architecture <https://developer.amd.com/wp-content/resources/RDNA2_Shader_ISA_November2020.pdf>`__ 15055.. [AMD-RADEON-HD-2000-3000] `AMD R6xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R600_Instruction_Set_Architecture.pdf>`__ 15056.. [AMD-RADEON-HD-4000] `AMD R7xx shader ISA <http://developer.amd.com/wordpress/media/2012/10/R700-Family_Instruction_Set_Architecture.pdf>`__ 15057.. [AMD-RADEON-HD-5000] `AMD Evergreen shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_Evergreen-Family_Instruction_Set_Architecture.pdf>`__ 15058.. [AMD-RADEON-HD-6000] `AMD Cayman/Trinity shader ISA <http://developer.amd.com/wordpress/media/2012/10/AMD_HD_6900_Series_Instruction_Set_Architecture.pdf>`__ 15059.. [AMD-ROCm] `AMD ROCm™ Platform <https://rocmdocs.amd.com/>`__ 15060.. [AMD-ROCm-github] `AMD ROCm™ github <http://github.com/RadeonOpenCompute>`__ 15061.. [AMD-ROCm-Release-Notes] `AMD ROCm Release Notes <https://github.com/RadeonOpenCompute/ROCm>`__ 15062.. [CLANG-ATTR] `Attributes in Clang <https://clang.llvm.org/docs/AttributeReference.html>`__ 15063.. [DWARF] `DWARF Debugging Information Format <http://dwarfstd.org/>`__ 15064.. [ELF] `Executable and Linkable Format (ELF) <http://www.sco.com/developers/gabi/>`__ 15065.. [HRF] `Heterogeneous-race-free Memory Models <http://benedictgaster.org/wp-content/uploads/2014/01/asplos269-FINAL.pdf>`__ 15066.. [HSA] `Heterogeneous System Architecture (HSA) Foundation <http://www.hsafoundation.com/>`__ 15067.. [MsgPack] `Message Pack <http://www.msgpack.org/>`__ 15068.. [OpenCL] `The OpenCL Specification Version 2.0 <http://www.khronos.org/registry/cl/specs/opencl-2.0.pdf>`__ 15069.. [SEMVER] `Semantic Versioning <https://semver.org/>`__ 15070.. [YAML] `YAML Ain't Markup Language (YAML™) Version 1.2 <http://www.yaml.org/spec/1.2/spec.html>`__ 15071