[mlir] Flip accessors to prefixed form (NFC)

Another mechanical sweep to keep diff small for flip to _Prefixed.

[mlir] Don't use Optional::hasValue (NFC)

[mlir][gpu] Move GPU headers into IR/ and Transforms/

Depends on D127350

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D127352

[MLIR][GPU] Add canonicalizer for gpu.memcpy

Erase gpu.memcpy op when only uses of dest are
the memcpy op in question, its allocation and deallocation
ops.

Reviewed By: bondhugula, csigg

Different

[MLIR][GPU] Add canonicalizer for gpu.memcpy

Erase gpu.memcpy op when only uses of dest are
the memcpy op in question, its allocation and deallocation
ops.

Reviewed By: bondhugula, csigg

Differential Revision: https://reviews.llvm.org/D124257

show more ...

[MLIR][GPU] NFC: simplify kernel operand accessor implementations.

Reviewed By: ThomasRaoux

Differential Revision: https://reviews.llvm.org/D125112

[mlir][gpu] Move async copy ops to NVGPU and add caching hints

Move async copy operations to NVGPU as they only exist on NV target and are
designed to match ptx semantic. This allows us to also add

[mlir][gpu] Move async copy ops to NVGPU and add caching hints

Move async copy operations to NVGPU as they only exist on NV target and are
designed to match ptx semantic. This allows us to also add more fine grain
caching hint attribute to the op.
Add hint to bypass L1 and hook it up to NVVM op.

Differential Revision: https://reviews.llvm.org/D125244

show more ...

[AsmParser] Introduce a new "Argument" abstraction + supporting logic

MLIR has a common pattern for "arguments" that uses syntax
like `%x : i32 {attrs} loc("sourceloc")` which is implemented
in adho

[AsmParser] Introduce a new "Argument" abstraction + supporting logic

MLIR has a common pattern for "arguments" that uses syntax
like `%x : i32 {attrs} loc("sourceloc")` which is implemented
in adhoc ways throughout the codebase. The approach this uses
is verbose (because it is implemented with parallel arrays) and
inconsistent (e.g. lots of things drop source location info).

Solve this by introducing OpAsmParser::Argument and make addRegion
(which sets up BlockArguments for the region) take it. Convert the
world to propagating this down. This means that we correctly
capture and propagate source location information in a lot more
cases (e.g. see the affine.for testcase example), and it also
simplifies much code.

Differential Revision: https://reviews.llvm.org/D124649

show more ...

[AsmParser] Rework logic around "region argument parsing"

The asm parser had a notional distinction between parsing an
operand (like "%foo" or "%4#3") and parsing a region argument
(which isn't supp

[AsmParser] Rework logic around "region argument parsing"

The asm parser had a notional distinction between parsing an
operand (like "%foo" or "%4#3") and parsing a region argument
(which isn't supposed to allow a result number like #3).

Unfortunately the implementation has two problems:

1) It didn't actually check for the result number and reject
it. parseRegionArgument and parseOperand were identical.
2) It had a lot of machinery built up around it that paralleled
operand parsing. This also was functionally identical, but
also had some subtle differences (e.g. the parseOptional
stuff had a different result type).

I thought about just removing all of this, but decided that the
missing error checking was important, so I reimplemented it with
a `allowResultNumber` flag on parseOperand. This keeps the
codepaths unified and adds the missing error checks.

Differential Revision: https://reviews.llvm.org/D124470

show more ...

[AsmParser/Printer] Rework sourceloc support for function arguments.

When Location tracking support for block arguments was added, we
discussed various approaches to threading support for this throu

[AsmParser/Printer] Rework sourceloc support for function arguments.

When Location tracking support for block arguments was added, we
discussed various approaches to threading support for this through
function-like argument parsing. At the time, we added a parallel array
of locations that could hold this. It turns out that that approach was
verbose and error prone, roughly no one adopted it.

This patch takes a different approach, adding an optional source
locator to the UnresolvedOperand class. This fits much more naturally
into the standard structure we use for representing locators, and gives
all the function like dialects locator support for free (e.g. see the
test adding an example for the LLVM dialect).

Differential Revision: https://reviews.llvm.org/D124188

show more ...

Revert D121279 "[MLIR][GPU] Add canonicalizer for gpu.memcpy"

This reverts commit 12f55cac69d8978d1c433756a8b2114bf9ed1e1b.

Causes miscompile. Will follow up with a reproduce.

Add async dependencies support for gpu.launch op

Add async dependencies support for gpu.launch op: this allows specifying
a list of async tokens ("streams") as dependencies for the launch.

Update t

Add async dependencies support for gpu.launch op

Add async dependencies support for gpu.launch op: this allows specifying
a list of async tokens ("streams") as dependencies for the launch.

Update the GPU kernel outlining pass lowering to propagate async
dependencies from gpu.launch to gpu.launch_func op. Previously, a new
stream was being created and destroyed for a kernel launch. The async
deps support allows the kernel launch to be serialized on an existing
stream.

Differential Revision: https://reviews.llvm.org/D123499

show more ...

[MLIR] NFC. Drop trailing white space in GPU async ops print

NFC. Drop trailing end of line white space in GPU async ops' printer
whenever the list of async deps is empty.

Reviewed By: mehdi_amini,

[MLIR] NFC. Drop trailing white space in GPU async ops print

NFC. Drop trailing end of line white space in GPU async ops' printer
whenever the list of async deps is empty.

Reviewed By: mehdi_amini, rriddle

Differential Revision: https://reviews.llvm.org/D123754

show more ...

[MLIR][GPU] Add canonicalizer for gpu.memcpy

Fold away gpu.memcpy op when only uses of dest are
the memcpy op in question, its allocation and deallocation
ops.

Reviewed By: bondhugula

Differential

[MLIR][GPU] Add canonicalizer for gpu.memcpy

Fold away gpu.memcpy op when only uses of dest are
the memcpy op in question, its allocation and deallocation
ops.

Reviewed By: bondhugula

Differential Revision: https://reviews.llvm.org/D121279

show more ...

Apply clang-tidy fixes for readability-identifier-naming in GPUDialect.cpp (NFC)

[MLIR][GPU] Add canonicalization patterns for folding simple gpu.wait ops.

* Fold away redundant %t = gpu.wait async + gpu.wait [%t] pairs.

* Fold away %t = gpu.wait async ... ops when %t has no us

[MLIR][GPU] Add canonicalization patterns for folding simple gpu.wait ops.

* Fold away redundant %t = gpu.wait async + gpu.wait [%t] pairs.

* Fold away %t = gpu.wait async ... ops when %t has no uses.

* Fold away gpu.wait [] ops.

* In case of %t1 = gpu.wait async [%t0], replace all uses of %t1
with %t0.

Differential Revision: https://reviews.llvm.org/D121878

show more ...

[mlir][gpu] Relax restriction on mma load/store op

Those ops can support more complex layout as long as the most inner
dimension is contiguous.

Differential Revision: https://reviews.llvm.org/D1224

[mlir][gpu] Relax restriction on mma load/store op

Those ops can support more complex layout as long as the most inner
dimension is contiguous.

Differential Revision: https://reviews.llvm.org/D122452

show more ...

[mlir] Rename `OpAsmParser::OperandType` to `OpAsmParser::UnresolvedOperand`

I am not sure about the meaning of Type in the name (was it meant be interpreted as Kind?), and given the importance and

[mlir] Rename `OpAsmParser::OperandType` to `OpAsmParser::UnresolvedOperand`

I am not sure about the meaning of Type in the name (was it meant be interpreted as Kind?), and given the importance and meaning of Type in the context of MLIR, its probably better to rename it. Given the comment in the source code, the suggestion in the GitHub issue and the final discussions in the review, this patch renames the OperandType to UnresolvedOperand.

Fixes https://github.com/llvm/llvm-project/issues/54446

Differential Revision: https://reviews.llvm.org/D122142

show more ...

[mlir:FunctionOpInterface] Rename the "type" attribute to "function_type"

This removes any potential confusion with the `getType` accessors
which correspond to SSA results of an operation, and makes

[mlir:FunctionOpInterface] Rename the "type" attribute to "function_type"

This removes any potential confusion with the `getType` accessors
which correspond to SSA results of an operation, and makes it
clear what the intent is (i.e. to represent the type of the function).

Differential Revision: https://reviews.llvm.org/D121762

show more ...

[mlir] Remove the deprecated ODS Op verifier/parser/printer code blocks

These have been deprecated for ~1 month now and can be removed.

Differential Revision: https://reviews.llvm.org/D121090

[mlir] Support verification order (3/3)

In this CL, update the function name of verifier according to the
behavior. If a verifier needs to access the region then it'll be updated
to `verifyRegions`.

[mlir] Support verification order (3/3)

In this CL, update the function name of verifier according to the
behavior. If a verifier needs to access the region then it'll be updated
to `verifyRegions`.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D120373

show more ...

Add verifier for gpu.alloc op

Add verifier for gpu.alloc op to verify if the dimension operand counts
and symbol operand counts are same as their memref counterparts.

Differential Revision: https:/

Add verifier for gpu.alloc op

Add verifier for gpu.alloc op to verify if the dimension operand counts
and symbol operand counts are same as their memref counterparts.

Differential Revision: https://reviews.llvm.org/D117427

show more ...

[mlir][gpu] Add device side async copy operations

Add new operations to the gpu dialect to represent device side
asynchronous copies. This also add the lowering of those operations to
nvvm dialect.

[mlir][gpu] Add device side async copy operations

Add new operations to the gpu dialect to represent device side
asynchronous copies. This also add the lowering of those operations to
nvvm dialect.
Those ops are meant to be low level and map directly to llvm dialects
like nvvm or rocdl.

We can further add higher level of abstraction by building on top of
those operations.
This has been discuss here:
https://discourse.llvm.org/t/modeling-gpu-async-copy-ampere-feature/4924

Differential Revision: https://reviews.llvm.org/D119191

show more ...

[mlir][GPU] Add ShuffleOp builder for constant offset/width

Differential Revision: https://reviews.llvm.org/D119345

[mlir] Update uses of `parser`/`printer` ODS op field to `hasCustomAssemblyFormat`

The parser/printer fields are deprecated and in the process of being removed.

[mlir][NFC] Update GPU/SCF operations to use `hasVerifier` instead of `verifier`

The verifier field is deprecated, and slated for removal.

Differential Revision: https://reviews.llvm.org/D118827