[mlir][Linalg] Retire LinalgPromotion pattern

This revision removes the LinalgPromotion pattern and adds a `transform.structured.promotion` op.
Since the LinalgPromotion transform allows the injecti

[mlir][Linalg] Retire LinalgPromotion pattern

This revision removes the LinalgPromotion pattern and adds a `transform.structured.promotion` op.
Since the LinalgPromotion transform allows the injection of arbitrary C++ via lambdas, the current
transform op does not handle it.
It is left for future work to decide what the right transform op control is for those cases.

Note the underlying implementation remains unchanged and the mechanism is still controllable by
lambdas from the API.

During this refactoring it was also determined that the `dynamicBuffers` option does not actually
connect to a change of behavior in the algorithm.
This also exhibits that the related test is wrong (and dangerous).
Both the option and the test are therefore removed.

Lastly, a test that connects patterns using the filter-based mechanism is removed: all the independent
pieces are already tested separately.

Context: https://discourse.llvm.org/t/psa-retire-linalg-filter-based-patterns/63785

Differential Revision: https://reviews.llvm.org/D129649

show more ...

[mlir][NFC] Update textual references of `func` to `func.func` in Linalg tests

The special case parsing of `func` operations is being removed.

Reland [GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

Reland Note: Adds a fix to properly mark a commutative operation as folded if we change the or

Reland [GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

Reland Note: Adds a fix to properly mark a commutative operation as folded if we change the order
of its operands. This was uncovered by the fact that we no longer re-process constants.

This avoids accidentally reversing the order of constants during successive
application, e.g. when running the canonicalizer. This helps reduce the number
of iterations, and also avoids unnecessary changes to input IR.

Fixes #51892

Differential Revision: https://reviews.llvm.org/D122692

show more ...

Revert "[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down"

This reverts commit 59bbc7a0851b6e0054bb3ed47df0958822f08880.

This exposes an issue breakin

Revert "[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down"

This reverts commit 59bbc7a0851b6e0054bb3ed47df0958822f08880.

This exposes an issue breaking the contract of
`applyPatternsAndFoldGreedily` where we "converge" without applying
remaining patterns.

show more ...

[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

This avoids accidentally reversing the order of constants during successive
application, e.g. when ru

[GreedPatternRewriter] Preprocess constants while building worklist when not processing top down

This avoids accidentally reversing the order of constants during successive
application, e.g. when running the canonicalizer. This helps reduce the number
of iterations, and also avoids unnecessary changes to input IR.

Fixes #51892

Differential Revision: https://reviews.llvm.org/D122692

show more ...

[mlir][linalg] Replace linalg.fill by OpDSL variant.

The revision removes the linalg.fill operation and renames the OpDSL generated linalg.fill_tensor operation to replace it. After the change, all

[mlir][linalg] Replace linalg.fill by OpDSL variant.

The revision removes the linalg.fill operation and renames the OpDSL generated linalg.fill_tensor operation to replace it. After the change, all named structured operations are defined via OpDSL and there are no handwritten operations left.

A side-effect of the change is that the pretty printed form changes from:
```
%1 = linalg.fill(%cst, %0) : f32, tensor<?x?xf32> -> tensor<?x?xf32>
```
changes to
```
%1 = linalg.fill ins(%cst : f32) outs(%0 : tensor<?x?xf32>) -> tensor<?x?xf32>
```
Additionally, the builder signature now takes input and output value ranges as it is the case for all other OpDSL operations:
```
rewriter.create<linalg::FillOp>(loc, val, output)
```
changes to
```
rewriter.create<linalg::FillOp>(loc, ValueRange{val}, ValueRange{output})
```
All other changes remain minimal. In particular, the canonicalization patterns are the same and the `value()`, `output()`, and `result()` methods are now implemented by the FillOpInterface.

Depends On D120726

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D120728

show more ...

Revert "Revert "[mlir] Purge `linalg.copy` and use `memref.copy` instead.""

This reverts commit 25bf6a2a9bc6ecb3792199490c70c4ce50a94aea.

Revert "[mlir] Purge `linalg.copy` and use `memref.copy` instead."

This reverts commit 016956b68081705ffee511c334e31e414fa1ddbf.
Reverting it to fix NVidia build without being in a hurry.

[mlir] Purge `linalg.copy` and use `memref.copy` instead.

Differential Revision: https://reviews.llvm.org/D118028

[MLIR] Replace std ops with arith dialect ops

Precursor: https://reviews.llvm.org/D110200

Removed redundant ops from the standard dialect that were moved to the
`arith` or `math` dialects.

Renamed

[MLIR] Replace std ops with arith dialect ops

Precursor: https://reviews.llvm.org/D110200

Removed redundant ops from the standard dialect that were moved to the
`arith` or `math` dialects.

Renamed all instances of operations in the codebase and in tests.

Reviewed By: rriddle, jpienaar

Differential Revision: https://reviews.llvm.org/D110797

show more ...

[mlir][linalg] Change the pretty printed FillOp operand order.

The patch changes the pretty printed FillOp operand order from output, value to value, output. The change is a follow up to https://rev

[mlir][linalg] Change the pretty printed FillOp operand order.

The patch changes the pretty printed FillOp operand order from output, value to value, output. The change is a follow up to https://reviews.llvm.org/D104121 that passes the fill value using a scalar input instead of the former capture semantics.

Differential Revision: https://reviews.llvm.org/D104356

show more ...

[mlir][Linalg] Drop spurious usage of OperationFolder

Instead, use createOrFold builders which result in more static information available.

Differential Revision: https://reviews.llvm.org/D102832

[mlir][linalg] Remove IndexedGenericOp support from LinalgInterchangePattern...

after introducing the IndexedGenericOp to GenericOp canonicalization (https://reviews.llvm.org/D101612).

Differential

[mlir][linalg] Remove IndexedGenericOp support from LinalgInterchangePattern...

after introducing the IndexedGenericOp to GenericOp canonicalization (https://reviews.llvm.org/D101612).

Differential Revision: https://reviews.llvm.org/D102245

show more ...

[mlir] Support complex numbers in Linalg promotion

FillOp allows complex ops, and filling a properly sized buffer with
a default zero complex number is implemented.

Differential Revision: https://r

[mlir] Support complex numbers in Linalg promotion

FillOp allows complex ops, and filling a properly sized buffer with
a default zero complex number is implemented.

Differential Revision: https://reviews.llvm.org/D99939

show more ...

[MLIR, test] Fix use of undef FileCheck var

MLIR test Dialect/Linalg/transform-patterns.mlir tries to check for the
absence of a sequence of instructions with several CHECK-NOT with one of
those dir

[MLIR, test] Fix use of undef FileCheck var

MLIR test Dialect/Linalg/transform-patterns.mlir tries to check for the
absence of a sequence of instructions with several CHECK-NOT with one of
those directives using a variable defined in another. However CHECK-NOT
are checked independently so that is using a variable defined in a
pattern that should not occur in the input.

This commit removes the dependency between those CHECK-NOT by replacing
occurences of variables by the regex that were used to define them.

Note to reviewers: please pay attention to whether the remaining
reference to l0 is correct. There was a l0 defined in those CHECK-NOT
and one defined before. I'm not sure what was the intent there.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D99957

show more ...

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit 361b7d125b438cda13fa45f13790767a62252be9 by Chris
Lattner <[email protected]>

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit 361b7d125b438cda13fa45f13790767a62252be9 by Chris
Lattner <[email protected]> dated Fri Mar 19 21:22:15 2021 -0700.

The change to the greedy rewriter driver picking a different order was
made without adequate analysis of the trade-offs and experimentation. A
change like this has far reaching consequences on transformation
pipelines, and a major impact upstream and downstream. For eg., one
can’t be sure that it doesn’t slow down a large number of cases by small
amounts or create other issues. More discussion here:
https://llvm.discourse.group/t/speeding-up-canonicalize/3015/25

Reverting this so that improvements to the traversal order can be made
on a clean slate, in bigger steps, and higher bar.

Differential Revision: https://reviews.llvm.org/D99329

show more ...

[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants.

This reapplies b5d9a3c / https://reviews.llvm.org/D98609 with a one line fix in
processExistingConstants to

[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants.

This reapplies b5d9a3c / https://reviews.llvm.org/D98609 with a one line fix in
processExistingConstants to skip() when erasing a constant we've already seen.

Original commit message:

1) Change the canonicalizer to walk the function in top-down order instead of
bottom-up order. This composes well with the "top down" nature of constant
folding and simplification, reducing iterations and re-evaluation of ops in
simple cases.
2) Explicitly enter existing constants into the OperationFolder table before
canonicalizing. Previously we would "constant fold" them and rematerialize
them, wastefully recreating a bunch fo constants, which lead to pointless
memory traffic.

Both changes together provide a 33% speedup for canonicalize on some mid-size
CIRCT examples.

One artifact of this change is that the constants generated in normal pattern
application get inserted at the top of the function as the patterns are applied.
Because of this, we get "inverted" constants more often, which is an aethetic
change to the IR but does permute some testcases.

Differential Revision: https://reviews.llvm.org/D99006

show more ...

[MLIR] Create memref dialect and move dialect-specific ops from std.

Create the memref dialect and move dialect-specific ops
from std dialect to this dialect.

Moved ops:
AllocOp -> MemRef_AllocOp
A

[MLIR] Create memref dialect and move dialect-specific ops from std.

Create the memref dialect and move dialect-specific ops
from std dialect to this dialect.

Moved ops:
AllocOp -> MemRef_AllocOp
AllocaOp -> MemRef_AllocaOp
AssumeAlignmentOp -> MemRef_AssumeAlignmentOp
DeallocOp -> MemRef_DeallocOp
DimOp -> MemRef_DimOp
MemRefCastOp -> MemRef_CastOp
MemRefReinterpretCastOp -> MemRef_ReinterpretCastOp
GetGlobalMemRefOp -> MemRef_GetGlobalOp
GlobalMemRefOp -> MemRef_GlobalOp
LoadOp -> MemRef_LoadOp
PrefetchOp -> MemRef_PrefetchOp
ReshapeOp -> MemRef_ReshapeOp
StoreOp -> MemRef_StoreOp
SubViewOp -> MemRef_SubViewOp
TransposeOp -> MemRef_TransposeOp
TensorLoadOp -> MemRef_TensorLoadOp
TensorStoreOp -> MemRef_TensorStoreOp
TensorToMemRefOp -> MemRef_BufferCastOp
ViewOp -> MemRef_ViewOp

The roadmap to split the memref dialect from std is discussed here:
https://llvm.discourse.group/t/rfc-split-the-memref-dialect-from-std/2667

Differential Revision: https://reviews.llvm.org/D98041

show more ...

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit b5d9a3c92358349d5444ab28de8ab5b2bee33a01.

The commit introduced a memory error

Revert "[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants."

This reverts commit b5d9a3c92358349d5444ab28de8ab5b2bee33a01.

The commit introduced a memory error in canonicalization/operation
walking that is exposed when compiled with ASAN. It leads to crashes in
some "release" configurations.

show more ...

[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants.

Two changes:
1) Change the canonicalizer to walk the function in top-down order instead of
bottom-up or

[Canonicalizer] Process regions top-down instead of bottom up & reuse existing constants.

Two changes:
1) Change the canonicalizer to walk the function in top-down order instead of
bottom-up order. This composes well with the "top down" nature of constant
folding and simplification, reducing iterations and re-evaluation of ops in
simple cases.
2) Explicitly enter existing constants into the OperationFolder table before
canonicalizing. Previously we would "constant fold" them and rematerialize
them, wastefully recreating a bunch fo constants, which lead to pointless
memory traffic.

Both changes together provide a 33% speedup for canonicalize on some mid-size
CIRCT examples.

One artifact of this change is that the constants generated in normal pattern
application get inserted at the top of the function as the patterns are applied.
Because of this, we get "inverted" constants more often, which is an aethetic
change to the IR but does permute some testcases.

Differential Revision: https://reviews.llvm.org/D98609

show more ...

Revert "[MLIR] Create memref dialect and move several dialect-specific ops from std."

This commit introduced a cyclic dependency:
Memref dialect depends on Standard because it used ConstantIndexOp.

Revert "[MLIR] Create memref dialect and move several dialect-specific ops from std."

This commit introduced a cyclic dependency:
Memref dialect depends on Standard because it used ConstantIndexOp.
Std depends on the MemRef dialect in its EDSC/Intrinsics.h

Working on a fix.

This reverts commit 8aa6c3765b924d86f623d452777eb76b83bf2787.

show more ...

[MLIR] Create memref dialect and move several dialect-specific ops from std.

Create the memref dialect and move several dialect-specific ops without
dependencies to other ops from std dialect to thi

[MLIR] Create memref dialect and move several dialect-specific ops from std.

Create the memref dialect and move several dialect-specific ops without
dependencies to other ops from std dialect to this dialect.

Moved ops:
AllocOp -> MemRef_AllocOp
AllocaOp -> MemRef_AllocaOp
DeallocOp -> MemRef_DeallocOp
MemRefCastOp -> MemRef_CastOp
GetGlobalMemRefOp -> MemRef_GetGlobalOp
GlobalMemRefOp -> MemRef_GlobalOp
PrefetchOp -> MemRef_PrefetchOp
ReshapeOp -> MemRef_ReshapeOp
StoreOp -> MemRef_StoreOp
TransposeOp -> MemRef_TransposeOp
ViewOp -> MemRef_ViewOp

The roadmap to split the memref dialect from std is discussed here:
https://llvm.discourse.group/t/rfc-split-the-memref-dialect-from-std/2667

Differential Revision: https://reviews.llvm.org/D96425

show more ...

[mlir] Generalize OpFoldResult usage in ops with offsets, sizes and operands.

This revision starts evolving the APIs to manipulate ops with offsets, sizes and operands towards a ValueOrAttr abstract

[mlir] Generalize OpFoldResult usage in ops with offsets, sizes and operands.

This revision starts evolving the APIs to manipulate ops with offsets, sizes and operands towards a ValueOrAttr abstraction that is already used in folding under the name OpFoldResult.

The objective, in the future, is to allow such manipulations all the way to the level of ODS to avoid all the genuflexions involved in distinguishing between values and attributes for generic constant foldings.

Once this evolution is accepted, the next step will be a mechanical OpFoldResult -> ValueOrAttr.

Differential Revision: https://reviews.llvm.org/D95310

show more ...

[mlir][linalg] Add vectorization for element-wise linalg ops

Add support for vectorization for linalg.generic representing element-wise ops.
Those are converted to transfer_read + vector ops + trans

[mlir][linalg] Add vectorization for element-wise linalg ops

Add support for vectorization for linalg.generic representing element-wise ops.
Those are converted to transfer_read + vector ops + transfer_write.
Also re-organize the vectorization tests to be together.

Implementation derived from the work of @burmako, @agrue and
@fedelebron.

Differential Revision: https://reviews.llvm.org/D92540

show more ...

[mlir][vector] Make linalg FillOp vectorization use Transfer op

Differential Revision: https://reviews.llvm.org/D90474