[mlir] Flip LinAlg dialect to _Both

This one required more changes than ideal due to overlapping generated name
with different return types. Changed getIndexingMaps to getIndexingMapsArray to
move i

[mlir] Flip LinAlg dialect to _Both

This one required more changes than ideal due to overlapping generated name
with different return types. Changed getIndexingMaps to getIndexingMapsArray to
move it out of the way/highlight that it returns (more expensively) a
SmallVector and uses the prefixed name for the Attribute.

Differential Revision: https://reviews.llvm.org/D129919

show more ...

Don't combine if there would remain no true reduction dim.

Differential Revision: https://reviews.llvm.org/D130109

Modernize Optional::{getValueOr,hasValue}

[mlir] Use value instead of getValue (NFC)

lowerParallel is also called on unit-size, one-sided reduction dims

See: https://gist.github.com/bjacob/d8be8ec7e70ed0be4b3a5794ced2a7e8

Differential Revision: https://reviews.llvm.org/D129096

[mlir] Use has_value instead of hasValue (NFC)

[mlir][vector] Add accumulator operand to MultiDimReduce op

This allows vectorizing linalg reductions without changing the operation
order. Therefore this produce a valid vectorization even if opera

[mlir][vector] Add accumulator operand to MultiDimReduce op

This allows vectorizing linalg reductions without changing the operation
order. Therefore this produce a valid vectorization even if operations
are not associative.

Differential Revision: https://reviews.llvm.org/D129535

show more ...

CombineContractBroadcast should not create dims unused in LHS+RHS

Differential Revision: https://reviews.llvm.org/D129087

Fix CombineContractBroadcast folding reduction iterators.

Fix CombineContractBroadcast folding reduction iterators.

Differential Revision: https://reviews.llvm.org/D128739

[mlir][Vector] Fix reordering of floating point adds during lower of `vector.contract`.

Adding the accumulator value after the `vector.contract` changes the
precision of the operation. This makes su

[mlir][Vector] Fix reordering of floating point adds during lower of `vector.contract`.

Adding the accumulator value after the `vector.contract` changes the
precision of the operation. This makes sure the accumulator is carried
through to `vector.reduce` (and down to LLVM).

Differential Revision: https://reviews.llvm.org/D128674

show more ...

Revert "Don't use Optional::hasValue (NFC)"

This reverts commit aa8feeefd3ac6c78ee8f67bf033976fc7d68bc6d.

Don't use Optional::hasValue (NFC)

[mlir] move SCF headers to SCF/{IR,Transforms} respectively

This aligns the SCF dialect file layout with the majority of the dialects.

Reviewed By: jpienaar

Differential Revision: https://reviews.

[mlir] move SCF headers to SCF/{IR,Transforms} respectively

This aligns the SCF dialect file layout with the majority of the dialects.

Reviewed By: jpienaar

Differential Revision: https://reviews.llvm.org/D128049

show more ...

[mlir][vector] Fix contraction op lowering with mixed types

contraction op can have mixed type, add support for this case to the pattern
lowering contraction op to outerproduct.

Differential Revisi

[mlir][vector] Fix contraction op lowering with mixed types

contraction op can have mixed type, add support for this case to the pattern
lowering contraction op to outerproduct.

Differential Revision: https://reviews.llvm.org/D127926

show more ...

[mlir][vector] Add new lowering mode to vector.contractionOp

Add lowering for cases where the reduction dimension is fully unrolled.
It is common to unroll the reduction dimension, therefore we woul

[mlir][vector] Add new lowering mode to vector.contractionOp

Add lowering for cases where the reduction dimension is fully unrolled.
It is common to unroll the reduction dimension, therefore we would want
to lower the contractions to an elementwise vector op in this case.

Differential Revision: https://reviews.llvm.org/D126120

show more ...

[mlir][vector] Fix crash in DropInnerMostUnitDims pattern

Fix number of dimensions when incrementally replacing dimensions in
affine map.

Differential Revision: https://reviews.llvm.org/D125984

[mlir][vector] Reorder elementwise(transpose)

Similar to the existing pattern for reodering cast(transpose),
this makes transpose following transpose and increases the chance
of embedding the transp

[mlir][vector] Reorder elementwise(transpose)

Similar to the existing pattern for reodering cast(transpose),
this makes transpose following transpose and increases the chance
of embedding the transposition inside contraction op. Actually
cast ops are just special instances of elementwise ops.

Reviewed By: ThomasRaoux

Differential Revision: https://reviews.llvm.org/D123596

show more ...

Apply clang-tidy fixes for llvm-qualified-auto in VectorTransforms.cpp (NFC)

[mlir][vector][nfc] Rename index optimizations option

We are using "enable-index-optimizations" and "indexOptimizations" as
names for an optimization that consists of using i32 for indices within
a

[mlir][vector][nfc] Rename index optimizations option

We are using "enable-index-optimizations" and "indexOptimizations" as
names for an optimization that consists of using i32 for indices within
a vector. For instance, when building a vector comparison for mask
generation. The name is confusing and suggests a scope beyond these
vector indices. This change makes the function of the option explicit
in its name.

Differential Revision: https://reviews.llvm.org/D122415

show more ...

[mlir] Flip Vector dialect accessors used to prefixed form.

This has been on _Both for a couple of weeks. Flip usages in core with
intention to flip flag to _Prefixed in follow up. Needed to add a c

[mlir] Flip Vector dialect accessors used to prefixed form.

This has been on _Both for a couple of weeks. Flip usages in core with
intention to flip flag to _Prefixed in follow up. Needed to add a couple
of helper methods in AffineOps and Linalg to facilitate a pure flag flip
in follow up as some of these classes are used in templates and so
sensitive to Vector dialect changes.

Differential Revision: https://reviews.llvm.org/D122151

show more ...

[bazel] Add missing dependency after a75a46db89

[mlir][Vector] Enable create_mask for scalable vectors

The way vector.create_mask is currently lowered is
vector-length-dependent, and therefore incompatible with scalable vector
types. This patch a

[mlir][Vector] Enable create_mask for scalable vectors

The way vector.create_mask is currently lowered is
vector-length-dependent, and therefore incompatible with scalable vector
types. This patch adds an alternative lowering path for create_mask
operations that return a scalable vector mask.

Differential Revision: https://reviews.llvm.org/D118248

show more ...

[mlir] Make OpBuilder::createOperation to accept raw inputs

This provides a way to create an operation without manipulating
OperationState directly. This is useful for creating unregistered ops.

Re

[mlir] Make OpBuilder::createOperation to accept raw inputs

This provides a way to create an operation without manipulating
OperationState directly. This is useful for creating unregistered ops.

Reviewed By: rriddle, mehdi_amini

Differential Revision: https://reviews.llvm.org/D120787

show more ...

[mlir][Vector] Modernize default lowering of vector transpose

This patch removes an old recursive implementation to lower vector.transpose to extract/insert operations
and replaces it with a iterati

[mlir][Vector] Modernize default lowering of vector transpose

This patch removes an old recursive implementation to lower vector.transpose to extract/insert operations
and replaces it with a iterative approach that leverages newer linearization/delinearization utilities.
The patch should be NFC except by the order in which the extract/insert ops are generated.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D121321

show more ...

[mlir][Vector] Use create_mask in transfer mask materializations

Currently, the transfer mask is materialized by generating the vector
comparison: [offset + 0, .., offset + length - 1] < [dim, .., d

[mlir][Vector] Use create_mask in transfer mask materializations

Currently, the transfer mask is materialized by generating the vector
comparison: [offset + 0, .., offset + length - 1] < [dim, .., dim]

A better alternative is to materialize the transfer mask by using the
operation: `vector.create_mask (dim - offset)`, which will generate
simpler code and compose better with scalable vectors.

Differential Revision: https://reviews.llvm.org/D120487

show more ...