[mlir] [VectorOps] generalize printing support for integers

This generalizes printing beyond just i1,i32,i64 and also accounts
for signed and unsigned interpretation in the output.

Reviewed By: nic

[mlir] [VectorOps] generalize printing support for integers

This generalizes printing beyond just i1,i32,i64 and also accounts
for signed and unsigned interpretation in the output.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D88290

show more ...

[MLIR][NFC] Adopt use of TypeRange in build() methods.

- Use TypeRange instead of ArrayRef<Type> where possible.
- Change some of the custom builders to also use TypeRange

Differential Revision: ht

[MLIR][NFC] Adopt use of TypeRange in build() methods.

- Use TypeRange instead of ArrayRef<Type> where possible.
- Change some of the custom builders to also use TypeRange

Differential Revision: https://reviews.llvm.org/D87944

show more ...

[mlir][VectorOps] Loosen restrictions on vector.reduction types

LLVM can deal with any integer or float type, don't arbitrarily restrict
it to f32/f64/i32/i64.

Differential Revision: https://review

[mlir][VectorOps] Loosen restrictions on vector.reduction types

LLVM can deal with any integer or float type, don't arbitrarily restrict
it to f32/f64/i32/i64.

Differential Revision: https://reviews.llvm.org/D88010

show more ...

Rename MemRefDescriptor::getElementType() to MemRefDescriptor::getElementPtrType().

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D87284

[mlir][VectorOps] Redo the scalar loop emission in VectoToSCF to pad instead of clipping

This replaces the select chain for edge-padding with an scf.if that
performs the memory operation when the in

[mlir][VectorOps] Redo the scalar loop emission in VectoToSCF to pad instead of clipping

This replaces the select chain for edge-padding with an scf.if that
performs the memory operation when the index is in bounds and uses the
pad value when it's not. For transfer_write the same mechanism is used,
skipping the store when the index is out of bounds.

The integration test has a bunch of cases of how I believe this should
work.

Differential Revision: https://reviews.llvm.org/D87241

show more ...

[mlir][VectorOps] Simplify code. NFCI.

[mlir] [VectorOps] Improve SIMD compares with narrower indices

When allowed, use 32-bit indices rather than 64-bit indices in the
SIMD computation of masks. This runs up to 2x and 4x faster on
a num

[mlir] [VectorOps] Improve SIMD compares with narrower indices

When allowed, use 32-bit indices rather than 64-bit indices in the
SIMD computation of masks. This runs up to 2x and 4x faster on
a number of AVX2 and AVX512 microbenchmarks.

Reviewed By: bkramer

Differential Revision: https://reviews.llvm.org/D87116

show more ...

[mlir][VectorOps] Fail fast when a strided memref is passed to vector_transfer

Otherwise we'll silently miscompile things.

Differential Revision: https://reviews.llvm.org/D86951

Make helpers static. NFC.

[mlir] Move data layout from LLVMDialect to module Op attributes

Legacy implementation of the LLVM dialect in MLIR contained an instance of
llvm::Module as it was required to parse LLVM IR types. Th

[mlir] Move data layout from LLVMDialect to module Op attributes

Legacy implementation of the LLVM dialect in MLIR contained an instance of
llvm::Module as it was required to parse LLVM IR types. The access to the data
layout of this module was exposed to the users for convenience, but in practice
this layout has always been the default one obtained by parsing an empty layout
description string. Current implementation of the dialect no longer relies on
wrapping LLVM IR types, but it kept an instance of DataLayout for
compatibility. This effectively forces a single data layout to be used across
all modules in a given MLIR context, which is not desirable. Remove DataLayout
from the LLVM dialect and attach it as a module attribute instead. Since MLIR
does not yet have support for data layouts, use the LLVM DataLayout in string
form with verification inside MLIR. Introduce the layout when converting a
module to the LLVM dialect and keep the default "" description for
compatibility.

This approach should be replaced with a proper MLIR-based data layout when it
becomes available, but provides an immediate solution to compiling modules with
different layouts, e.g. for GPUs.

This removes the need for LLVMDialectImpl, which is also removed.

Depends On D85650

Reviewed By: aartbik

Differential Revision: https://reviews.llvm.org/D85652

show more ...

[mlir] Fix mlir build break due to warning when NDEBUG is not set

[mlir][vector] Relax transfer_read/transfer_write restriction on memref operand

Relax the verifier for transfer_read/transfer_write operation so that it can
take a memref with a different element ty

[mlir][vector] Relax transfer_read/transfer_write restriction on memref operand

Relax the verifier for transfer_read/transfer_write operation so that it can
take a memref with a different element type than the vector being read/written.

This is based on the discourse discussion:
https://llvm.discourse.group/t/memref-cast/1514

Differential Revision: https://reviews.llvm.org/D85244

show more ...

[mlir] [VectorOps] Improve lowering of extract_strided_slice (and friends like shape_cast)

Using a shuffle for the last recursive step in progressive lowering not only
results in much more compact I

[mlir] [VectorOps] Improve lowering of extract_strided_slice (and friends like shape_cast)

Using a shuffle for the last recursive step in progressive lowering not only
results in much more compact IR, but also more efficient code (since the
backend is no longer confused on subvector aliasing for longer vectors).

E.g. the following

%f = vector.shape_cast %v0: vector<1024xf32> to vector<32x32xf32>

yields much better x86-64 code that runs 3x faster than the original.

Reviewed By: bkramer, nicolasvasilache

Differential Revision: https://reviews.llvm.org/D85482

show more ...

[mlir] Remove llvm::LLVMContext and llvm::Module from mlir::LLVMDialectImpl

Original modeling of LLVM IR types in the MLIR LLVM dialect had been wrapping
LLVM IR types and therefore required the LLV

[mlir] Remove llvm::LLVMContext and llvm::Module from mlir::LLVMDialectImpl

Original modeling of LLVM IR types in the MLIR LLVM dialect had been wrapping
LLVM IR types and therefore required the LLVMContext in which they were created
to outlive them, which was solved by placing the LLVMContext inside the dialect
and thus having the lifetime of MLIRContext. This has led to numerous issues
caused by the lack of thread-safety of LLVMContext and the need to re-create
LLVM IR modules, obtained by translating from MLIR, in different LLVM contexts
to enable parallel compilation. Similarly, llvm::Module had been introduced to
keep track of identified structure types that could not be modeled properly.

A recent series of commits changed the modeling of LLVM IR types in the MLIR
LLVM dialect so that it no longer wraps LLVM IR types and has no dependence on
LLVMContext and changed the ownership model of the translated LLVM IR modules.
Remove LLVMContext and LLVM modules from the implementation of MLIR LLVM
dialect and clean up the remaining uses.

The only part of LLVM IR that remains necessary for the LLVM dialect is the
data layout. It should be moved from the dialect level to the module level and
replaced with an MLIR-based representation to remove the dependency of the
LLVMDialect on LLVM IR library.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D85445

show more ...

[mlir] take MLIRContext instead of LLVMDialect in getters of LLVMType's

Historical modeling of the LLVM dialect types had been wrapping LLVM IR types
and therefore needed access to the instance of L

[mlir] take MLIRContext instead of LLVMDialect in getters of LLVMType's

Historical modeling of the LLVM dialect types had been wrapping LLVM IR types
and therefore needed access to the instance of LLVMContext stored in the
LLVMDialect. The new modeling does not rely on that and only needs the
MLIRContext that is used for uniquing, similarly to other MLIR types. Change
LLVMType::get<Kind>Ty functions to take `MLIRContext *` instead of
`LLVMDialect *` as first argument. This brings the code base closer to
completely removing the dependence on LLVMContext from the LLVMDialect,
together with additional support for thread-safety of its use.

Depends On D85371

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D85372

show more ...

[mlir] Remove most uses of LLVMDialect::getModule

This prepares for the removal of llvm::Module and LLVMContext from the
mlir::LLVMDialect.

Reviewed By: rriddle

Differential Revision: https://revi

[mlir] Remove most uses of LLVMDialect::getModule

This prepares for the removal of llvm::Module and LLVMContext from the
mlir::LLVMDialect.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D85371

show more ...

[mlir] [VectorOps] Add masked load/store operations to Vector dialect

The intrinsics were already supported and vector.transfer_read/write lowered
direclty into these operations. By providing them a

[mlir] [VectorOps] Add masked load/store operations to Vector dialect

The intrinsics were already supported and vector.transfer_read/write lowered
direclty into these operations. By providing them as individual ops, however,
clients can used them directly, and it opens up progressively lowering transfer
operations at higher levels (rather than direct lowering to LLVM IR as done now).

Reviewed By: bkramer

Differential Revision: https://reviews.llvm.org/D85357

show more ...

[mlir] use the new stateful LLVM type translator by default

Previous type model in the LLVM dialect did not support identified structure
types properly and therefore could use stateless translations

[mlir] use the new stateful LLVM type translator by default

Previous type model in the LLVM dialect did not support identified structure
types properly and therefore could use stateless translations implemented as
free functions. The new model supports identified structs and must keep track
of the identified structure types present in the target context (LLVMContext or
MLIRContext) to avoid creating duplicate structs due to LLVM's type
auto-renaming. Expose the stateful type translation classes and use them during
translation, storing the state as part of ModuleTranslation.

Drop the test type translation mechanism that is no longer necessary and update
the tests to exercise type translation as part of the main translation flow.

Update the code in vector-to-LLVM dialect conversion that relied on stateless
translation to use the new class in a stateless manner.

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D85297

show more ...

[mlir] [VectorOps] Add expand/compress operations to Vector dialect

Introduces the expand and compress operations to the Vector dialect
(important memory operations for sparse computations), togethe

[mlir] [VectorOps] Add expand/compress operations to Vector dialect

Introduces the expand and compress operations to the Vector dialect
(important memory operations for sparse computations), together
with a first reference implementation that lowers to the LLVM IR
dialect to enable running on CPU (and other targets that support
the corresponding LLVM IR intrinsics).

Reviewed By: reidtatge

Differential Revision: https://reviews.llvm.org/D84888

show more ...

[mlir] switch the modeling of LLVM types to use the new mechanism

A new first-party modeling for LLVM IR types in the LLVM dialect has been
developed in parallel to the existing modeling based on wr

[mlir] switch the modeling of LLVM types to use the new mechanism

A new first-party modeling for LLVM IR types in the LLVM dialect has been
developed in parallel to the existing modeling based on wrapping LLVM `Type *`
instances. It resolves the long-standing problem of modeling identified
structure types, including recursive structures, and enables future removal of
LLVMContext and related locking mechanisms from LLVMDialect.

This commit only switches the modeling by (a) renaming LLVMTypeNew to LLVMType,
(b) removing the old implementaiton of LLVMType, and (c) updating the tests. It
is intentionally minimal. Separate commits will remove the infrastructure built
for the transition and update API uses where appropriate.

Depends On D85020

Reviewed By: rriddle

Differential Revision: https://reviews.llvm.org/D85021

show more ...

[mlir] LLVMType: make getUnderlyingType private

The current modeling of LLVM IR types in MLIR is based on the LLVMType class
that wraps a raw `llvm::Type *` and delegates uniquing, printing and pars

[mlir] LLVMType: make getUnderlyingType private

The current modeling of LLVM IR types in MLIR is based on the LLVMType class
that wraps a raw `llvm::Type *` and delegates uniquing, printing and parsing to
LLVM itself. This is model makes thread-safe type manipulation hard and is
being progressively replaced with a cleaner MLIR model that replicates the type
system. In the new model, LLVMType will no longer have an underlying LLVM IR
type. Restrict access to this type in the current model in preparation for the
change.

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D84389

show more ...

[mlir] [VectorOps] Improve scatter/gather CPU performance

Replaced the linearized address with the proper LLVM way of
defining vector of base + indices in SIMD style. This yields
much better code. S

[mlir] [VectorOps] Improve scatter/gather CPU performance

Replaced the linearized address with the proper LLVM way of
defining vector of base + indices in SIMD style. This yields
much better code. Some prototype results with microbencmarking
sparse matrix x vector with 50% sparsity (about 2-3x faster):

LINEARIZED IMPROVED
GFLOPS sdot saxpy sdot saxpy
16x16 1.6 1.4 4.4 2.1
32x32 1.7 1.6 5.8 5.9
64x64 1.7 1.7 6.4 6.4
128x128 1.7 1.7 5.9 5.9
256x256 1.6 1.6 6.1 6.0
512x512 1.4 1.4 4.9 4.7

Reviewed By: nicolasvasilache

Differential Revision: https://reviews.llvm.org/D84368

show more ...

[mlir] [VectorOps] Add scatter/gather operations to Vector dialect

Introduces the scatter/gather operations to the Vector dialect
(important memory operations for sparse computations), together
with

[mlir] [VectorOps] Add scatter/gather operations to Vector dialect

Introduces the scatter/gather operations to the Vector dialect
(important memory operations for sparse computations), together
with a first reference implementation that lowers to the LLVM IR
dialect to enable running on CPU (and other targets that support
the corresponding LLVM IR intrinsics).

The operations can be used directly where applicable, or can be used
during progressively lowering to bring other memory operations closer to
hardware ISA support for a gather/scatter. The semantics of the operation
closely correspond to those of the corresponding llvm intrinsics.

Note that the operation allows for a dynamic index vector (which is
important for sparse computations). However, this first reference
lowering implementation "serializes" the address computation when
base + index_vector is converted to a vector of pointers. Exploring
how to use SIMD properly during these step is TBD. More general
memrefs and idiomatic versions of striding are also TBD.

Reviewed By: arpith-jacob

Differential Revision: https://reviews.llvm.org/D84039

show more ...

[mlir] Add alignment attribute to LLVM memory ops and use in vector.transfer

Summary: The native alignment may generally not be used when lowering a vector.transfer to the underlying load/store oper

[mlir] Add alignment attribute to LLVM memory ops and use in vector.transfer

Summary: The native alignment may generally not be used when lowering a vector.transfer to the underlying load/store operation. This revision fixes the unmasked load/store alignment to match that of the masked path.

Differential Revision: https://reviews.llvm.org/D83684

show more ...

[mlir][VectorOps] Lower vector.fma to llvm.fmuladd instead of llvm.fma

Summary:
These are semantically equivalent, but fmuladd allows decaying the op
into fmul+fadd if there is no fma instruction av

[mlir][VectorOps] Lower vector.fma to llvm.fmuladd instead of llvm.fma

Summary:
These are semantically equivalent, but fmuladd allows decaying the op
into fmul+fadd if there is no fma instruction available. llvm.fma lowers
to scalar calls to libm fmaf, which is a lot slower.

Reviewers: nicolasvasilache, aartbik, ftynse

Subscribers: mehdi_amini, rriddle, jpienaar, shauheen, antiagainst, arpith-jacob, mgester, lucyrfox, liufengdb, stephenneuendorffer, Joonsoo, grosul1, Kayjukh, jurahul, msifontes

Tags: #mlir

Differential Revision: https://reviews.llvm.org/D83666

show more ...