[Matrix] Add assert to catch extracted vectors with poison elements

Assert when the extracted vector is wider than the row/column.

Differential Revision: https://reviews.llvm.org/D130173

[Matrix] Refactor tiled loops in a struct. NFC

The three loops have the same structure: index, header, latch.

[LowerMatrixMultiplication] Switch dummy values from undef to poison [NFC]

Revert [LowerMatrixMultiplication] Switch dummy values from undef to poison [NFC]

This reverts commits 47e6f98f84ac3 and 3e701bcd2a6aee2

[LowerMatrixMultiplication] Switch dummy values from undef to poison [NFC]

[Matrix] Check if iterator is at beginning of BB in optimizeTranspose.

If an instruction at the beginning of a block is erased, this may
trigger crash due to dereferencing an invalid iterator.

Che

[Matrix] Check if iterator is at beginning of BB in optimizeTranspose.

If an instruction at the beginning of a block is erased, this may
trigger crash due to dereferencing an invalid iterator.

Check if II is at the end before dereferencing it.

Reviewed By: thegameg

Differential Revision: https://reviews.llvm.org/D127736

show more ...

[Scalar] Use llvm::make_early_inc_range (NFC)

Cleanup includes: Transform/Scalar

Estimated impact on preprocessor output line:
before: 1062981579
after: 1062494547

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cl

Cleanup includes: Transform/Scalar

Estimated impact on preprocessor output line:
before: 1062981579
after: 1062494547

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120817

show more ...

Cleanup includes: TransformsUtils

Estimation on the impact on preprocessor output:
before: 1065307662
after: 1064800684

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-

Cleanup includes: TransformsUtils

Estimation on the impact on preprocessor output:
before: 1065307662
after: 1064800684

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120741

show more ...

[MatrixBuilder] Remove unnecessary IRBuilder template (NFC)

IRBuilderBase exists specifically to avoid the need for this.

[Matrix] Use ArrayType for allocas instead of VectorType.

When creating an alloca to copy a matrix due to memory conflicts, those
allocas used to use VectorTypes, which forced them to have huge
alig

[Matrix] Use ArrayType for allocas instead of VectorType.

When creating an alloca to copy a matrix due to memory conflicts, those
allocas used to use VectorTypes, which forced them to have huge
alignments for large vectors.

This patch updates LowerMatrixIntrinsics to use a corresponding array
type, like Clang already does, to get more manageable alignments.

Reviewed By: anemet, thegameg

Differential Revision: https://reviews.llvm.org/D118239

show more ...

[LowerMatrixIntrinsics] Call getRegisterClassForType before getNumberOfRegisters.

getNumberOfRegisters takes a ClassID as it's argument. It shouldn't be passed a bool. Assuming the bool meant vector

[LowerMatrixIntrinsics] Call getRegisterClassForType before getNumberOfRegisters.

getNumberOfRegisters takes a ClassID as it's argument. It shouldn't be passed a bool. Assuming the bool meant vector or not, we should call getRegisterClassForType first.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D116903

show more ...

[llvm] Remove redundant member initialization (NFC)

Identified with readability-redundant-member-init.

[LowerMatrixIntrinsics] writeFnName - don't dereference a dyn_cast<>. NFC.

dyn_cast<> can return null - use cast<> instead to assert the cast is valid before dereferencing the casted pointer.

Fixes

[LowerMatrixIntrinsics] writeFnName - don't dereference a dyn_cast<>. NFC.

dyn_cast<> can return null - use cast<> instead to assert the cast is valid before dereferencing the casted pointer.

Fixes static-analyzer null dereference warning.

show more ...

Revert "[llvm] Remove redundant member initialization (NFC)"

This reverts commit fd4808887ee47f3ec8a030e9211169ef4fb094c3.

This patch causes gcc to issue a lot of warnings like:

warning: base cl

Revert "[llvm] Remove redundant member initialization (NFC)"

This reverts commit fd4808887ee47f3ec8a030e9211169ef4fb094c3.

This patch causes gcc to issue a lot of warnings like:

warning: base class ‘class llvm::MCParsedAsmOperand’ should be
explicitly initialized in the copy constructor [-Wextra]

show more ...

[llvm] Remove redundant member initialization (NFC)

Identified with readability-redundant-member-init.

[NPM] Added -print-pipeline-passes print params for a few passes.

Added '-print-pipeline-passes' printing of parameters for those passes
declared with *_WITH_PARAMS macro in PassRegistry.def.

Note

[NPM] Added -print-pipeline-passes print params for a few passes.

Added '-print-pipeline-passes' printing of parameters for those passes
declared with *_WITH_PARAMS macro in PassRegistry.def.

Note that it only prints the parameters declared inside *_WITH_PARAMS as
in a few cases there appear to be additional parameters not parsable.

The following passes are now covered (i.e. all of those with *_WITH_PARAMS in
PassRegistry.def).

LoopExtractorPass - loop-extract
HWAddressSanitizerPass - hwsan
EarlyCSEPass - early-cse
EntryExitInstrumenterPass - ee-instrument
LowerMatrixIntrinsicsPass - lower-matrix-intrinsics
LoopUnrollPass - loop-unroll
AddressSanitizerPass - asan
MemorySanitizerPass - msan
SimplifyCFGPass - simplifycfg
LoopVectorizePass - loop-vectorize
MergedLoadStoreMotionPass - mldst-motion
GVN - gvn
StackLifetimePrinterPass - print<stack-lifetime>
SimpleLoopUnswitchPass - simple-loop-unswitch

Differential Revision: https://reviews.llvm.org/D109310

show more ...

[Scalar] Use make_early_inc_range (NFC)

Recommit "[Matrix] Overload stride arg in matrix.columnwise.load/store."

This reverts the revert 28c04794df74ad3c38155a244729d1f8d57b9400.

The failing MLIR test that caused the revert should be fix

Recommit "[Matrix] Overload stride arg in matrix.columnwise.load/store."

This reverts the revert 28c04794df74ad3c38155a244729d1f8d57b9400.

The failing MLIR test that caused the revert should be fixed in this
version.

Also includes a PPC test fix previously in 1f87c7c478a6.

show more ...

Revert "[Matrix] Overload stride arg in matrix.columnwise.load/store."

This reverts commit a1ef81de35a4bac6d3b22e9d7186d880124d7a55.

Broke the MLIR buildbot.

[Matrix] Overload stride arg in matrix.columnwise.load/store.

This patch adjusts the intrinsics definition of
llvm.matrix.column.major.load and llvm.matrix.column.major.store to
allow overloading th

[Matrix] Overload stride arg in matrix.columnwise.load/store.

This patch adjusts the intrinsics definition of
llvm.matrix.column.major.load and llvm.matrix.column.major.store to
allow overloading the type of the stride. The bitwidth of the stride is
used to perform the offset computation.

This fixes a crash when using __builtin_matrix_column_major_load or
__builtin_matrix_column_major_store on 32 bit platforms. The stride argument
of the builtins are defined as `size_t`, which is 32 bits wide on 32 bit
platforms.

Note that we still perform offset computations with 64 bit width on 32
bit platforms for accesses that do not take a user-specified stride.
This can be fixed separately.

Fixes PR51304.

Reviewed By: erichkeane

Differential Revision: https://reviews.llvm.org/D107349

show more ...

[Matrix] Fix shape for factored transpose

The shape of the input is C x R.

Differential Revision: https://reviews.llvm.org/D106722

[Matrix] RAUW should only replace an instruction in ShapeMap if supportsShapeInfo

As an instruction is replaced in optimizeTransposes RAUW will replace it in
the ShapeMap (ShapeMap is ValueMap so th

[Matrix] RAUW should only replace an instruction in ShapeMap if supportsShapeInfo

As an instruction is replaced in optimizeTransposes RAUW will replace it in
the ShapeMap (ShapeMap is ValueMap so that uses are updated). In
finalizeLowering however we skip updating uses if they are in the ShapeMap
since they will be lowered separately at which point we pick up the lowered
operands.

In the testcase what happened was that since we replaced the doubled-transpose
with the shuffle, it ended up in the ShapeMap. As we lowered the
columnwise-load the use in the shuffle was not updated. Then as we removed
the original columnwise-load we changed that to an undef. I.e. we ended up
with:

```
%shuf = shufflevector <8 x double> undef, <8 x double> poison, <6 x i32>
^^^^^
<i32 0, i32 1, i32 2, i32 4, i32 5, i32 6>
```

Besides the fix itself, I have fortified this last bit. As we change uses to
undef when removing instruction we track the undefed instruction to make sure
we eventually remove those too. This would have caught the issue at compile
time.

Differential Revision: https://reviews.llvm.org/D106714

show more ...

[Matrix] Fix -Wunused-variable in -DLLVM_ENABLE_ASSERTIONS=off build after D106457. NFC

[Matrix] Fix miscompile for NT matmul if the transpose has other use

We should only add the fake lowering entry for the matrix remark if the
transpose is not lowered on its own. `MapVector::insert`

[Matrix] Fix miscompile for NT matmul if the transpose has other use

We should only add the fake lowering entry for the matrix remark if the
transpose is not lowered on its own. `MapVector::insert` is used to insert
the entry during proper lowering which does not overwrite the fake entry in
the map.

We actually had test coverage for this but the reference output code was
wrong; it was storing undef rather than the transposed column.

Also add an assert that would have caught this.

Differential Revision: https://reviews.llvm.org/D106457

show more ...