[VP] Legalize the stride operand for EXPERIMENTAL_VP_STRIDED SDNodes

Add promotion and expansion of integer operands for
experimental_vp_strided SelectionDAG nodes; the expansion is actually
just a

[VP] Legalize the stride operand for EXPERIMENTAL_VP_STRIDED SDNodes

Add promotion and expansion of integer operands for
experimental_vp_strided SelectionDAG nodes; the expansion is actually
just a truncation of the stride operand.

Reviewed By: simoll

Differential Revision: https://reviews.llvm.org/D123112

show more ...

[stackmaps] Legalise patchpoint arguments.

This is similar to D125680, but for llvm.experimental.patchpoint
(instead of llvm.experimental.stackmap).

Differential review: https://reviews.llvm.org/D1

[stackmaps] Legalise patchpoint arguments.

This is similar to D125680, but for llvm.experimental.patchpoint
(instead of llvm.experimental.stackmap).

Differential review: https://reviews.llvm.org/D129268

show more ...

[stackmaps] Start legalizing live variable operands

Prior to this change, live variable operands passed to
`llvm.experimental.stackmap` would be emitted directly to target nodes,
meaning that they d

[stackmaps] Start legalizing live variable operands

Prior to this change, live variable operands passed to
`llvm.experimental.stackmap` would be emitted directly to target nodes,
meaning that they don't get legalised. The upshot of this is that LLVM
may crash when encountering illegally typed target nodes.

e.g. https://github.com/llvm/llvm-project/issues/21657

This change introduces a platform independent stackmap DAG node whose
operands are legalised as per usual, thus avoiding aforementioned
crashes.

Note that some kinds of argument are still not handled properly, namely
vectors, structs, and large integers, like i128s. These will need to be
addressed in follow-up changes.

Note also that this does not change the behaviour of
`llvm.experimental.patchpoint`. A follow up change will do the same for
this intrinsic.

Differential review:
https://reviews.llvm.org/D125680

show more ...

[LegalizeTypes][NFC] Merge promote SPLAT_VECTOR and promote SCALAR_TO_VECTOR to one function

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D127825

[LegalizeTypes] Don't assume fshl/fshr shift amount type matches the other operands.

Like other shifts, the type isn't required to match. We shouldn't
assume we can call ZExtPromotedInteger.

I test

[LegalizeTypes] Don't assume fshl/fshr shift amount type matches the other operands.

Like other shifts, the type isn't required to match. We shouldn't
assume we can call ZExtPromotedInteger.

I tested the PromoteIntOp_FunnelShift locally by removing the promotion
of the shift amount from PromoteIntRes_FunnelShift. But with the final
version of this patch it is never executed on any tests.

Differential Revision: https://reviews.llvm.org/D125106

show more ...

Intrinsic for checking floating point class

This change introduces a new intrinsic, `llvm.is.fpclass`, which checks
if the provided floating-point number belongs to any of the the specified
value cl

Intrinsic for checking floating point class

This change introduces a new intrinsic, `llvm.is.fpclass`, which checks
if the provided floating-point number belongs to any of the the specified
value classes. The intrinsic implements the checks made by C standard
library functions `isnan`, `isinf`, `isfinite`, `isnormal`, `issubnormal`,
`issignaling` and corresponding IEEE-754 operations.

The primary motivation for this intrinsic is the support of strict FP
mode. In this mode using compare instructions or other FP operations is
not possible, because if the value is a signaling NaN, floating-point
exception `Invalid` is raised, but the aforementioned functions must
never raise exceptions.

Currently there are two solutions for this problem, both are
implemented partially. One of them is using integer operations to
implement the check. It was implemented in https://reviews.llvm.org/D95948
for `isnan`. It solves the problem of exceptions, but offers one
solution for all targets, although some can do the check in more
efficient way.

The other, implemented in https://reviews.llvm.org/D96568, introduced a
hook 'clang::TargetCodeGenInfo::testFPKind', which injects a target
specific code into IR to implement `isnan` and some other functions. It is
convenient for targets that have dedicated instruction to determine FP data
class. However using target-specific intrinsic complicates analysis and can
prevent some optimizations.

A special intrinsic for value class checks allows representing data class
tests with enough flexibility. During IR transformations it represents the
check in target-independent way and saves it from undesired transformations.
In the instruction selector it allows efficient lowering depending on the
used target and mode.

This implementation is an extended variant of `llvm.isnan` introduced
in https://reviews.llvm.org/D104854. It is limited to minimal intrinsic
support. Target-specific treatment will be implemented in separate
patches.

Differential Revision: https://reviews.llvm.org/D112025

show more ...

Cleanup includes: DebugInfo & CodeGen

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121332

[LegalizeTypes][VP] Add splitting support for vp.gather and vp.scatter

Split these nodes in a similar way as their masked versions.

Reviewed By: frasercrmck, craig.topper

Differential Revision: ht

[LegalizeTypes][VP] Add splitting support for vp.gather and vp.scatter

Split these nodes in a similar way as their masked versions.

Reviewed By: frasercrmck, craig.topper

Differential Revision: https://reviews.llvm.org/D117760

show more ...

[LegalizeTypes][VP] Add widening support for vp.gather and vp.scatter

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D117557

[LegalizeTypes][VP] Add widening support for vp.reduce.*

When widening these intrinsics, we do not have to insert neutral
elements at the end of the vector as when widening vector.reduce.*
intrinsic

[LegalizeTypes][VP] Add widening support for vp.reduce.*

When widening these intrinsics, we do not have to insert neutral
elements at the end of the vector as when widening vector.reduce.*
intrinsics, thanks to vector predication semantics.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D117467

show more ...

[LegalizeTypes][VP] Add splitting support for vp.reduction.*

Split vp.reduction.* intrinsics by splitting the vector to reduce in
two halves, perform the reduction operation in each one of them and

[LegalizeTypes][VP] Add splitting support for vp.reduction.*

Split vp.reduction.* intrinsics by splitting the vector to reduce in
two halves, perform the reduction operation in each one of them and
accumulate the results of both operations.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D117469

show more ...

Revert "[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants"

This reverts commit 197f3c0deb76951315118ef13937b67ea9cbd5aa.

Reverting after miscompilation errors disc

Revert "[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants"

This reverts commit 197f3c0deb76951315118ef13937b67ea9cbd5aa.

Reverting after miscompilation errors discovered with ffmpeg.

show more ...

[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants

When we know the value we're extending is a negative constant then it
makes sense to use SIGN_EXTEND because this

[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants

When we know the value we're extending is a negative constant then it
makes sense to use SIGN_EXTEND because this may improve code quality in
some cases, particularly when doing a constant splat of an unpacked vector
type. For example, for SVE when splatting the value -1 into all elements
of a vector of type <vscale x 2 x i32> the element type will get promoted
from i32 -> i64. In this case we want the splat value to sign-extend from
(i32 -1) -> (i64 -1), whereas currently it zero-extends from
(i32 -1) -> (i64 0xFFFFFFFF). Sign-extending the constant means we can use
a single mov immediate instruction.

New tests added here:

CodeGen/AArch64/sve-vector-splat.ll

I believe we see some code quality improvements in these existing
tests too:

CodeGen/AArch64/reduce-and.ll
CodeGen/AArch64/unfold-masked-merge-vector-variablemask.ll

The apparent regressions in CodeGen/AArch64/fast-isel-cmp-vec.ll only
occur because the test disables codegen prepare and branch folding.

Differential Revision: https://reviews.llvm.org/D114357

show more ...

[SelectionDAG][VP] Add splitting/widening for VP_LOAD and VP_STORE

Original patch by @hussainjk.

This patch was split off from D109377 to keep vector legalization
(widening/splitting) separate from

[SelectionDAG][VP] Add splitting/widening for VP_LOAD and VP_STORE

Original patch by @hussainjk.

This patch was split off from D109377 to keep vector legalization
(widening/splitting) separate from vector element legalization
(promoting).

While the original patch added a third overload of
SelectionDAG::getVPStore, this patch takes the liberty of collapsing
those all down to 1, as three overloads seems excessive for a
little-used node.

The original patch also used ModifyToType in places, but that method
still crashes on scalable vector types. Seeing as the other VP
legalization methods only work when all operands need identical
widening, this patch follows in that vein.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D117235

show more ...

Revert "[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants"

This reverts commit 31009f0b5afb504fc1f30769c038e1b7be6ea45b.

It seems to be causing SVE VLA buildbot fa

Revert "[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants"

This reverts commit 31009f0b5afb504fc1f30769c038e1b7be6ea45b.

It seems to be causing SVE VLA buildbot failures and has introduced a
genuine regression. Reverting for now.

show more ...

[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants

When we know the value we're extending is a negative constant then it
makes sense to use SIGN_EXTEND because this

[CodeGen][AArch64] Ensure isSExtCheaperThanZExt returns true for negative constants

When we know the value we're extending is a negative constant then it
makes sense to use SIGN_EXTEND because this may improve code quality in
some cases, particularly when doing a constant splat of an unpacked vector
type. For example, for SVE when splatting the value -1 into all elements
of a vector of type <vscale x 2 x i32> the element type will get promoted
from i32 -> i64. In this case we want the splat value to sign-extend from
(i32 -1) -> (i64 -1), whereas currently it zero-extends from
(i32 -1) -> (i64 0xFFFFFFFF). Sign-extending the constant means we can use
a single mov immediate instruction.

New tests added here:

CodeGen/AArch64/sve-vector-splat.ll

I believe we see some code quality improvements in these existing
tests too:

CodeGen/AArch64/dag-numsignbits.ll
CodeGen/AArch64/reduce-and.ll
CodeGen/AArch64/unfold-masked-merge-vector-variablemask.ll

The apparent regressions in CodeGen/AArch64/fast-isel-cmp-vec.ll only
occur because the test disables codegen prepare and branch folding.

Differential Revision: https://reviews.llvm.org/D114357

show more ...

[LegalizeTypes][VP] Add splitting support for vp.select

Split vp.select in a similar way as vselect, splitting also the length
parameter.

Reviewed By: craig.topper

Differential Revision: https://r

[LegalizeTypes][VP] Add splitting support for vp.select

Split vp.select in a similar way as vselect, splitting also the length
parameter.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D116651

show more ...

[LegalizeTypes] Remove IsVP argument from type legalization methods. NFC

We can either check the opcode or number of operands or use
ISD::isVPOpcode inside the methods.

In some places I've used num

[LegalizeTypes] Remove IsVP argument from type legalization methods. NFC

We can either check the opcode or number of operands or use
ISD::isVPOpcode inside the methods.

In some places I've used number of operands figuring that it is
cheaper than isVPOpcode. I've included isVPOpcode in an assert to
verify.

Reviewed By: frasercrmck

Differential Revision: https://reviews.llvm.org/D116578

show more ...

[LegalizeTypes][VP] Add integer promotion support for vp.select

Promote select, vselect and vp.select in a similar way.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D1

[LegalizeTypes][VP] Add integer promotion support for vp.select

Promote select, vselect and vp.select in a similar way.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D116400

show more ...

[LegalizeTypes][VP] Add widening support for vp.select

Widen vp.select the same way as select and vselect.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D116407

[SelectionDAG] Allow FindMemType to fail when widening loads & stores

This patch removes an internal failure found in FindMemType and "bubbles
it up" to the users of that method: GenWidenVectorLoads

[SelectionDAG] Allow FindMemType to fail when widening loads & stores

This patch removes an internal failure found in FindMemType and "bubbles
it up" to the users of that method: GenWidenVectorLoads and
GenWidenVectorStores. FindMemType -- renamed findMemType -- now returns
an optional value, returning None if no such type is found.

Each of the aforementioned users now pre-calculates the list of types it
will use to widen the memory access. If the type breakdown is not
possible they will signal a failure, at which point the compiler will
crash as it does currently.

This patch is preparing the ground for alternative legalization
strategies for vector loads and stores, such as using vector-predication
versions of loads or stores.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D112000

show more ...

[SelectionDAG] Widening the result of INSERT_SUBVECTOR.

Widens the result and first input vector because they have the same size.
The subvector to be inserted is widened in the operand widen functio

[SelectionDAG] Widening the result of INSERT_SUBVECTOR.

Widens the result and first input vector because they have the same size.
The subvector to be inserted is widened in the operand widen function.

Differential Revision: https://reviews.llvm.org/D112187

show more ...

Revert "[SelectionDAG] Remove PromoteIntOp_EXTRACT_SUBVECTOR."

This reverts commit 3e8d2008f74245e9e1ca60bc97e4e619b0d42c6c.

The code removed in this commit is actually required for extracting
fixe

Revert "[SelectionDAG] Remove PromoteIntOp_EXTRACT_SUBVECTOR."

This reverts commit 3e8d2008f74245e9e1ca60bc97e4e619b0d42c6c.

The code removed in this commit is actually required for extracting
fixed types from illegal scalable types, hence this commit causes
assertion failures in such extracts.

show more ...

[ISel] Legalized arithmetic.fence.f128 for 32-bits target

Reviewed By: Craig Topper, Wang Pengfei

Differential Revision: https://reviews.llvm.org/D110467

[RISCV][VP] Add support for VP_REDUCE_* operations

This patch adds codegen support for lowering the vector-predicated
reduction intrinsics to RVV instructions. The process is similar to that
of the

[RISCV][VP] Add support for VP_REDUCE_* operations

This patch adds codegen support for lowering the vector-predicated
reduction intrinsics to RVV instructions. The process is similar to that
of the other reduction intrinsics, save for the fact that every VP
reduction has a start value. We reuse the existing custom "VL" nodes,
adding extra patterns where required to handle non-true masks.

To support these nodes, the `RISCVISD::VECREDUCE_*_VL` nodes have been
given an explicit "merge" operand. This is to faciliate the VP
reductions, where we must be careful to ensure that even if no operation
is performed (when VL=0) we still produce the start value. The RVV
reductions don't update the destination register under these conditions,
so we tie the splatted start value to the output register.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D107657

show more ...