[ARM] Fix for MVE i128 vector icmp costs.

We were hitting an assert as the legalied type needn't be a vector.

Fixes #58364

(cherry picked from commit de6dfbbb300e552efa1cd86a023063a39d408b06)

[AArch64] Add target hook for preferPredicateOverEpilogue

This patch adds the AArch64 hook for preferPredicateOverEpilogue,
which currently returns true if SVE is enabled and one of the
following co

[AArch64] Add target hook for preferPredicateOverEpilogue

This patch adds the AArch64 hook for preferPredicateOverEpilogue,
which currently returns true if SVE is enabled and one of the
following conditions (non-exhaustive) is met:

1. The "sve-tail-folding" option is set to "all", or
2. The "sve-tail-folding" option is set to "all+noreductions"
and the loop does not contain reductions,
3. The "sve-tail-folding" option is set to "all+norecurrences"
and the loop has no first-order recurrences.

Currently the default option is "disabled", but this will be
changed in a later patch.

I've added new tests to show the options behave as expected here:

Transforms/LoopVectorize/AArch64/sve-tail-folding-option.ll

Differential Revision: https://reviews.llvm.org/D129560

show more ...

[LoopVectorize] Add option to use active lane mask for loop control flow

Currently, for vectorised loops that use the get.active.lane.mask
intrinsic we only use the mask for predicated vector operat

[LoopVectorize] Add option to use active lane mask for loop control flow

Currently, for vectorised loops that use the get.active.lane.mask
intrinsic we only use the mask for predicated vector operations,
such as masked loads and stores, etc. The loop itself is still
controlled by comparing the canonical induction variable with the
trip count. However, for some targets this is inefficient when it's
cheap to use the mask itself to control the loop.

This patch adds support for using the active lane mask for control
flow by:

1. Generating the active lane mask for the next iteration of the
vector loop, rather than the current one. If there are still any
remaining iterations then at least the first bit of the mask will
be set.
2. Extract the first bit of this mask and use this bit for the
conditional branch.

I did this by creating a new VPActiveLaneMaskPHIRecipe that sets
up the initial PHI values in the vector loop pre-header. I've also
made use of the new BranchOnCond VPInstruction for the final
instruction in the loop region.

Differential Revision: https://reviews.llvm.org/D125301

show more ...

[ARM] Fix MVE getShuffleCost legalized type check

The MVE shuffle costing for VREV instructions was making incorrect
assumptions as to legalized vector types remaining as vectors. Add a
quick check

[ARM] Fix MVE getShuffleCost legalized type check

The MVE shuffle costing for VREV instructions was making incorrect
assumptions as to legalized vector types remaining as vectors. Add a
quick check to ensure they are indeed vectors before attempting to get
the number of elements.

show more ...

[ARM] Cost modelling for MVE vector fptoi_sat

Building on top of D125665, this adds MVE costs for fptosi.sat and
fptoui.sat, providing MVE is available and the types are legal.

Differential Revisio

[ARM] Cost modelling for MVE vector fptoi_sat

Building on top of D125665, this adds MVE costs for fptosi.sat and
fptoui.sat, providing MVE is available and the types are legal.

Differential Revision: https://reviews.llvm.org/D125666

show more ...

[ARM] Cost modelling for scalar fptoi_sat

Similar to D124357, this adds some cost modelling for fptoi_sat for Arm
targets. Where VFP2 is available (and FP64/FP16 for the relevant types),
the operati

[ARM] Cost modelling for scalar fptoi_sat

Similar to D124357, this adds some cost modelling for fptoi_sat for Arm
targets. Where VFP2 is available (and FP64/FP16 for the relevant types),
the operations are legal as the Arm instructions naturally saturate.
Otherwise they will need an extra smin/smax clamp, similar to AArch64.

Differential Revision: https://reviews.llvm.org/D125665

show more ...

Recommit "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 6a9bbd9f20dcd700e28738788bb63a160c6c088c.

Code review: https://reviews.llvm.or

Recommit "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 6a9bbd9f20dcd700e28738788bb63a160c6c088c.

Code review: https://reviews.llvm.org/D124202

show more ...

Revert "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 55ce296d6f217fd0defed2592ff7b74b79b2c1f0.

[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`

Before this patch `Args` was used to pass a broadcat's arguments by SLP.
This patch changes this. `Args` is no

[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`

Before this patch `Args` was used to pass a broadcat's arguments by SLP.
This patch changes this. `Args` is now used for passing the operands of
the shuffle.

Differential Revision: https://reviews.llvm.org/D124202

show more ...

[ARM] Add missing return to ARMTTIImpl::isLoweredToCall.

I assume we meant to return the result of the call to
BaseT::isLoweredToCall(F).

This might not be a functional change in practice because i

[ARM] Add missing return to ARMTTIImpl::isLoweredToCall.

I assume we meant to return the result of the call to
BaseT::isLoweredToCall(F).

This might not be a functional change in practice because it would
still hit the default case in the switch and call
BaseT::isLoweredToCall(F) at the end.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D123333

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 3, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit e6ead19b774718113007ecb1a4

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 3, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit e6ead19b774718113007ecb1a4449d7af0cbcfeb.

show more ...

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash."

This reverts commit 27bd8f94928201f87f6b659fc2228efd539e8245.

Causes crashes, see comme

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash."

This reverts commit 27bd8f94928201f87f6b659fc2228efd539e8245.

Causes crashes, see comments in D121973

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit f7d7d2a08d16356c57f6d2d36b

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit f7d7d2a08d16356c57f6d2d36bc2fc0589a55df9.

show more ...

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads.""

This reverts commit 79613185d305013de743cdbd6690e4d77c8af27e.

Causes crashes, see comments in https://reviews.llvm.org/D1

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads.""

This reverts commit 79613185d305013de743cdbd6690e4d77c8af27e.

Causes crashes, see comments in https://reviews.llvm.org/D121973.

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads."

Original review: https://reviews.llvm.org/D121354

The original commit 9136145eb019e1d18c966d4d06a3df349b88cc14 broke the build on

Recommit "[SLP] Fix lookahead operand reordering for splat loads."

Original review: https://reviews.llvm.org/D121354

The original commit 9136145eb019e1d18c966d4d06a3df349b88cc14 broke the build on several targets.

Differential Revision: https://reviews.llvm.org/D121973

show more ...

[ARM] Don't look through pointer types in canTailPredicateLoop()

Inspecting the pointer element type here is incompatible with
opaque pointers, and doesn't seem necessary to me. I think the
intentio

[ARM] Don't look through pointer types in canTailPredicateLoop()

Inspecting the pointer element type here is incompatible with
opaque pointers, and doesn't seem necessary to me. I think the
intention might have been to check the type of load/store pointer
arguments, but I believe those should get checked through their
return type or value operand anyway. I don't get any test failures
if I simply drop this.

Differential Revision: https://reviews.llvm.org/D118353

show more ...

[LoopVectorize] Pass a vector type to isLegalMaskedGather/Scatter

This is required to query the legality more precisely in the LoopVectorizer.

This adds another TTI function named 'forceScalarizeMa

[LoopVectorize] Pass a vector type to isLegalMaskedGather/Scatter

This is required to query the legality more precisely in the LoopVectorizer.

This adds another TTI function named 'forceScalarizeMaskedGather/Scatter'
function to work around the hack introduced for MVE, where
isLegalMaskedGather/Scatter would return an answer by second-guessing
where the function was called from, based on the Type passed in (vector
vs scalar). The new interface makes this explicit. It is also used by
X86 to check for vector widths where gather/scatters aren't profitable
(or don't exist) for certain subtargets.

Differential Revision: https://reviews.llvm.org/D115329

show more ...

[ARM] Teach getIntImmCostInst about the cost of saturating fp converts

Given a min(max(fptosi, INT_MIN), INT_MAX) with the correct constants,
we can now generate a fptosi.sat. But in the arm backend

[ARM] Teach getIntImmCostInst about the cost of saturating fp converts

Given a min(max(fptosi, INT_MIN), INT_MAX) with the correct constants,
we can now generate a fptosi.sat. But in the arm backend, the constant
can be treated as high cost, pulling it out of the basic block in a way
that the DAG combine can no longer see it. This teaches it again that it
is a low cost constant, not worth hoisting out.

Recommitted from 0e98659ea1193c with a fix for APInt comparison.

Differential Revision: https://reviews.llvm.org/D114380

show more ...

Revert "[ARM] Teach getIntImmCostInst about the cost of saturating fp converts"

This reverts commit 6d41de380f223c8da02fd4d6a7f7dd1e7a404a24 as the
windows bots are not happy, in a way I do not unde

Revert "[ARM] Teach getIntImmCostInst about the cost of saturating fp converts"

This reverts commit 6d41de380f223c8da02fd4d6a7f7dd1e7a404a24 as the
windows bots are not happy, in a way I do not understand. Revert whilst
we figure out what is wrong.

show more ...

[ARM] Teach getIntImmCostInst about the cost of saturating fp converts

Given a min(max(fptosi, INT_MIN), INT_MAX) with the correct constants,
we can now generate a fptosi.sat. But in the arm backend

[ARM] Teach getIntImmCostInst about the cost of saturating fp converts

Given a min(max(fptosi, INT_MIN), INT_MAX) with the correct constants,
we can now generate a fptosi.sat. But in the arm backend, the constant
can be treated as high cost, pulling it out of the basic block in a way
that the DAG combine can no longer see it. This teaches it again that it
is a low cost constant, not worth hoisting out.

Differential Revision: https://reviews.llvm.org/D114380

show more ...

[ADT] Add APInt::isNegatedPowerOf2() helper

Inspired by D111968, provide a isNegatedPowerOf2() wrapper instead of obfuscating code with (-Value).isPowerOf2() patterns, which I'm sure are likely aven

[ADT] Add APInt::isNegatedPowerOf2() helper

Inspired by D111968, provide a isNegatedPowerOf2() wrapper instead of obfuscating code with (-Value).isPowerOf2() patterns, which I'm sure are likely avenues for typos.....

Differential Revision: https://reviews.llvm.org/D111998

show more ...

[APInt] Stop using soft-deprecated constructors and methods in llvm. NFC.

Stop using APInt constructors and methods that were soft-deprecated in
D109483. This fixes all the uses I found in llvm, exc

[APInt] Stop using soft-deprecated constructors and methods in llvm. NFC.

Stop using APInt constructors and methods that were soft-deprecated in
D109483. This fixes all the uses I found in llvm, except for the APInt
unit tests which should still test the deprecated methods.

Differential Revision: https://reviews.llvm.org/D110807

show more ...

[ARM] Mark <= -1 immediate constant as cheap

A <= -1 constant on a compare can be converted to a < 0 operation, which
is usually cheap. If we mark the constant as cheap, preventing hoisting,
we allo

[ARM] Mark <= -1 immediate constant as cheap

A <= -1 constant on a compare can be converted to a < 0 operation, which
is usually cheap. If we mark the constant as cheap, preventing hoisting,
we allow that fold to happen even across different blocks.

Differential Revision: https://reviews.llvm.org/D109360

show more ...

[Target] Migrate from getNumArgOperands to arg_size (NFC)

Note that getNumArgOperands is considered a legacy name. See
llvm/include/llvm/IR/InstrTypes.h for details.

[ARM] Teach DemandedVectorElts about VMOVN lanes

The class of instructions that write to narrow top/bottom lanes only
demand the even or odd elements of the input lanes. Which means that a
pair of V

[ARM] Teach DemandedVectorElts about VMOVN lanes

The class of instructions that write to narrow top/bottom lanes only
demand the even or odd elements of the input lanes. Which means that a
pair of VMOVNT; VMOVNB demands no lanes from the original input. This
teaches that to instcombine from the target hooks available through
ARMTTIImpl.

Differential Revision: https://reviews.llvm.org/D109325

show more ...