[SelectionDAG] Fix fptoi.sat scalable vector lowering

Vector fptosi_sat and fptoui_sat were being expanded by unrolling the
vector operation. This doesn't work for scalable vector, so this patch
add

[SelectionDAG] Fix fptoi.sat scalable vector lowering

Vector fptosi_sat and fptoui_sat were being expanded by unrolling the
vector operation. This doesn't work for scalable vector, so this patch
adds a call to TLI.expandFP_TO_INT_SAT if the vector is scalable.

Scalable tests are added for AArch64 and RISCV. Some of the AArch64
fptoi_sat operations should be legal, but that will be handled in
another patch.

Differential Revision: https://reviews.llvm.org/D130028

show more ...

[RISCV][VP] Add basic RVV codegen for vp.fcmp

This patch adds the necessary infrastructure to lower vp.fcmp via
ISD::VP_SETCC to RVV instructions.

Most notably this patch adds cond-code legalizatio

[RISCV][VP] Add basic RVV codegen for vp.fcmp

This patch adds the necessary infrastructure to lower vp.fcmp via
ISD::VP_SETCC to RVV instructions.

Most notably this patch adds cond-code legalization for VP_SETCC,
reusing the existing TargetLowering::LegalizeSetCCCondCode by passing in
additional SDValue parameters for the Mask and EVL. This method then
uses VP operations to legalize the condcode.

There is still a general lack of canonicalization on VP_SETCC as opposed
to SETCC which results in worse code than is theoretically possible.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D123051

show more ...

Cleanup include: codegen second round

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D122180

Cleanup includes: DebugInfo & CodeGen

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121332

Simplify mask creation with llvm::seq. NFCI.

[SelectionDAG][VP] Provide expansion for VP_MERGE

This patch adds support for expanding VP_MERGE through a sequence of
vector operations producing a full-length mask setting up the elements
past EVL

[SelectionDAG][VP] Provide expansion for VP_MERGE

This patch adds support for expanding VP_MERGE through a sequence of
vector operations producing a full-length mask setting up the elements
past EVL/pivot to be false, combining this with the original mask, and
culminating in a full-length vector select.

This expansion should work for any data type, though the only use for
RVV is for boolean vectors, which themselves rely on an expansion for
the VSELECT.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D118058

show more ...

[VP][ISel] use LEGALPOS for legalization action

Use the VPIntrinsics.def's LEGALPOS that is specified with every VP
SDNode to determine which return or operand value type shall be used to
infer the

[VP][ISel] use LEGALPOS for legalization action

Use the VPIntrinsics.def's LEGALPOS that is specified with every VP
SDNode to determine which return or operand value type shall be used to
infer the legalization action.

Reviewed By: frasercrmck

Differential Revision: https://reviews.llvm.org/D116594

show more ...

[RISCV][VP] Add RVV codegen for [nX]vXi1 vp.select

Expand [nX]vXi1 vp.select the same way as [nX]vXi1 vselect.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D115546

[LegalizeVectorOps] code refactor for LegalizeOp; NFC

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D115636

[LegalizeVectorOps] fix a typo

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D115637

Remove unused variable. NFC.

[DAG] Update expandFunnelShift/expandROT to return the expansion directly. NFCI.

Don't return a bool to indicate if the expansion was successful, just return the SDValue result directly, like we do

[DAG] Update expandFunnelShift/expandROT to return the expansion directly. NFCI.

Don't return a bool to indicate if the expansion was successful, just return the SDValue result directly, like we do for most other basic expansions.

show more ...

[RISCV] Expand scalable vector bswap. Fix crash for bitreverse.

Fix LegalizeVectorOps to not try shuffle or unrolling expansions for
scalable vectors.

Differential Revision: https://reviews.llvm.or

[RISCV] Expand scalable vector bswap. Fix crash for bitreverse.

Fix LegalizeVectorOps to not try shuffle or unrolling expansions for
scalable vectors.

Differential Revision: https://reviews.llvm.org/D112236

show more ...

[TargetLowering] Simplify the interface of expandABS. NFC

Instead of returning a bool to indicate success and a separate
SDValue, return the SDValue and have the callers check if it is
null.

Review

[TargetLowering] Simplify the interface of expandABS. NFC

Instead of returning a bool to indicate success and a separate
SDValue, return the SDValue and have the callers check if it is
null.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D112331

show more ...

[TargetLowering] Simplify the interface for expandCTPOP/expandCTLZ/expandCTTZ.

There is no need to return a bool and have an SDValue output
parameter. Just return the SDValue and let the caller chec

[TargetLowering] Simplify the interface for expandCTPOP/expandCTLZ/expandCTTZ.

There is no need to return a bool and have an SDValue output
parameter. Just return the SDValue and let the caller check if it
is null.

I have another patch to add more callers of these so I thought
I'd clean up the interface first.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D112267

show more ...

[LegalizeVectorOps][X86] Don't defer BITREVERSE expansion to LegalizeDAG.

By expanding early it allows the shifts to be custom lowered in
LegalizeVectorOps. Then a DAG combine is able to run on them

[LegalizeVectorOps][X86] Don't defer BITREVERSE expansion to LegalizeDAG.

By expanding early it allows the shifts to be custom lowered in
LegalizeVectorOps. Then a DAG combine is able to run on them before
LegalizeDAG handles the BUILD_VECTORS for the masks used.

v16Xi8 shift lowering on X86 requires a mask to be applied to a v8i16
shift. The BITREVERSE expansion applied an AND mask before SHL ops and
after SRL ops. This was done to share the same mask constant for both shifts.
It looks like this patch allows DAG combine to remove the AND mask added
after v16i8 SHL by X86 lowering. This maintains the mask sharing that
BITREVERSE was trying to achieve. Prior to this patch it looks like
we kept the mask after the SHL instead which required an extra constant
pool or a PANDN to invert it.

This is dependent on D112248 because RISCV will end up scalarizing the BSWAP
portion of the BITREVERSE expansion if we don't disable BSWAP scalarization in
LegalizeVectorOps first.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D112254

show more ...

[TargetLowering][RISCV] Prevent scalarization of fixed vector bswap.

It's better to do the ands, shifts, ors in the vector domain than
to scalarize it and do those operations on each element.

Revie

[TargetLowering][RISCV] Prevent scalarization of fixed vector bswap.

It's better to do the ands, shifts, ors in the vector domain than
to scalarize it and do those operations on each element.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D112248

show more ...

[SelectionDAG] Use DAG.getNOT to further simplify some code. NFC

Followup to D109483

[CodeGen] Use DAG.getAllOnesConstant where possible to simplify code. NFC.

[APInt] Normalize naming on keep constructors / predicate methods.

This renames the primary methods for creating a zero value to `getZero`
instead of `getNullValue` and renames predicates like `isAl

[APInt] Normalize naming on keep constructors / predicate methods.

This renames the primary methods for creating a zero value to `getZero`
instead of `getNullValue` and renames predicates like `isAllOnesValue`
to simply `isAllOnes`. This achieves two things:

1) This starts standardizing predicates across the LLVM codebase,
following (in this case) ConstantInt. The word "Value" doesn't
convey anything of merit, and is missing in some of the other things.

2) Calling an integer "null" doesn't make any sense. The original sin
here is mine and I've regretted it for years. This moves us to calling
it "zero" instead, which is correct!

APInt is widely used and I don't think anyone is keen to take massive source
breakage on anything so core, at least not all in one go. As such, this
doesn't actually delete any entrypoints, it "soft deprecates" them with a
comment.

Included in this patch are changes to a bunch of the codebase, but there are
more. We should normalize SelectionDAG and other APIs as well, which would
make the API change more mechanical.

Differential Revision: https://reviews.llvm.org/D109483

show more ...

[NFC] Simply update a FIXME comment

X86 overrided LowerOperationWrapper was moved to common implementation
in a7eae62.

[AArch64][SVE] Add support for fixed length MSCATTER/MGATHER

Since gather lowering can now lower to nodes that may need expansion via
the vector legalizer, do MGATHER lowering via vector legalizer.

[AArch64][SVE] Add support for fixed length MSCATTER/MGATHER

Since gather lowering can now lower to nodes that may need expansion via
the vector legalizer, do MGATHER lowering via vector legalizer.

Additionally, as part of adding passthru support for fixed typed
gathers, fix passthru support for scalable types.

Depends on D104910

Differential Revision: https://reviews.llvm.org/D104217

show more ...

[SelectionDAG][RISCV] Don't unroll 0/1-type bool VSELECTs

This patch extends the cases in which the legalizer is able to express
VSELECT in terms of XOR/AND/OR. When dealing with a VSELECT between
b

[SelectionDAG][RISCV] Don't unroll 0/1-type bool VSELECTs

This patch extends the cases in which the legalizer is able to express
VSELECT in terms of XOR/AND/OR. When dealing with a VSELECT between
boolean vector types, the mask itself is an all-ones or all-ones value
of the operand type, so a 0/1 boolean type behaves identically to a 0/-1
type.

This greatly helps RISC-V which relies on expansion for these nodes. It
also allows scalable-vector bool VSELECTs to use the default expansion,
where before it would crash in SelectionDAG::UnrollVectorOp.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D103147

show more ...

[LegalizeVectorOps][RISCV] Add scalable-vector SELECT expansion

This patch extends VectorLegalizer::ExpandSELECT to permit expansion
also for scalable vector types. The only real change is condition

[LegalizeVectorOps][RISCV] Add scalable-vector SELECT expansion

This patch extends VectorLegalizer::ExpandSELECT to permit expansion
also for scalable vector types. The only real change is conditionally
checking for BUILD_VECTOR or SPLAT_VECTOR legality depending on the
vector type.

We can use this to fix "cannot select" errors for scalable vector
selects on the RISCV target. Note that in future patches RISCV will
possibly custom-lower vector SELECTs to VSELECTs for branchless codegen.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D102063

show more ...

[SelectionDAG][AArch64][SVE] Perform SETCC condition legalization in LegalizeVectorOps

This is currently performed in SelectionDAGLegalize, here we make it also
happen in LegalizeVectorOps, allowing

[SelectionDAG][AArch64][SVE] Perform SETCC condition legalization in LegalizeVectorOps

This is currently performed in SelectionDAGLegalize, here we make it also
happen in LegalizeVectorOps, allowing a target to lower the SETCC condition
codes first in LegalizeVectorOps and then lower to a custom node afterwards,
without having to duplicate all of the SETCC condition legalization in the
target specific lowering.

As a result of this, fixed length floating point SETCC nodes can now be
properly lowered for SVE.

Differential Revision: https://reviews.llvm.org/D98939

show more ...