[RISCV] Add cost model for vector.reverse mask operation

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D128784

[RISCV] Fix mistake in RISCVTTIImpl::getIntImmCostInst.

zext.w requires Zba not Zbb. The test was also wrong, but had the
correct comment.

[RISCV] Cost model for scalable reductions

This extends the existing cost model for reductions for scalable vectors.

The existing cost model assumes that reductions are roughly logarithmic in cost

[RISCV] Cost model for scalable reductions

This extends the existing cost model for reductions for scalable vectors.

The existing cost model assumes that reductions are roughly logarithmic in cost for unordered variants and linear for ordered ones. This change keeps that same basic model, and extends it out to the maximum number of elements a scalable vector could possibly have.

This results in costs which aren't terribly high for unordered reductions, but are for ordered ones. This seems about right; we want to strongly bias away from using scalable ordered reductions if the cost might be linear in VL.

Differential Revision: https://reviews.llvm.org/D127447

show more ...

[RISCV] Implement getVScaleForTuning and thus prefer scalable vectorization when enabled

LoopVectorizer uses getVScaleForTuning for deciding how to discount the cost of a potential vector factor by

[RISCV] Implement getVScaleForTuning and thus prefer scalable vectorization when enabled

LoopVectorizer uses getVScaleForTuning for deciding how to discount the cost of a potential vector factor by the amount of work performed. Without the callback implemented, the vectorizer was defaulting to an estimated vscale of 1. This results in fixed vectorization looking falsely profitable (since it used the command line VLEN).

The test change is pretty limited since a) we don't have much coverage of the vectorizer with scalable vectors at all, and b) what little coverage we have mostly uses i64 element types. There's a separate issue with <vscale x 1 x i64> which prevents us from getting to this stage of costing, and thus only the one test explicitly written to avoid that is visible in the diff. However, this is actually a very wide impact change as it changes the practical vectorization result when both fixed and scalable is enabled to scalable.

As an aside, I think the vectorizer is at little too strongly biased towards scalable when both are legal, but we can explore that separately. For now, let's just get the cost model working the way it was intended.

Differential Revision: https://reviews.llvm.org/D128547

show more ...

[RISCV] Implement RISCVTTIImpl::getMaxVScale correctly

The comments in the existing code appear to pre-exist the standardization of the +v extension. In particular, the specification *does* provide

[RISCV] Implement RISCVTTIImpl::getMaxVScale correctly

The comments in the existing code appear to pre-exist the standardization of the +v extension. In particular, the specification *does* provide a bound on the maximum value VLEN can take. From what I can tell, the LMUL comment was simply a misunderstanding of what this API returns.

This API returns the maximum value that vscale can take at runtime. This is used in the vectorizer to bound the largest scalable VF (e.g. LMUL in RISCV terms) which can be used without violating memory dependence.

Differential Revision: https://reviews.llvm.org/D128538

show more ...

[RISCV] Replace two calls to getMinRVVVectorSizeInBits with getRealMinVLen [nfc]

This doesn't change behavior, it just makes it slightly more obvious what's
going on. Note that getRealMinVLen is al

[RISCV] Replace two calls to getMinRVVVectorSizeInBits with getRealMinVLen [nfc]

This doesn't change behavior, it just makes it slightly more obvious what's
going on. Note that getRealMinVLen is always >= getMinRVVVectorSizeInBits.

The first case is a bit tricky, as you have to know that
getMinRVVVectorSizeInBits returns 0 when not set, and thus is equivalent
to the else value clause. The new code structure makes it more obvious we
return 0 unless using RVV for fixed length vectors.

show more ...

[RISCV] Fix crash when costing scalable gather/scatter of pointer

This was a bug introduced in d764aa. A pointer type is not a primitive type, and thus we were ending up dividing by zero when comput

[RISCV] Fix crash when costing scalable gather/scatter of pointer

This was a bug introduced in d764aa. A pointer type is not a primitive type, and thus we were ending up dividing by zero when computing VLMax.

Differential Revision: https://reviews.llvm.org/D128219

show more ...

[RISCV] Rename VTy param of RISCVTTIImpl::getArithmeticReductionCost [NFC]

Having it be consistent with getMinMaxReductionCost for ease of copy paste outweights the minor clarity of calling it VTy i

[RISCV] Rename VTy param of RISCVTTIImpl::getArithmeticReductionCost [NFC]

Having it be consistent with getMinMaxReductionCost for ease of copy paste outweights the minor clarity of calling it VTy instead of Ty.

show more ...

[RISCV] Add cost model for scalable scatter and gather

The costing we use for fixed length vector gather and scatter is to simply count up the memory ops, and multiply by a fixed memory op cost. For

[RISCV] Add cost model for scalable scatter and gather

The costing we use for fixed length vector gather and scatter is to simply count up the memory ops, and multiply by a fixed memory op cost. For scalable vectors, we don't actually know how many lanes are active. Instead, we have to end up making a worst case assumption on how many lanes could be active. In the generic +V case, this results in very high costs, but we can do better when we know an upper bound on the VLEN.

There's some obvious ways to improve this - e.g. using information about VL and mask bits from the instruction to reduce the upper bound - but this seems like a reasonable starting point.

The resulting costs do bias us pretty strongly away from generating scatter/gather for generic +V. Without this, we'd be returning an invalid cost and thus definitely not vectorizing, so no major change in practical behavior expected.

Differential Revision: https://reviews.llvm.org/D127541

show more ...

[RISCV] Refine costs for i1 reductions

Our actual lowering for i1 reductions uses ctpop combined with possibly a vector negate and possibly a logic op afterwards. I believe ctpop to be low cost on a

[RISCV] Refine costs for i1 reductions

Our actual lowering for i1 reductions uses ctpop combined with possibly a vector negate and possibly a logic op afterwards. I believe ctpop to be low cost on all reasonable hardware.

The default costing implementation here was returning quite inconsistent costs. and/or were returning very high costs (because we seem to think moving into scalar registers is very expensive?) and others were returning lower but still too high (because of the assumed tree reduce strategy). While we should probably improve the generic costing strategy for i1 vectors, let's start by fixing the immediate problem.

Differential Revision: https://reviews.llvm.org/D127511

show more ...

[RISCV] Add cost model for reverse shuffle

The majority of the cost appears to be forming the indices vector.

Differential Revision: https://reviews.llvm.org/D127141

[RISCV] Reorganize getShuffleCost to make it more clear what's going on [nfc]

[LV] Improve register pressure estimate at high VFs

Previously, `getRegUsageForType` was implemented using
`getTypeLegalizationCost`. `getRegUsageForType` is used by the loop
vectorizer to estimate

[LV] Improve register pressure estimate at high VFs

Previously, `getRegUsageForType` was implemented using
`getTypeLegalizationCost`. `getRegUsageForType` is used by the loop
vectorizer to estimate the register pressure caused by using a vector
type. However, `getTypeLegalizationCost` currently only appears to
understand splitting and not scalarization, so significantly
underestimates the register requirements.

Instead, use `getNumRegisters`, which understands when scalarization
can occur (via computeRegisterProperties).

This was discovered while investigating D118979 (Set maximum VF with
shouldMaximizeVectorBandwidth), where under fixed-length 512-bit SVE the
loop vectorizer previously ends up costing an v128i1 as 2 v64i*
registers where it actually occupies 128 i32 registers.

I'm sending this patch early for comment, I'm still doing some sanity checking
with LNT. I note that getRegisterClassForType appears to return VectorRC even
though the type in question (large vNi1 types) end up occupying scalar
registers. That might be worth fixing too.

Differential Revision: https://reviews.llvm.org/D125918

show more ...

[NFC][RISCV] Enable TuneNoDefaultUnroll feature to control targets which use default unroll preference

In RISCVTargetTransformInfo, enumerating the processor family is not a good way to predict.
Bec

[NFC][RISCV] Enable TuneNoDefaultUnroll feature to control targets which use default unroll preference

In RISCVTargetTransformInfo, enumerating the processor family is not a good way to predict.
Because it needs to enumerate many subtarget family and is hard to update if add new subtarget.
Instead, create a feature to distinguish whether targets want to use default unroll preference or not.

Keep TuneSiFive7 because it's flag to indicate subtarget family, which may used in other place.

Differential Revision: https://reviews.llvm.org/D125741

show more ...

[RISCV] Add cost model for SK_Broadcast

Add cost model for broadcast shuffle in RISCVTTIImpl::getShuffleCost
with scalable vector. The cost model might not the best.

For scalable vector, BasicTTIIm

[RISCV] Add cost model for SK_Broadcast

Add cost model for broadcast shuffle in RISCVTTIImpl::getShuffleCost
with scalable vector. The cost model might not the best.

For scalable vector, BasicTTIImpl::getShuffleCost return invalid cost,
so this patch relies on the existing cost model in BasicTTIImpl.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D124101

show more ...

Recommit "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 6a9bbd9f20dcd700e28738788bb63a160c6c088c.

Code review: https://reviews.llvm.or

Recommit "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 6a9bbd9f20dcd700e28738788bb63a160c6c088c.

Code review: https://reviews.llvm.org/D124202

show more ...

[RISCV] Remove riscv-v-fixed-length-vector-elen-max command line option.

This was added before Zve extensions were defined. I think users
should use Zve32x or Zve32f now. Though we will lose support

[RISCV] Remove riscv-v-fixed-length-vector-elen-max command line option.

This was added before Zve extensions were defined. I think users
should use Zve32x or Zve32f now. Though we will lose support for limiting
ELEN to 16 or 8, but I hope no one was using that.

Reviewed By: frasercrmck

Differential Revision: https://reviews.llvm.org/D123418

show more ...

[RISCV] Add basic code modeling for llvm.experimental.stepvector intrinsic

Scalable vectors llvm.experimental.stepvector intrinsic
will crash due to an invalid cost when run the code through the loo

[RISCV] Add basic code modeling for llvm.experimental.stepvector intrinsic

Scalable vectors llvm.experimental.stepvector intrinsic
will crash due to an invalid cost when run the code through the loopunroll.

Reviewed By: kito-cheng

Differential Revision: https://reviews.llvm.org/D122782

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 3, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit e6ead19b774718113007ecb1a4

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 3, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit e6ead19b774718113007ecb1a4449d7af0cbcfeb.

show more ...

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash."

This reverts commit 27bd8f94928201f87f6b659fc2228efd539e8245.

Causes crashes, see comme

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash."

This reverts commit 27bd8f94928201f87f6b659fc2228efd539e8245.

Causes crashes, see comments in D121973

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit f7d7d2a08d16356c57f6d2d36b

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit f7d7d2a08d16356c57f6d2d36bc2fc0589a55df9.

show more ...

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads.""

This reverts commit 79613185d305013de743cdbd6690e4d77c8af27e.

Causes crashes, see comments in https://reviews.llvm.org/D1

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads.""

This reverts commit 79613185d305013de743cdbd6690e4d77c8af27e.

Causes crashes, see comments in https://reviews.llvm.org/D121973.

show more ...

[RISCV] Add basic cost model for vector casting

To perform the cost model of vector casting, the patch consider most vector
casts as their scalar form and consider those vector form of free scalr ca

[RISCV] Add basic cost model for vector casting

To perform the cost model of vector casting, the patch consider most vector
casts as their scalar form and consider those vector form of free scalr castings
as 1.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D121771

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads."

Original review: https://reviews.llvm.org/D121354

The original commit 9136145eb019e1d18c966d4d06a3df349b88cc14 broke the build on

Recommit "[SLP] Fix lookahead operand reordering for splat loads."

Original review: https://reviews.llvm.org/D121354

The original commit 9136145eb019e1d18c966d4d06a3df349b88cc14 broke the build on several targets.

Differential Revision: https://reviews.llvm.org/D121973

show more ...

[RISCV] Add basic code modeling for fixed length vector reduction.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D121447