[LV] Update handling of scalable pointer inductions after b73d2c8.

The dependent code has been changed quite a lot since 151c144 which
b73d2c8 effectively reverts. Now we run into a case where lower

[LV] Update handling of scalable pointer inductions after b73d2c8.

The dependent code has been changed quite a lot since 151c144 which
b73d2c8 effectively reverts. Now we run into a case where lowering
didn't expect/support the behavior pre 151c144 any longer.

Update the code dealing with scalable pointer inductions to also check
for uniformity in combination with isScalarAfterVectorization. This
should ensure scalable pointer inductions are handled properly during
epilogue vectorization.

Fixes #57912.

(cherry picked from commit 2c692d891ed639779b1c4b504ca63037bbacc0e8)

show more ...

[LV] Keep track of cost-based ScalarAfterVec in VPWidenPointerInd.

Epilogue vectorization uses isScalarAfterVectorization to check if
widened versions for inductions need to be generated and bails o

[LV] Keep track of cost-based ScalarAfterVec in VPWidenPointerInd.

Epilogue vectorization uses isScalarAfterVectorization to check if
widened versions for inductions need to be generated and bails out in
those cases.

At the moment, there are scenarios where isScalarAfterVectorization
returns true but VPWidenPointerInduction::onlyScalarsGenerated would
return false, causing widening.

This can lead to widened phis with incorrect start values being created
in the epilogue vector body.

This patch addresses the issue by storing the cost-model decision in
VPWidenPointerInductionRecipe and restoring the behavior before 151c144.
This effectively reverts 151c144, but the long-term fix is to properly
support widened inductions during epilogue vectorization

Fixes #57712.

show more ...

[LV] Move VPPredInstPHIRecipe::execute to VPlanRecipes.cpp (NFC)

[LV] Move VPBranchOnMaskRecipe::execute to VPlanRecipes.cpp (NFC).

[LV] Move VPBlendRecipe::execute to VPlanRecipes.cpp (NFC).

[LoopVectorize] Add option to use active lane mask for loop control flow

Currently, for vectorised loops that use the get.active.lane.mask
intrinsic we only use the mask for predicated vector operat

[LoopVectorize] Add option to use active lane mask for loop control flow

Currently, for vectorised loops that use the get.active.lane.mask
intrinsic we only use the mask for predicated vector operations,
such as masked loads and stores, etc. The loop itself is still
controlled by comparing the canonical induction variable with the
trip count. However, for some targets this is inefficient when it's
cheap to use the mask itself to control the loop.

This patch adds support for using the active lane mask for control
flow by:

1. Generating the active lane mask for the next iteration of the
vector loop, rather than the current one. If there are still any
remaining iterations then at least the first bit of the mask will
be set.
2. Extract the first bit of this mask and use this bit for the
conditional branch.

I did this by creating a new VPActiveLaneMaskPHIRecipe that sets
up the initial PHI values in the vector loop pre-header. I've also
made use of the new BranchOnCond VPInstruction for the final
instruction in the loop region.

Differential Revision: https://reviews.llvm.org/D125301

show more ...

[LoopVectorize][NFC] Add optional Name parameter to VPInstruction

This patch is a simple piece of refactoring that now permits users
to create VPInstructions and specify the name of the value being

[LoopVectorize][NFC] Add optional Name parameter to VPInstruction

This patch is a simple piece of refactoring that now permits users
to create VPInstructions and specify the name of the value being
generated. This is useful for creating more readable/meaningful
names in IR.

Differential Revision: https://reviews.llvm.org/D128982

show more ...

[LV] Move VPWidenGEPRecipe::execute to VPlanRecipes.cpp (NFC).

[LV] Move VPWidenRecipe::execute to VPlanRecipes.cpp (NFC).

[VPlan] Move VPWidenSelectRecipe::execute to VPlanRecipes.cpp (NFC).

Depends on D127968.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D127970

[VPlan] Move recipe implementations to separate file (NFC).

This patch moves the code for recipe implementations to a separate file.

The benefits are:
* Keep VPlan.cpp smaller => faster compile-ti

[VPlan] Move recipe implementations to separate file (NFC).

This patch moves the code for recipe implementations to a separate file.

The benefits are:
* Keep VPlan.cpp smaller => faster compile-time during parallel builds.
* Keep code for logical units together

As a follow-up I am also planning on moving all ::execute
implemetnations from LoopVectorize.cpp over to the new file, which
should help to reduce the size of the file a bit.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D127965

show more ...