[VPlan] Make sure optimizeInductions removes wide ind from scalar plan.

In some cases, there may be widened users of inductions even though the
plan includes the scalar VF. In those cases, make sure

[VPlan] Make sure optimizeInductions removes wide ind from scalar plan.

In some cases, there may be widened users of inductions even though the
plan includes the scalar VF. In those cases, make sure we still replace
the VPWidenIntOrFpInductionRecipe with scalar steps, as otherwise we may
try to execute a VPWidenIntOrFpInductionRecipe with a scalar VF.

Alternatively the patch could also split the range if needed.

This fixes a crash exposed by D123720.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D128755

show more ...

[VPlan] Remove dead recipes across whole plan.

This extends removeDeadRecipe to remove recipes across the whole plan.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D127580

[VPlan] Remove dead OrigLoop argument from removeDeadRecipes (NFC).

The use of the argument has been remove a while ago. Remove the dead
argument.

[VPlan] Replace remaining use of needsScalarIV.

All information is already available in VPlan. Note that there are some
test changes, because we now can correctly look through instructions
like trun

[VPlan] Replace remaining use of needsScalarIV.

All information is already available in VPlan. Note that there are some
test changes, because we now can correctly look through instructions
like truncates to analyze the actual users.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123541

show more ...

Recommit "[VPlan] Remove uneeded needsVectorIV check."

This reverts commit 266ea446ab747671eb6c736569c3c9c5f3c53d11.

The reasons for the revert have been addressed by cleaning up condition
handling

Recommit "[VPlan] Remove uneeded needsVectorIV check."

This reverts commit 266ea446ab747671eb6c736569c3c9c5f3c53d11.

The reasons for the revert have been addressed by cleaning up condition
handling in VPlan and properly marking VPBranchOnMaskRecipe as using
scalars.

The test case for the revert from D123720 has been added in 3d663308a5d.

show more ...

[VPlan] Replace CondBit with BranchOnCond VPInstruction.

This patch removes CondBit and Predicate from VPBasicBlock. To do so,
the patch introduces a new branch-on-cond VPInstruction opcode to model

[VPlan] Replace CondBit with BranchOnCond VPInstruction.

This patch removes CondBit and Predicate from VPBasicBlock. To do so,
the patch introduces a new branch-on-cond VPInstruction opcode to model
a branch on a condition explicitly.

This addresses a long-standing TODO/FIXME that blocks shouldn't be users
of VPValues. Those extra users can cause issues for VPValue-based
analyses that don't expect blocks. Addressing this fixme should allow us
to re-introduce 266ea446ab7476.

The generic branch opcode can also be used in follow-up patches.

Depends on D123005.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D126618

show more ...

[VPlan] Use region for each loop in native path.

This patch updates the VPlan native path to use VPRegionBlocks for all
loops in a loop nest. Up to now, only the outermost loop used a region.

This

[VPlan] Use region for each loop in native path.

This patch updates the VPlan native path to use VPRegionBlocks for all
loops in a loop nest. Up to now, only the outermost loop used a region.

This is a step towards unifying both paths and keep things consistent
between them. It also prepares various code-gen parts for modeling the
pre-header in the inner loop vectorizer (D121624).

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123005

show more ...

[VPlan] Model first exit values using VPLiveOut.

This patch introduces a new VPLiveOut subclass of VPUser to model
exit values explicitly. The initial version handles exit values that
are neither

[VPlan] Model first exit values using VPLiveOut.

This patch introduces a new VPLiveOut subclass of VPUser to model
exit values explicitly. The initial version handles exit values that
are neither part of induction or reduction chains nor first order
recurrence phis.

Fixes #51366, #54867, #55167, #55459

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123537

show more ...

[VPlan] Move usesScalars/onlyFirstLaneUsed to VPUser.

Those helpers model properties of a user and they should also be
available to non-recipe users. This will be used in D123537 for a new
exit valu

[VPlan] Move usesScalars/onlyFirstLaneUsed to VPUser.

Those helpers model properties of a user and they should also be
available to non-recipe users. This will be used in D123537 for a new
exit value user.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D124936

show more ...

Revert "Recommit "[VPlan] Remove uneeded needsVectorIV check.""

This reverts commit 8b48223447311af8b3022697dd58858e1ce6975f.

This triggers an assertion on a test case mentioned in D123720.
Revert

Revert "Recommit "[VPlan] Remove uneeded needsVectorIV check.""

This reverts commit 8b48223447311af8b3022697dd58858e1ce6975f.

This triggers an assertion on a test case mentioned in D123720.
Revert while I investigate.

show more ...

Recommit "[VPlan] Remove uneeded needsVectorIV check."

This reverts commit f4e1eaa3755a13f85696be3b74b387122b74a558.

The patch was originally reverted because it uncovered an issue that has
now bee

Recommit "[VPlan] Remove uneeded needsVectorIV check."

This reverts commit f4e1eaa3755a13f85696be3b74b387122b74a558.

The patch was originally reverted because it uncovered an issue that has
now been fixed in 0ef8ca6d88aa7e4abc.

show more ...

Revert "[VPlan] Remove uneeded needsVectorIV check."

This reverts commit 43842b887e0a7b918bb2d6c9f672025b2c621f8a while I
investigate a buildbot failure.

It also reverts the follow-up commit
2883de

Revert "[VPlan] Remove uneeded needsVectorIV check."

This reverts commit 43842b887e0a7b918bb2d6c9f672025b2c621f8a while I
investigate a buildbot failure.

It also reverts the follow-up commit
2883de05145fc5b4afb99b91f69ebb835af36af5.

show more ...

[VPlan] Fix comment formatting from 43842b887e.

[VPlan] Remove uneeded needsVectorIV check.

Remove one of the last remaining uses of ::needsVectorIV, preparing for
its removal. Now that usesScalars is available and based on the
information explic

[VPlan] Remove uneeded needsVectorIV check.

Remove one of the last remaining uses of ::needsVectorIV, preparing for
its removal. Now that usesScalars is available and based on the
information explicit in VPlan, there is no need to use the pre-computed
needsVectorIV.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123720

show more ...

[VPlan] Expand induction step in VPlan pre-header.

This patch moves SCEV expansion of steps used by
VPWidenIntOrFpInductionRecipes to the pre-header using
VPExpandSCEVRecipe. This ensures that those

[VPlan] Expand induction step in VPlan pre-header.

This patch moves SCEV expansion of steps used by
VPWidenIntOrFpInductionRecipes to the pre-header using
VPExpandSCEVRecipe. This ensures that those steps are expanded while the
CFG is in a valid state. Previously, SCEV expansion may happen during
vector body code-generation, during which the CFG may be invalid,
causing issues with SCEV expansion.

Depends on D122095.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D122096

show more ...

[VPlan] Turn external defs in Value -> VPValue mapping.

This addresses an existing TODO by keeping a mapping of external IR
Value * definitions wrapped in VPValues for use in a VPlan.

Reviewed By:

[VPlan] Turn external defs in Value -> VPValue mapping.

This addresses an existing TODO by keeping a mapping of external IR
Value * definitions wrapped in VPValues for use in a VPlan.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123700

show more ...

[VPlan] Place VPExpandSCEVRecipe in pre-header.

After D121624 models the pre-header in VPlan, VPExpandSCEVRecipes can be
placed there. This ensures SCEV expansion happens before modifying the
CFG du

[VPlan] Place VPExpandSCEVRecipe in pre-header.

After D121624 models the pre-header in VPlan, VPExpandSCEVRecipes can be
placed there. This ensures SCEV expansion happens before modifying the
CFG during VPlan execution, when CFG is incomplete.

Depends on D121624.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D122095

show more ...

[LV] Use getVectorLoopRegion to retrieve header. (NFC)

Update all places that currently assume the entry block to the plan is
also the vector loop header to use getVectorLoopRegion instead.

getVect

[LV] Use getVectorLoopRegion to retrieve header. (NFC)

Update all places that currently assume the entry block to the plan is
also the vector loop header to use getVectorLoopRegion instead.

getVectorLoopRegion will keep doing the right thing when the pre-header
is modeled explicitly (and becomes the new entry block in the plan).

show more ...

Cleanup includes: Transforms/Instrumentation & Transforms/Vectorize

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.or

Cleanup includes: Transforms/Instrumentation & Transforms/Vectorize

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D122181

show more ...

[LV] Check for users of truncated IVs, add more detailed comment.

Add missing outside user check for truncated IVs. Also hoist the code in
the helper with additional explanations.

Fixes #54370.

[VPlan] Ensure each iv user is only visited once in transform.

If a recipe has multiple uses of an IV, we crash. It causes a crash when
building llvm-test-suite.

Exposed by 95f76bff1c40bc1c2f.

[LV] Create & use VPScalarIVSteps for all scalar users.

This patch is a follow-up to D115953. It updates optimizeInductions
to also introduce new VPScalarIVStepsRecipes if an IV has both vector
and

[LV] Create & use VPScalarIVSteps for all scalar users.

This patch is a follow-up to D115953. It updates optimizeInductions
to also introduce new VPScalarIVStepsRecipes if an IV has both vector
and scalar uses.

It updates all uses that only need scalar values to use the newly
created recipe for the scalar steps.

This completes untangling of VPWidenIntOrFpInductionRecipe
code-generation. Now the recipe *only* creates the widened vector
values, as it says on the tin.

The code to genereate IR has been moved directly to
VPWidenIntOrFpInductionRecipe::execute.

Note that the recipe has been updated to hold a reference to
ScalarEvolution, which is needed to expand the step, until we can place
the corresponding SCEV expansion in the pre-header.

Depends on D120827.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D120828

show more ...

[LV] Do not consider instrs dead if used by phi that's not in plan.

Single value phis won't be modeled in VPlan. If the phi only gets used
outside the loop, the current code misses the fact that the

[LV] Do not consider instrs dead if used by phi that's not in plan.

Single value phis won't be modeled in VPlan. If the phi only gets used
outside the loop, the current code misses the fact that the incoming
value is not dead. Update the code to also look through such phis to
check for outside users.

Fixes #54266

show more ...

[VPlan] Remove reliance on underlying instr for ScalarIVSteps (NFCI).

Instead of relying on underlying instructions, this patch updates
VPScalarIVStepsRecipe to only store the required type informat

[VPlan] Remove reliance on underlying instr for ScalarIVSteps (NFCI).

Instead of relying on underlying instructions, this patch updates
VPScalarIVStepsRecipe to only store the required type information.

This removes access to unrelated information, as well as avoiding issues
with the same underlying instruction being shared by multiple recipes.

This change should only change the debug output and not cause any
codegen changes, hence NFCI.

show more ...

Recommit "[VPlan] Introduce recipe to build scalar steps."

This reverts the revert commit ff93260bf6bddfbad1fa65c4d5184988885b900f.

The underlying issue causing the PPC bot failures has been fixed

Recommit "[VPlan] Introduce recipe to build scalar steps."

This reverts the revert commit ff93260bf6bddfbad1fa65c4d5184988885b900f.

The underlying issue causing the PPC bot failures has been fixed in
cbaac1473403 and a corresponding test case has been added in
ad2cad1c521c.

Original message:

This patch adds a new VPScalarIVStepsRecipe to handle building scalar
steps.

In the first patch, it only handles the case where there is no vector
induction variable needed.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D115953

show more ...