[VPlan] Update unit test after 569d84fe99e63.

[VPlan] Use region for each loop in native path.

This patch updates the VPlan native path to use VPRegionBlocks for all
loops in a loop nest. Up to now, only the outermost loop used a region.

This

[VPlan] Use region for each loop in native path.

This patch updates the VPlan native path to use VPRegionBlocks for all
loops in a loop nest. Up to now, only the outermost loop used a region.

This is a step towards unifying both paths and keep things consistent
between them. It also prepares various code-gen parts for modeling the
pre-header in the inner loop vectorizer (D121624).

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123005

show more ...

[VPlan] Remove unused native utilities incompatible with nested regions.

The implementations of VPlanDominatorTree, VPlanLoopInfo and VPlanPredicator
are all incompatible with modeling loops in VPla

[VPlan] Remove unused native utilities incompatible with nested regions.

The implementations of VPlanDominatorTree, VPlanLoopInfo and VPlanPredicator
are all incompatible with modeling loops in VPlans as region without
explicit back-edges.

Those pieces are not actively used and only exercised by a few gtest
unit tests. They are at the moment blocking progress towards unifying
the native and inner-loop vectorizer paths in D121624 and D123005.

I think we should not block forward progress on unused pieces of code,
so this patch removes the utilities for now. The plan is to re-introduce
them as needed in a way that is compatible with the unified VPlan scheme
used in both the inner loop vectorizer and the native path.

Reviewed By: sguggill

Differential Revision: https://reviews.llvm.org/D123017

show more ...

[VPlan] Turn external defs in Value -> VPValue mapping.

This addresses an existing TODO by keeping a mapping of external IR
Value * definitions wrapped in VPValues for use in a VPlan.

Reviewed By:

[VPlan] Turn external defs in Value -> VPValue mapping.

This addresses an existing TODO by keeping a mapping of external IR
Value * definitions wrapped in VPValues for use in a VPlan.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D123700

show more ...

[VPlan] Add value numbering in unit test after 65c4d6191fc4.

[LV] Record memory widening decisions (NFCI)

Record widening decisions for memory operations within the planned recipes and
use the recorded decisions in code-gen rather than querying the cost model

[LV] Record memory widening decisions (NFCI)

Record widening decisions for memory operations within the planned recipes and
use the recorded decisions in code-gen rather than querying the cost model.

Differential Revision: https://reviews.llvm.org/D110479

show more ...

[VPlan] Add mayReadOrWriteMemory & friends.

This patch adds initial implementation of mayReadOrWriteMemory,
mayReadFromMemory and mayWriteToMemory to VPRecipeBase.

Used by D100258.

[VPlan] Add getVPSingleValue helper.

As suggested in D99294, this adds a getVPSingleValue helper to use for
recipes that are guaranteed to define a single value. This replaces uses
of getVPValue() w

[VPlan] Add getVPSingleValue helper.

As suggested in D99294, this adds a getVPSingleValue helper to use for
recipes that are guaranteed to define a single value. This replaces uses
of getVPValue() which used to default to I = 0.

show more ...

[VPlan] Make blocksOnly work properly with ranges over const pointers.

When iterating over const blocks, the base type in the lambdas needs
to use const VPBlockBase *, otherwise it cannot be used wi

[VPlan] Make blocksOnly work properly with ranges over const pointers.

When iterating over const blocks, the base type in the lambdas needs
to use const VPBlockBase *, otherwise it cannot be used with input
iterators over const VPBlockBase.

Also adjust the type of the input iterator range to const &, as it
does not take ownership of the input range.

show more ...

[VPlan] Add VPBlockUtils::blocksOnly helper.

This patch adds a blocksOnly helpers which take an iterator range
over VPBlockBase * or const VPBlockBase * and returns an interator
range that only incl

[VPlan] Add VPBlockUtils::blocksOnly helper.

This patch adds a blocksOnly helpers which take an iterator range
over VPBlockBase * or const VPBlockBase * and returns an interator
range that only include BlockTy blocks. The accesses are casted to
BlockTy.

Reviewed By: a.elovikov

Differential Revision: https://reviews.llvm.org/D101093

show more ...

[VPlan] Add GraphTraits impl to traverse through VPRegionBlock.

This patch adds a new iterator to traverse through VPRegionBlocks and a
GraphTraits specialization using the iterator to traverse thro

[VPlan] Add GraphTraits impl to traverse through VPRegionBlock.

This patch adds a new iterator to traverse through VPRegionBlocks and a
GraphTraits specialization using the iterator to traverse through
VPRegionBlocks.

Because there is already a GraphTraits specialization for VPBlockBase *
and co, a new VPBlockRecursiveTraversalWrapper helper is introduced.
This allows us to provide a new GraphTraits specialization for that
type. Users can use the new recursive traversal by using this wrapper.

The graph trait visits both the entry block of a region, as well as all
its successors. Exit blocks of a region implicitly have their parent
region's successors. This ensures all blocks in a region are visited
before any blocks in a successor region when doing a reverse post-order
traversal of the graph.

Reviewed By: a.elovikov

Differential Revision: https://reviews.llvm.org/D100175

show more ...

[VPlan] Add VPRecipeBase::mayHaveSideEffects.

Add an initial version of a helper to determine whether a recipe may
have side-effects.

Reviewed By: a.elovikov

Differential Revision: https://reviews

[VPlan] Add VPRecipeBase::mayHaveSideEffects.

Add an initial version of a helper to determine whether a recipe may
have side-effects.

Reviewed By: a.elovikov

Differential Revision: https://reviews.llvm.org/D100259

show more ...

[NFC][VPlan] Guard print routines with "#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)"

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D98897

[VPlan] Add plain text (not DOT's digraph) dumps

I foresee two uses for this:
1) It's easier to use those in debugger.
2) Once we start implementing more VPlan-to-VPlan transformations (especially

[VPlan] Add plain text (not DOT's digraph) dumps

I foresee two uses for this:
1) It's easier to use those in debugger.
2) Once we start implementing more VPlan-to-VPlan transformations (especially
inner loop massaging stuff), using the vectorized LLVM IR as CHECK targets in
LIT test would become too obscure. I can imagine that we'd want to CHECK
against VPlan dumps after multiple transformations instead. That would be
easier with plain text dumps than with DOT format.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D96628

show more ...

Revert "[VPlan] Add plain text (not DOT's digraph) dumps"

This reverts commit 6b053c9867a3ede32e51cef3ed972d5ce5b38bc0.
The build is broken:

ld.lld: error: undefined symbol: llvm::VPlan::printDOT(l

Revert "[VPlan] Add plain text (not DOT's digraph) dumps"

This reverts commit 6b053c9867a3ede32e51cef3ed972d5ce5b38bc0.
The build is broken:

ld.lld: error: undefined symbol: llvm::VPlan::printDOT(llvm::raw_ostream&) const
>>> referenced by LoopVectorize.cpp
>>> LoopVectorize.cpp.o:(llvm::LoopVectorizationPlanner::printPlans(llvm::raw_ostream&)) in archive lib/libLLVMVectorize.a

show more ...

[VPlan] Add plain text (not DOT's digraph) dumps

I foresee two uses for this:
1) It's easier to use those in debugger.
2) Once we start implementing more VPlan-to-VPlan transformations (especially

[VPlan] Add plain text (not DOT's digraph) dumps

I foresee two uses for this:
1) It's easier to use those in debugger.
2) Once we start implementing more VPlan-to-VPlan transformations (especially
inner loop massaging stuff), using the vectorized LLVM IR as CHECK targets in
LIT test would become too obscure. I can imagine that we'd want to CHECK
against VPlan dumps after multiple transformations instead. That would be
easier with plain text dumps than with DOT format.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D96628

show more ...

[VPlan] Make VPRecipeBase inherit from VPUser directly (NFC).

The individual recipes have been updated to manage their operands using
VPUser a while back. Now that the transition is done, we can ins

[VPlan] Make VPRecipeBase inherit from VPUser directly (NFC).

The individual recipes have been updated to manage their operands using
VPUser a while back. Now that the transition is done, we can instead
make VPRecipeBase a VPUser and get rid of the toVPUser helper.

show more ...

[LoopVectorize] use IR fast-math-flags exclusively (not FP function attributes)

I am trying to untangle the fast-math-flags propagation logic
in the vectorizers (see a6f022127 for SLP).

The loop ve

[LoopVectorize] use IR fast-math-flags exclusively (not FP function attributes)

I am trying to untangle the fast-math-flags propagation logic
in the vectorizers (see a6f022127 for SLP).

The loop vectorizer has a mix of checking FP function attributes,
IR-level FMF, and just wrong assumptions.

I am trying to avoid regressions while fixing this, and I think
the IR-level logic is good enough for that, but it's hard to say
for sure. This would be the 1st step in the clean-up.

The existing test that I changed to include 'fast' actually shows
a miscompile: the function only had the equivalent of nnan, but we
created new instructions that had fast (all FMF set). This is
similar to the example in https://llvm.org/PR35538

Differential Revision: https://reviews.llvm.org/D95452

show more ...

[VPlan] Unify value/recipe printing after VPDef transition.

This patch unifies the way recipes and VPValues are printed after the
transition to VPDef.

VPSlotTracker has been updated to iterate over

[VPlan] Unify value/recipe printing after VPDef transition.

This patch unifies the way recipes and VPValues are printed after the
transition to VPDef.

VPSlotTracker has been updated to iterate over all recipes and all
their defined values to number those. There is no need to number
values in Value2VPValue.

It also updates a few places that only used slot numbers for
VPInstruction. All recipes now can produce numbered VPValues.

show more ...

[LV] Don't sink into replication regions

The new test case here contains a first order recurrences and an
instruction that is replicated. The first order recurrence forces an
instruction to be sunk

[LV] Don't sink into replication regions

The new test case here contains a first order recurrences and an
instruction that is replicated. The first order recurrence forces an
instruction to be sunk _into_, as opposed to after the replication
region. That causes several things to go wrong including registering
vector instructions multiple times and failing to create dominance
relations correctly.

Instead we should be sinking to after the replication region, which is
what this patch makes sure happens.

Differential Revision: https://reviews.llvm.org/D93629

show more ...

[VPlan] Make VPRecipeBase inherit from VPDef.

This patch makes VPRecipeBase a direct subclass of VPDef, moving the
SubclassID to VPDef.

Reviewed By: gilr

Differential Revision: https://reviews.llv

[VPlan] Make VPRecipeBase inherit from VPDef.

This patch makes VPRecipeBase a direct subclass of VPDef, moving the
SubclassID to VPDef.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90564

show more ...

[VPlan] Use VPDef for VPInterleaveRecipe.

This patch turns updates VPInterleaveRecipe to manage the values it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration i

[VPlan] Use VPDef for VPInterleaveRecipe.

This patch turns updates VPInterleaveRecipe to manage the values it defines
using VPDef. The VPValue is used during VPlan construction and
codegeneration instead of the plain IR reference where possible.

Reviewed By: gilr

Differential Revision: https://reviews.llvm.org/D90562

show more ...

[VPlan] Manage stored values of interleave groups using VPUser (NFC)

Interleave groups also depend on the values they store. Manage the
stored values as VPUser operands. This is currently a NFC, but

[VPlan] Manage stored values of interleave groups using VPUser (NFC)

Interleave groups also depend on the values they store. Manage the
stored values as VPUser operands. This is currently a NFC, but is
required to allow VPlan transforms and to manage generated vector values
exclusively in VPTransformState.

show more ...

[VPlan] Add VPReductionSC to VPUser::classof, unify VPValue IDs.

This is a follow-up to 00a66011366c7b037d6680e6015524a41b761c34 to make
isa<VPReductionRecipe> work and unifies the VPValue ID names,

[VPlan] Add VPReductionSC to VPUser::classof, unify VPValue IDs.

This is a follow-up to 00a66011366c7b037d6680e6015524a41b761c34 to make
isa<VPReductionRecipe> work and unifies the VPValue ID names, by making
sure they all consistently start with VPV*.

show more ...

[VPlan] Add VPDef class.

This patch introduces a new VPDef class, which can be used to
manage VPValues defined by recipes/VPInstructions.

The idea here is to mirror VPUser for values defined by a r

[VPlan] Add VPDef class.

This patch introduces a new VPDef class, which can be used to
manage VPValues defined by recipes/VPInstructions.

The idea here is to mirror VPUser for values defined by a recipe. A
VPDef can produce either zero (e.g. a store recipe), one (most recipes)
or multiple (VPInterleaveRecipe) result VPValues.

To traverse the def-use chain from a VPDef to its users, one has to
traverse the users of all values defined by a VPDef.

VPValues now contain a pointer to their corresponding VPDef, if one
exists. To traverse the def-use chain upwards from a VPValue, we first
need to check if the VPValue is defined by a VPDef. If it does not have
a VPDef, this means we have a VPValue that is not directly defined
iniside the plan and we are done.

If we have a VPDef, it is defined inside the region by a recipe, which
is a VPUser, and the upwards def-use chain traversal continues by
traversing all its operands.

Note that we need to add an additional field to to VPVAlue to link them
to their defs. The space increase is going to be offset by being able to
remove the SubclassID field in future patches.

Reviewed By: Ayal

Differential Revision: https://reviews.llvm.org/D90558

show more ...