Revert "[LV] Remove `LoopVectorizationCostModel::useEmulatedMaskMemRefHack()`"

This reverts commit 77a0da926c9ea86afa9baf28158d79c7678fc6b9 as we've
received multiple reports of this significantly i

Revert "[LV] Remove `LoopVectorizationCostModel::useEmulatedMaskMemRefHack()`"

This reverts commit 77a0da926c9ea86afa9baf28158d79c7678fc6b9 as we've
received multiple reports of this significantly impacting performance,
in ways that don't seem to just be target specific cost models going
wrong. I would offer some reproducers, but the test changes here seem to
be full of them!

Reverting for now and hopefully we can remove the "hack" more carefully
as we go.

show more ...

[LV] Remove `LoopVectorizationCostModel::useEmulatedMaskMemRefHack()`

D43208 extracted `useEmulatedMaskMemRefHack()` from legality into cost model.
What it essentially does is prevents scalarized ve

[LV] Remove `LoopVectorizationCostModel::useEmulatedMaskMemRefHack()`

D43208 extracted `useEmulatedMaskMemRefHack()` from legality into cost model.
What it essentially does is prevents scalarized vectorization of masked memory operations:
```
// TODO: Cost model for emulated masked load/store is completely
// broken. This hack guides the cost model to use an artificially
// high enough value to practically disable vectorization with such
// operations, except where previously deployed legality hack allowed
// using very low cost values. This is to avoid regressions coming simply
// from moving "masked load/store" check from legality to cost model.
// Masked Load/Gather emulation was previously never allowed.
// Limited number of Masked Store/Scatter emulation was allowed.
```

While i don't really understand about what specifically `is completely broken`
was talking about, i believe that at least on X86 with AVX2-or-later,
this is no longer true. (or at least, i would like to know what is still broken).
So i would like to follow suit after D111460, and like wise disable that hack for AVX2+.

But since this was added for X86 specifically, let's just instead completely remove this hack.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D114779

show more ...

[IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder

This patch updates IRBuilder to create insertelement/shufflevector using poison as a placeholder.

Reviewed By: n

[IR] Let IRBuilder's CreateVectorSplat/CreateShuffleVector use poison as placeholder

This patch updates IRBuilder to create insertelement/shufflevector using poison as a placeholder.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D93793

show more ...

Recommit "[VPlan] Use VPValue def for VPWidenSelectRecipe."

This reverts the revert commit c8d73d939fa4fda9c87b3979225d02d63062bd68.

It includes a fix for cases where we missed inserting VPValues
f

Recommit "[VPlan] Use VPValue def for VPWidenSelectRecipe."

This reverts the revert commit c8d73d939fa4fda9c87b3979225d02d63062bd68.

It includes a fix for cases where we missed inserting VPValues
for some selects, which should fix PR48142.

show more ...

[NPM][test] Fix some LoopVectorize tests under NPM

[LV] Tail folded inloop reductions.

This expands upon the inloop reductions added in e9761688e41cb9e976,
allowing them to be inserted into tail folded loops. Reductions are
generates with the form:

[LV] Tail folded inloop reductions.

This expands upon the inloop reductions added in e9761688e41cb9e976,
allowing them to be inserted into tail folded loops. Reductions are
generates with the form:

x = select(mask, vecop, zero)
v = vecreduce.add(x)
c = add chain, v

Where zero here is chosen as the identity value for add reductions. The
backend is then expected to fold the select and the vecreduce into a
single predicated instruction.

Most of the code is fairly straight forward, except for the creation of
blockmasks which need to ensure they are created in dominance order. The
order they are added is altered to be after any phis, keeping the
requirements for the underlying IR.

Differential Revision: https://reviews.llvm.org/D84451

show more ...

[PGO][PGSO][LV] Fix loop not vectorized issue under profile guided size opts.

D81345 appears to accidentally disables vectorization when explicitly
enabled. As PGSO isn't currently accessible from L

[PGO][PGSO][LV] Fix loop not vectorized issue under profile guided size opts.

D81345 appears to accidentally disables vectorization when explicitly
enabled. As PGSO isn't currently accessible from LoopAccessInfo, revert back to
the vectorization with versioning-for-unit-stride for PGSO.

Differential Revision: https://reviews.llvm.org/D85784

show more ...

[PGO][PGSO] Add profile guided size optimization to loop vectorization legality.

Revert "[PGO][PGSO] Add profile guided size optimization to loop vectorization legality."

This reverts commit 30c382a7c6607a7d898730f8d288768110cdf1d2.

See https://crbug.com/1106813.

[PGO][PGSO] Add profile guided size optimization to loop vectorization legality.

Differential Revision: https://reviews.llvm.org/D83329

[LV] Fixing versioning-for-unit-stide of loops with small trip count

This patch fixes D81345 and PR46652.

If a loop with a small trip count is compiled w/o -Os/-Oz, Loop Access Analysis
still gener

[LV] Fixing versioning-for-unit-stide of loops with small trip count

This patch fixes D81345 and PR46652.

If a loop with a small trip count is compiled w/o -Os/-Oz, Loop Access Analysis
still generates runtime checks for unit strides that will version the loop.

In such cases, the loop vectorizer should either re-run the analysis or bail-out
from vectorizing the loop, as done prior to D81345. The latter is applied for
now as the former requires refactoring.

Differential Revision: https://reviews.llvm.org/D83470

show more ...

[LV] Vectorize without versioning-for-unit-stride under -Os/-Oz

If a loop is in a function marked OptSize, Loop Access Analysis should refrain
from generating runtime checks for unit strides that wi

[LV] Vectorize without versioning-for-unit-stride under -Os/-Oz

If a loop is in a function marked OptSize, Loop Access Analysis should refrain
from generating runtime checks for unit strides that will version the loop.

If a loop is in a function marked OptSize and its vectorization is enabled, it
should be vectorized w/o any versioning.

Fixes PR46228.

Differential Revision: https://reviews.llvm.org/D81345

show more ...

[LV] Mark first-order recurrences as allowed exits

First-order recurrences require special treatment when they are live-out;
such treatment is provided by fixFirstOrderRecurrence(), so they should b

[LV] Mark first-order recurrences as allowed exits

First-order recurrences require special treatment when they are live-out;
such treatment is provided by fixFirstOrderRecurrence(), so they should be
included in AllowedExit set.

(Should probably have been included originally in D16197.)

Fixes PR45526: AllowedExit set is used by prepareToFoldTailByMasking() to
check whether the treatment for live-outs also holds when folding the tail,
which is not (yet) the case for first-order recurrences.

Differential Revision: https://reviews.llvm.org/D78210

show more ...

[LV] Emitting SCEV checks with OptForSize

When optimising for size and SCEV runtime checks need to be emitted to check
overflow behaviour, the loop vectorizer can run in this assert:

LoopVectoriz

[LV] Emitting SCEV checks with OptForSize

When optimising for size and SCEV runtime checks need to be emitted to check
overflow behaviour, the loop vectorizer can run in this assert:

LoopVectorize.cpp:2699: void llvm::InnerLoopVectorizer::emitSCEVChecks(
llvm::Loop *, llvm::BasicBlock *): Assertion `!BB->getParent()->hasOptSize()
&& "Cannot SCEV check stride or overflow when opt

We should not generate predicates while optimising for size because
code will be generated for predicates such as these SCEV overflow runtime
checks.

This should fix PR43371.

Differential Revision: https://reviews.llvm.org/D68082

llvm-svn: 374166

show more ...

Revert "Temporarily Revert "Add basic loop fusion pass.""

The reversion apparently deleted the test/Transforms directory.

Will be re-reverting again.

llvm-svn: 358552

[PGO] Profile guided code size optimization.

Summary:
Enable some of the existing size optimizations for cold code under PGO.

A ~5% code size saving in big internal app under PGO.

The way it gets

[PGO] Profile guided code size optimization.

Summary:
Enable some of the existing size optimizations for cold code under PGO.

A ~5% code size saving in big internal app under PGO.

The way it gets BFI/PSI is discussed in the RFC thread

http://lists.llvm.org/pipermail/llvm-dev/2019-March/130894.html

Note it doesn't currently touch loop passes.

Reviewers: davidxl, eraman

Reviewed By: eraman

Subscribers: mgorny, javed.absar, smeenai, mehdi_amini, eraman, zzheng, steven_wu, dexonsmith, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D59514

llvm-svn: 358422

show more ...

fix minsize detection: minsize attribute implies optimizing for size

llvm-svn: 244617

remove unnecessary settings/attributes from test case

llvm-svn: 244612

[LoopVectorizer] Fix bailing-out condition for OptForSize case.

With option OptForSize enabled, the Loop Vectorizer is not supposed to
create tail loop. The condition checking that was invalid and w

[LoopVectorizer] Fix bailing-out condition for OptForSize case.

With option OptForSize enabled, the Loop Vectorizer is not supposed to
create tail loop. The condition checking that was invalid and was not
matching to the comment above.

Patch by Marianne Mailhot-Sarrasin.

llvm-svn: 240556

show more ...