[LV] Refresh a couple of autogen tests for naming change

These appear to just be changes in temporary identifiers; bit suprising we have so many.

Revert "[NFCI] Regenerate SROA/LoopVectorize test checks"

This reverts commit 14e3450fb57305aa9ff3e9e60687b458e43835c9.

[NFCI] Regenerate LoopVectorize test checks

[SCEVExpand] Only create required predicate checks.

Currently generateOverflowCheck always creates code for Step being
negative and positive, followed by a select at the end depending on
Step's sign

[SCEVExpand] Only create required predicate checks.

Currently generateOverflowCheck always creates code for Step being
negative and positive, followed by a select at the end depending on
Step's sign.

This patch updates the code to only create either the checks for step
being positive or negative, if the sign is known.

Follow-up to D116696.

Reviewed By: reames

Differential Revision: https://reviews.llvm.org/D116747

show more ...

[SCEVExpand] Do not create redundant 'or false' for pred expansion.

This patch updates SCEVExpander::expandUnionPredicate to not create
redundant 'or false, x' instructions. While those are triviall

[SCEVExpand] Do not create redundant 'or false' for pred expansion.

This patch updates SCEVExpander::expandUnionPredicate to not create
redundant 'or false, x' instructions. While those are trivially
foldable, they can be easily avoided and hinder code that checks the
size/cost of the generated checks before further folds.

I am planning on look into a few other similar improvements to code
generated by SCEVExpander.

I remember a while ago @lebedev.ri working on doing some trivial folds
like that in IRBuilder itself, but there where concerns that such
changes may subtly break existing code.

Reviewed By: reames, lebedev.ri

Differential Revision: https://reviews.llvm.org/D116696

show more ...

Revert "[LoopVectorize] Extract the last lane from a uniform store"

This reverts commit 0d748b4d32cbddf58a1ff83f3ff178ec1ad49edc.
This is causing some failures when building Spec2017 with scalable
v

Revert "[LoopVectorize] Extract the last lane from a uniform store"

This reverts commit 0d748b4d32cbddf58a1ff83f3ff178ec1ad49edc.
This is causing some failures when building Spec2017 with scalable
vectors. Reverting to investigate.

show more ...

[LoopVectorize] Extract the last lane from a uniform store

Changes VPReplicateRecipe to extract the last lane from an unconditional,
uniform store instruction. collectLoopUniforms will also add stor

[LoopVectorize] Extract the last lane from a uniform store

Changes VPReplicateRecipe to extract the last lane from an unconditional,
uniform store instruction. collectLoopUniforms will also add stores to
the list of uniform instructions where Legal->isUniformMemOp is true.

setCostBasedWideningDecision now sets the widening decision for
all uniform memory ops to Scalarize, where previously GatherScatter
may have been chosen for scalable stores.

This fixes an assert ("Cannot yet scalarize uniform stores") in
setCostBasedWideningDecision when we have a loop containing a
uniform i1 store and a scalable VF, which we cannot create a scatter for.

Reviewed By: sdesmalen, david-arm, fhahn

Differential Revision: https://reviews.llvm.org/D112725

show more ...

Revert rest of `IRBuilderBase`'s short-circuiting folds

Upon further investigation and discussion,
this is actually the opposite direction from what we should be taking,
and this direction wouldn't

Revert rest of `IRBuilderBase`'s short-circuiting folds

Upon further investigation and discussion,
this is actually the opposite direction from what we should be taking,
and this direction wouldn't solve the motivational problem anyway.

Additionally, some more (polly) tests have escaped being updated.
So, let's just take a step back here.

This reverts commit f3190dedeef9da2109ea57e4cb372f295ff53b88.
This reverts commit 749581d21f2b3f53e4fca4eb8728c942d646893b.
This reverts commit f3df87d57e096143670e0fd396e81d43393a2dd2.
This reverts commit ab1dbcecd6f0969976fafd62af34730436ad5944.

show more ...

[IR] `SCEVExpander::generateOverflowCheck()`: short-circuit `umul_with_overflow`-by-one

It's a no-op, no overflow happens ever: https://alive2.llvm.org/ce/z/Zw89rZ

While generally i don't like such

[IR] `SCEVExpander::generateOverflowCheck()`: short-circuit `umul_with_overflow`-by-one

It's a no-op, no overflow happens ever: https://alive2.llvm.org/ce/z/Zw89rZ

While generally i don't like such hacks,
we have a very good reason to do this: here we are expanding
a run-time correctness check for the vectorization,
and said `umul_with_overflow` will not be optimized out
before we query the cost of the checks we've generated.

Which means, the cost of run-time checks would be artificially inflated,
and after https://reviews.llvm.org/D109368 that will affect
the minimal trip count for which these checks are even evaluated.
And if they aren't even evaluated, then the vectorized code
certainly won't be run.

We could consider doing this in IRBuilder, but then we'd need to
also teach `CreateExtractValue()` to look into chain of `insertvalue`'s,
and i'm not sure there's precedent for that.

Refs. https://reviews.llvm.org/D109368#3089809

show more ...

[IR] `IRBuilderBase::CreateOr()`: fix short-circuiting for constant on LHS

There is no guarantee that the constant is on RHS here,
we have to handle both cases.

Refs. https://reviews.llvm.org/D1093

[IR] `IRBuilderBase::CreateOr()`: fix short-circuiting for constant on LHS

There is no guarantee that the constant is on RHS here,
we have to handle both cases.

Refs. https://reviews.llvm.org/D109368#3089809

show more ...

[IR] `IRBuilderBase::CreateSelect()`: if cond is a constant i1, short-circuit

While we could emit such a tautological `select`,
it will stick around until the next instsimplify invocation,
which may

[IR] `IRBuilderBase::CreateSelect()`: if cond is a constant i1, short-circuit

While we could emit such a tautological `select`,
it will stick around until the next instsimplify invocation,
which may happen after we count the cost of this redundant `select`.
Which is precisely what happens with loop vectorization legality checks,
and that artificially increases the cost of said checks,
which is bad.

There is prior art for this in `IRBuilderBase::CreateAnd()`/`IRBuilderBase::CreateOr()`.

Refs. https://reviews.llvm.org/D109368#3089809

show more ...

[NFC] Re-autogenerate check lines in some tests to ease of future update

[LV] Mark increment of main vector loop induction variable as NUW.

This patch marks the induction increment of the main induction variable
of the vector loop as NUW when not folding the tail.

If th

[LV] Mark increment of main vector loop induction variable as NUW.

This patch marks the induction increment of the main induction variable
of the vector loop as NUW when not folding the tail.

If the tail is not folded, we know that End - Start >= Step (either
statically or through the minimum iteration checks). We also know that both
Start % Step == 0 and End % Step == 0. We exit the vector loop if %IV +
%Step == %End. Hence we must exit the loop before %IV + %Step unsigned
overflows and we can mark the induction increment as NUW.

This should make SCEV return more precise bounds for the created vector
loops, used by later optimizations, like late unrolling.

At the moment quite a few tests still need to be updated, but before
doing so I'd like to get initial feedback to make sure I am not missing
anything.

Note that this could probably be further improved by using information
from the original IV.

Attempt of modeling of the assumption in Alive2:
https://alive2.llvm.org/ce/z/H_DL_g

Part of a set of fixes required for PR50412.

Reviewed By: mkazantsev

Differential Revision: https://reviews.llvm.org/D103255

show more ...

Revert "[NFC] remove explicit default value for strboolattr attribute in tests"

This reverts commit bda6e5bee04c75b1f1332b4fd1ac4e8ef6c3c247.

See https://lab.llvm.org/buildbot/#/builders/109/builds

Revert "[NFC] remove explicit default value for strboolattr attribute in tests"

This reverts commit bda6e5bee04c75b1f1332b4fd1ac4e8ef6c3c247.

See https://lab.llvm.org/buildbot/#/builders/109/builds/15424 for instance

show more ...

[NFC] remove explicit default value for strboolattr attribute in tests

Since d6de1e1a71406c75a4ea4d5a2fe84289f07ea3a1, no attributes is quivalent to
setting attribute to false.

This is a preliminar

[NFC] remove explicit default value for strboolattr attribute in tests

Since d6de1e1a71406c75a4ea4d5a2fe84289f07ea3a1, no attributes is quivalent to
setting attribute to false.

This is a preliminary commit for https://reviews.llvm.org/D99080

show more ...

[LV] Generate RT checks up-front and remove them if required.

This patch updates LV to generate the runtime checks just after cost
modeling, to allow a more precise estimate of the actual cost of th

[LV] Generate RT checks up-front and remove them if required.

This patch updates LV to generate the runtime checks just after cost
modeling, to allow a more precise estimate of the actual cost of the
checks. This information will be used in future patches to generate
larger runtime checks in cases where the checks only make up a small
fraction of the expected scalar loop execution time.

The runtime checks are created up-front in a temporary block to allow better
estimating the cost and un-linked from the existing IR. After deciding to
vectorize, the checks are moved backed. If deciding not to vectorize, the
temporary block is completely removed.

This patch is similar in spirit to D71053, but explores a different
direction: instead of delaying the decision on whether to vectorize in
the presence of runtime checks it instead optimistically creates the
runtime checks early and discards them later if decided to not
vectorize. This has the advantage that the cost-modeling decisions
can be kept together and can be done up-front and thus preserving the
general code structure. I think delaying (part) of the decision to
vectorize would also make the VPlan migration a bit harder.

One potential drawback of this patch is that we speculatively
generate IR which we might have to clean up later. However it seems like
the code required to do so is quite manageable.

Reviewed By: lebedev.ri, ebrevnov

Differential Revision: https://reviews.llvm.org/D75980

show more ...

Migrate function attribute "no-frame-pointer-elim"="false" to "frame-pointer"="none" as cleanups after D56351

Revert "[IRBuilder] Fold consistently for or/and whether constant is LHS or RHS"

This reverts commit r365260 which broke the following tests:

Clang :: CodeGenCXX/cfi-mfcall.cpp
Clang :: Cod

Revert "[IRBuilder] Fold consistently for or/and whether constant is LHS or RHS"

This reverts commit r365260 which broke the following tests:

Clang :: CodeGenCXX/cfi-mfcall.cpp
Clang :: CodeGenObjC/ubsan-nullability.m
LLVM :: Transforms/LoopVectorize/AArch64/pr36032.ll

llvm-svn: 365284

show more ...

[IRBuilder] Fold consistently for or/and whether constant is LHS or RHS

Without this, we have the unfortunate property that tests are dependent on the order of operads passed the CreateOr and Create

[IRBuilder] Fold consistently for or/and whether constant is LHS or RHS

Without this, we have the unfortunate property that tests are dependent on the order of operads passed the CreateOr and CreateAnd functions. In actual usage, we'd promptly optimize them away, but it made tests slightly more verbose than they should have been.

llvm-svn: 365260

show more ...

[SCEV] Pass NoWrapFlags when expanding an AddExpr

InsertBinop now accepts NoWrapFlags, so pass them through when
expanding a simple add expression.

This is the first re-commit of the functional cha

[SCEV] Pass NoWrapFlags when expanding an AddExpr

InsertBinop now accepts NoWrapFlags, so pass them through when
expanding a simple add expression.

This is the first re-commit of the functional changes from rL362687,
which was previously reverted.

Differential Revision: https://reviews.llvm.org/D61934

llvm-svn: 363364

show more ...

Revert "[SCEV] Use wrap flags in InsertBinop"

This reverts commit r362687. Miscompiles llvm-profdata during selfhost.

llvm-svn: 362699

[SCEV] Use wrap flags in InsertBinop

If the given SCEVExpr has no (un)signed flags attached to it, transfer
these to the resulting instruction or use them to find an existing
instruction.

Different

[SCEV] Use wrap flags in InsertBinop

If the given SCEVExpr has no (un)signed flags attached to it, transfer
these to the resulting instruction or use them to find an existing
instruction.

Differential Revision: https://reviews.llvm.org/D61934

llvm-svn: 362687

show more ...

Revert "Temporarily Revert "Add basic loop fusion pass.""

The reversion apparently deleted the test/Transforms directory.

Will be re-reverting again.

llvm-svn: 358552

[LoopVectorize] auto-generate complete checks; NFC

The first test claims to show that the vectorizer will
generate a vector load/loop, but then this file runs
other passes which might scalarize that

[LoopVectorize] auto-generate complete checks; NFC

The first test claims to show that the vectorizer will
generate a vector load/loop, but then this file runs
other passes which might scalarize that op. I'm removing
instcombine from the RUN line here to break that dependency.
Also, I'm generating full checks to make it clear exactly
what the vectorizer has done.

llvm-svn: 349554

show more ...

[LV] Teach vectorizer about variant value store into uniform address

Summary:
Teach vectorizer about vectorizing variant value stores to uniform
address. Similar to rL343028, we do not allow vectori

[LV] Teach vectorizer about variant value store into uniform address

Summary:
Teach vectorizer about vectorizing variant value stores to uniform
address. Similar to rL343028, we do not allow vectorization if we have
multiple stores to the same uniform address.

Cost model already has the change for considering the extract
instruction cost for a variant value store. See added test cases for how
vectorization is done.
The patch also contains changes to the ORE messages.

Reviewers: Ayal, mkuper, anemet, hsaito

Subscribers: rkruppe, llvm-commits

Differential Revision: https://reviews.llvm.org/D52656

llvm-svn: 344613

show more ...