[InstCombine][SVE] Bail out of isSafeToLoadUnconditionally for scalable types

`isSafeToLoadUnconditionally` currently assumes sized types. Bail out for now.
This fixes a TypeSize warning reachable f

[InstCombine][SVE] Bail out of isSafeToLoadUnconditionally for scalable types

`isSafeToLoadUnconditionally` currently assumes sized types. Bail out for now.
This fixes a TypeSize warning reachable from instcombine via (load (select
cond, ptr, ptr)).

Differential Revision: https://reviews.llvm.org/D129477

show more ...

[Loads] Check type size in bits during store to load forwarding

Rather than checking the rounded type store size, check the type
size in bits. We don't want to forward a store of i1 to a load
of i8

[Loads] Check type size in bits during store to load forwarding

Rather than checking the rounded type store size, check the type
size in bits. We don't want to forward a store of i1 to a load
of i8 for example, even though they have the same type store size.
The padding bits have unspecified contents.

This is a partial fix for the issue reported at
https://reviews.llvm.org/D115924#inline-1179482,
the problem also needs to be addressed more generally in the
constant folding code.

show more ...

Cleanup includes: LLVMAnalysis

Number of lines output by preprocessor:
before: 1065940348
after: 1065307662

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Diff

Cleanup includes: LLVMAnalysis

Number of lines output by preprocessor:
before: 1065940348
after: 1065307662

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120659

show more ...

[Loads] Require Align in isDereferenceableAndAlignedPointer() (NFC)

Now that loads always have an alignment, we should not perform an
ABI alignment fallback here.

[Loads] Handle addrspacecast constant expressions when determining dereferenceability

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D114015

[Loads] Use more powerful constant folding API

This follows up on D111023 by exporting the generic "load value
from constant at given offset as given type" and using it in the
store to load forwardi

[Loads] Use more powerful constant folding API

This follows up on D111023 by exporting the generic "load value
from constant at given offset as given type" and using it in the
store to load forwarding code. We now need to make sure that the
load size is smaller than the store size, previously this was
implicitly ensured by ConstantFoldLoadThroughBitcast().

Differential Revision: https://reviews.llvm.org/D112260

show more ...

Fix incorrect GEP bitwidth in areNonOverlapSameBaseLoadAndStore()

When using a datalayout that has pointer index width != pointer size this
code triggers an assertion in Value::stripAndAccumulateCon

Fix incorrect GEP bitwidth in areNonOverlapSameBaseLoadAndStore()

When using a datalayout that has pointer index width != pointer size this
code triggers an assertion in Value::stripAndAccumulateConstantOffsets().
I encountered this this while compiling FreeBSD for CHERI-RISC-V.
Also update LoadsTest.cpp to use a DataLayout with index width != pointer
width to ensure this case is tested.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D110406

show more ...

[CSSPGO] Unblock optimizations with pseudo probe instrumentation part 2.

As a follow-up to D95982, this patch continues unblocking optimizations that are blocked by pseudu probe instrumention.

The

[CSSPGO] Unblock optimizations with pseudo probe instrumentation part 2.

As a follow-up to D95982, this patch continues unblocking optimizations that are blocked by pseudu probe instrumention.

The optimizations unblocked are:
- In-block load propagation.
- In-block dead store elimination
- Memory copy optimization that turns stores to consecutive memories into a memset.

These optimizations are local to a block, so they shouldn't affect the profile quality.

Reviewed By: wmi

Differential Revision: https://reviews.llvm.org/D100075

show more ...

fix whitespace type

[Analysis] isDereferenceableAndAlignedPointer(): recurse into select's hands

By doing this within the method itself,
we support traversing multiple levels of selects (TODO: PHI's),
fixing the SROA `

[Analysis] isDereferenceableAndAlignedPointer(): recurse into select's hands

By doing this within the method itself,
we support traversing multiple levels of selects (TODO: PHI's),
fixing the SROA `std::clamp()` testcase.

Fixes https://bugs.llvm.org/show_bug.cgi?id=47271
Mostly fixes https://bugs.llvm.org/show_bug.cgi?id=49909

show more ...

Move GCRelocateInst and GCResultInst to IntrinsicInst.h [nfc]

These two are part of the IntrinsicInst class hierarchy and it helps to cut down on some redundant includes.

[Loads] Forward constant vector store to load of first element

InstCombine performs simple forwarding from stores to loads, but
currently only handles the case where the load and store have the
same

[Loads] Forward constant vector store to load of first element

InstCombine performs simple forwarding from stores to loads, but
currently only handles the case where the load and store have the
same size. This extends it to also handle a store of a constant
with a larger size followed by a load with a smaller size.

This is implemented through ConstantFoldLoadThroughBitcast() which
is fairly primitive (e.g. does not allow storing a large integer
and then loading a small one), but at least can forward the first
element of a vector store. Unfortunately it seems that we currently
don't have a generic helper for "read a constant value as a different
type", it's all tangled up with other logic in either
ConstantFolding or VNCoercion.

Differential Revision: https://reviews.llvm.org/D98114

show more ...

Infer dereferenceability from malloc and friends

Hookup TLI when inferring object size from allocation calls. This allows the analysis to prove dereferenceability for known allocation functions (suc

Infer dereferenceability from malloc and friends

Hookup TLI when inferring object size from allocation calls. This allows the analysis to prove dereferenceability for known allocation functions (such as malloc/new/etc) in addition to those marked explicitly with the allocsize attribute.

This is a follow up to 0129cd5 now that the bug fixed by e2c6621e6 is resolved.

As noted in the test, this relies on being able to prove that there is no free between allocation and context (e.g. hoist location). At the moment, this is handled conservatively. I'm working strengthening out ability to reason about no-free regions separately.

Differential Revision: https://reviews.llvm.org/D99737

show more ...

[deref-at-point] restrict inference of dereferenceability based on allocsize attribute

Support deriving dereferenceability facts from allocation sites with known object sizes while correctly account

[deref-at-point] restrict inference of dereferenceability based on allocsize attribute

Support deriving dereferenceability facts from allocation sites with known object sizes while correctly accounting for any possibly frees between allocation and use site. (At the moment, we're conservative and only allowing it in functions where we know we can't free.)

This is part of the work on deref-at-point semantics. I'm making the change unconditional as the miscompile in this case is way too easy to trip by accident, and the optimization was only recently added (by me).

There will be a follow up patch wiring through TLI since that should now be doable without introducing widespread miscompiles.

Differential Revision: https://reviews.llvm.org/D95815

show more ...

Make FindAvailableLoadedValue TBAA aware

FindAvailableLoadedValue() relies on FindAvailablePtrLoadStore() to run
the alias analysis when searching for an equivalent value. However,
FindAvailablePtrL

Make FindAvailableLoadedValue TBAA aware

FindAvailableLoadedValue() relies on FindAvailablePtrLoadStore() to run
the alias analysis when searching for an equivalent value. However,
FindAvailablePtrLoadStore() calls the alias analysis framework with a
memory location for the load constructed from an address and a size,
which thus lacks TBAA metadata info. This commit modifies
FindAvailablePtrLoadStore() to accept an optional memory location as
parameter to allow FindAvailableLoadedValue() to create it based on the
load instruction, which would then have TBAA metadata info attached.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D99206

show more ...

Update basic deref API to account for possiblity of free [NFC]

This patch is plumbing to support work towards the goal outlined in the recent llvm-dev post "[llvm-dev] RFC: Decomposing deref(N) into

Update basic deref API to account for possiblity of free [NFC]

This patch is plumbing to support work towards the goal outlined in the recent llvm-dev post "[llvm-dev] RFC: Decomposing deref(N) into deref(N) + nofree".

The point of this change is purely to simplify iteration on other pieces on way to making the switch. Rebuilding with a change to Value.h is slow and painful, so I want to get the API change landed. Once that's done, I plan to more closely audit each caller, add the inference rules in their own patch, then post a patch with the langref changes and test diffs. The value of the command line flag is that we can exercise the inference logic in standalone patches without needing the whole switch ready to go just yet.

Differential Revision: https://reviews.llvm.org/D98908

show more ...

[Loads] Restructure getAvailableLoadStore implementation (NFC)

Separate out some conditions with early exits, to make it easier to
support additional cases.

[Loads] Add optimized FindAvailableLoadedValue() overload (NFCI)

FindAvailableLoadedValue() accepts an iterator by reference. If no
available value is found, then the iterator will either be left
at

[Loads] Add optimized FindAvailableLoadedValue() overload (NFCI)

FindAvailableLoadedValue() accepts an iterator by reference. If no
available value is found, then the iterator will either be left
at a clobbering instruction or the beginning of the basic block.
This allows using FindAvailableLoadedValue() across multiple blocks.

If this functionality is not needed, as is the case in InstCombine,
then we can use a much more efficient implementation: First try
to find an available value, and only perform clobber checks if
we actually found one. As this function only looks at a very small
number of instructions (6 by default) and usually doesn't find an
available value, this saves many expensive alias analysis queries.

show more ...

[Loads] Extract helper frunction for available load/store (NFC)

This contains the logic for extracting an available load/store
from a given instruction, to be reused in a following patch.

reland [InstCombine] convert assumes to operand bundles

Instcombine will convert the nonnull and alignment assumption that use the boolean condtion
to an assumption that uses the operand bundles whe

reland [InstCombine] convert assumes to operand bundles

Instcombine will convert the nonnull and alignment assumption that use the boolean condtion
to an assumption that uses the operand bundles when knowledge retention is enabled.

Differential Revision: https://reviews.llvm.org/D82703

show more ...

Revert "[InstCombine] convert assumes to operand bundles"

This reverts commit 5eb2e994f9b3a5aff0a156d0a1f7e6121342cc11.

[InstCombine] convert assumes to operand bundles

Instcombine will convert the nonnull and alignment assumption that use the boolean condtion
to an assumption that uses the operand bundles when knowl

[InstCombine] convert assumes to operand bundles

Instcombine will convert the nonnull and alignment assumption that use the boolean condtion
to an assumption that uses the operand bundles when knowledge retention is enabled.

Differential Revision: https://reviews.llvm.org/D82703

show more ...

[Loads] Plumb through TLI argument [NFC]

This is a (rather delayed) follow up to commit 0129cd5. This commit is entirely NFC, the semantic change to leverage the new information will be submitted s

[Loads] Plumb through TLI argument [NFC]

This is a (rather delayed) follow up to commit 0129cd5. This commit is entirely NFC, the semantic change to leverage the new information will be submitted separate with a test case.

show more ...

Use deref facts derived from minimum object size of allocations

This change should be fairly straight forward. If we've reached a call, check to see if we can tell the result is dereferenceable from

Use deref facts derived from minimum object size of allocations

This change should be fairly straight forward. If we've reached a call, check to see if we can tell the result is dereferenceable from information about the minimum object size returned by the call.

To control compile time impact, I'm only adding the call for base facts in the routine. getObjectSize can also do recursive reasoning, and we don't want that general capability here.

As a follow up patch (without separate review), I will plumb through the missing TLI parameter. That will have the effect of extending this to known libcalls - malloc, new, and the like - whereas currently this only covers calls with the explicit allocsize attribute.

Differential Revision: https://reviews.llvm.org/D90341

show more ...

[Deref] Use maximum trip count instead of exact trip count

When trying to prove that a memory access touches only dereferenceable memory across all iterations of a loop, use the maximum exit count r

[Deref] Use maximum trip count instead of exact trip count

When trying to prove that a memory access touches only dereferenceable memory across all iterations of a loop, use the maximum exit count rather than an exact one. In many cases we can't prove exact exit counts whereas we can prove an upper bound.

The test included is for a single exit loop with a min(C,V) exit count, but the true motivation is support for multiple exits loops. It's just really hard to write a test case for multiple exits because the vectorizer (the primary user of this API), bails far before this. For multiple exits, this allows a mix of analyzeable and unanalyzable exits when only analyzeable exits are needed to prove deref.

show more ...