[GlobalOpt] Enable evaluation of atomic loads

Relax the check to allow evaluation of atomic loads
(but still skip volatile loads).

Test plan:
1/ ninja check-llvm check-clang
2/ Bootstrapped LLVM/Cl

[GlobalOpt] Enable evaluation of atomic loads

Relax the check to allow evaluation of atomic loads
(but still skip volatile loads).

Test plan:
1/ ninja check-llvm check-clang
2/ Bootstrapped LLVM/Clang pass tests

Differential revision: https://reviews.llvm.org/D130211

show more ...

[GlobalOpt] Enable evaluation of atomic stores

Relax the check to allow evaluation of atomic stores
(but still skip volatile stores).

Test plan:
1/ ninja check-llvm check-clang
2/ Bootstrapped LLVM

[GlobalOpt] Enable evaluation of atomic stores

Relax the check to allow evaluation of atomic stores
(but still skip volatile stores).

Test plan:
1/ ninja check-llvm check-clang
2/ Bootstrapped LLVM/Clang pass tests

Differential revision: https://reviews.llvm.org/D129841

show more ...

[Evaluator] Add missing LLVM_DEBUG()

Missed these in 41f0b6a78143776d673565cfa830849e3b468b8e, resulting
in unconditional debug output.

[Evaluator] Use ConstantFoldInstOperands()

For instructions that don't need any special handling, use
ConstantFoldInstOperands(), rather than re-implementing individual
cases.

This is probably not

[Evaluator] Use ConstantFoldInstOperands()

For instructions that don't need any special handling, use
ConstantFoldInstOperands(), rather than re-implementing individual
cases.

This is probably not NFC because it can handle cases the previous
code missed (e.g. vector operations).

show more ...

[IR] Remove support for extractvalue constant expression

This removes the extractvalue constant expression, as part of
https://discourse.llvm.org/t/rfc-remove-most-constant-expressions/63179.
extrac

[IR] Remove support for extractvalue constant expression

This removes the extractvalue constant expression, as part of
https://discourse.llvm.org/t/rfc-remove-most-constant-expressions/63179.
extractvalue is already not supported in bitcode, so we do not need
to worry about bitcode auto-upgrade.

Uses of ConstantExpr::getExtractValue() should be replaced with
IRBuilder::CreateExtractValue() (if the fact that the result is
constant is not important) or ConstantFoldExtractValueInstruction()
(if it is). Though for this particular case, it is also possible
and usually preferable to use getAggregateElement() instead.

The C API function LLVMConstExtractValue() is removed, as the
underlying constant expression no longer exists. Instead,
LLVMBuildExtractValue() should be used (which will constant fold
or create an instruction). Depending on the use-case,
LLVMGetAggregateElement() may also be used instead.

Differential Revision: https://reviews.llvm.org/D125795

show more ...

[GlobalOpt] Fix memset handling in global ctor evaluation (PR55859)

The global ctor evaluator currently handles by checking whether the
memset memory is already zero, and skips it in that case. Howe

[GlobalOpt] Fix memset handling in global ctor evaluation (PR55859)

The global ctor evaluator currently handles by checking whether the
memset memory is already zero, and skips it in that case. However,
it only actually checks the first byte of the memory being set.

This patch extends the code to check all bytes being set. This is
done byte-by-byte to avoid converting undef values to zeros in
larger reads. However, the handling is still not completely correct,
because there might still be padding bytes (though probably this
doesn't matter much in practice, as I'd expect global variable
padding to be zero-initialized in practice).

Mostly fixes https://github.com/llvm/llvm-project/issues/55859.

Differential Revision: https://reviews.llvm.org/D128532

show more ...

[OpaquePtr][GlobalOpt] Don't attempt to evaluate global constructors with arguments

Previously all entries in global_ctors had to have the void()* type and
we'd skip evaluating bitcasted functions.

[OpaquePtr][GlobalOpt] Don't attempt to evaluate global constructors with arguments

Previously all entries in global_ctors had to have the void()* type and
we'd skip evaluating bitcasted functions. With opaque pointers we may
see the function directly.

Fixes #55147.

Reviewed By: #opaque-pointers, nikic

Differential Revision: https://reviews.llvm.org/D124553

show more ...

Cleanup includes: TransformsUtils

Estimation on the impact on preprocessor output:
before: 1065307662
after: 1064800684

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-

Cleanup includes: TransformsUtils

Estimation on the impact on preprocessor output:
before: 1065307662
after: 1064800684

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120741

show more ...

[Evaluator] Simplify handling of bitcasted calls

When fetching the function, strip casts. When casting the result,
use the call result type. Don't actually inspect the bitcast.

[GlobalOpt][Evaluator] Fix off by one error in bounds check (PR53002)

We should bail out if the index is >= the size, not > the size.

Fixes https://github.com/llvm/llvm-project/issues/53002.

[GlobalOpt][Evaluator] Don't create bitcast for same type (PR52994)

isBitOrNoopPointerCastable() returns true if the types are the
same, but it's not actually possible to create a bitcast for all
su

[GlobalOpt][Evaluator] Don't create bitcast for same type (PR52994)

isBitOrNoopPointerCastable() returns true if the types are the
same, but it's not actually possible to create a bitcast for all
such types. The assumption seems to be that the user will omit
creating the cast in that case, as it is unnecessary.

Fixes https://github.com/llvm/llvm-project/issues/52994.

show more ...

[GlobalOpt][Evaluator] Rewrite global ctor evaluation (fixes PR51879)

Global ctor evaluation currently models memory as a map from Constant*
to Constant*. For this to be correct, it is required that

[GlobalOpt][Evaluator] Rewrite global ctor evaluation (fixes PR51879)

Global ctor evaluation currently models memory as a map from Constant*
to Constant*. For this to be correct, it is required that there is
only a single Constant* referencing a given memory location. The
Evaluator tries to ensure this by imposing certain limitations that
could result in ambiguities (by limiting types, casts and GEP formats),
but ultimately still fails, as can be seen in PR51879. The approach
is fundamentally fragile and will get more so with opaque pointers.

My original thought was to instead store memory for each global as an
offset => value representation. However, we also need to make sure
that we can actually rematerialize the modified global initializer
into a Constant in the end, which may not be possible if we allow
arbitrary writes.

What this patch does instead is to represent globals as a MutableValue,
which is either a Constant* or a MutableAggregate*. The mutable
aggregate exists to allow efficient mutation of individual aggregate
elements, as mutating an element on a Constant would require interning
a new constant. When a write to the Constant* is made, it is converted
into a MutableAggregate* as needed.

I believe this should make the evaluator more robust, compatible
with opaque pointers, and a bit simpler as well.

Fixes https://github.com/llvm/llvm-project/issues/51221.

Differential Revision: https://reviews.llvm.org/D115530

show more ...

[llvm] Use range-based for loops (NFC)

[NFC][OpaquePtr][Evaluator] Remove call to PointerType::getElementType

There are still another 2 uses of PointerType::getElementType in
Evaluator when evaluating BitCast's on pointers. BitCast's on

[NFC][OpaquePtr][Evaluator] Remove call to PointerType::getElementType

There are still another 2 uses of PointerType::getElementType in
Evaluator when evaluating BitCast's on pointers. BitCast's on pointers
should be removed when opaque ptr is ready, so I just keep them as is.

Reviewed By: aeubanks

Differential Revision: https://reviews.llvm.org/D114131

show more ...

[Transforms] Migrate from getNumArgOperands to arg_size (NFC)

Note that getNumArgOperands is considered a legacy name. See
llvm/include/llvm/IR/InstrTypes.h for details.

[GlobalOpt] Fix a miscompile when evaluating struct initializers.

The bug was that evaluateBitcastFromPtr attempts a narrowing to a struct's 0th
element of a store that covers other elements. While

[GlobalOpt] Fix a miscompile when evaluating struct initializers.

The bug was that evaluateBitcastFromPtr attempts a narrowing to a struct's 0th
element of a store that covers other elements. While this is okay on the load
side, applying it to stores causes us to miss the writes to the additionally
covered elements.

rdar://79503568

Differential revision: https://reviews.llvm.org/D105838

show more ...

[Evaluator] Bitcast result of pointer stripping

Trying to evaluate a GEP would assert with
"Ty == cast<PointerType>(C->getType()->getScalarType())->getElementType()"
because the type of the pointe

[Evaluator] Bitcast result of pointer stripping

Trying to evaluate a GEP would assert with
"Ty == cast<PointerType>(C->getType()->getScalarType())->getElementType()"
because the type of the pointer we would evaluate the GEP argument to
would be a different type than the GEP was expecting. We should treat
pointer stripping as a bitcast.

The test adds a redundant GEP that would crash due to type mismatch.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D100970

show more ...

Explicitly pass type to cast load constant folding result

Previously we would use the type of the pointee to determine what to
cast the result of constant folding a load. To aid with opaque pointer

Explicitly pass type to cast load constant folding result

Previously we would use the type of the pointee to determine what to
cast the result of constant folding a load. To aid with opaque pointer
types, we should explicitly pass the type of the load rather than
looking at pointee types.

ConstantFoldLoadThroughBitcast() converts the const prop'd value to the
proper load type (e.g. [1 x i32] -> i32). Instead of calling this in
every intermediate step like bitcasts, we only call this when we
actually see the global initializer value.

In some existing uses of this API, we don't know the exact type we're
loading from immediately (e.g. first we visit a bitcast, then we visit
the load using the bitcast). In those cases we have to manually call
ConstantFoldLoadThroughBitcast() when simplifying the load to make sure
that we cast to the proper type.

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D100718

show more ...

[Evaluator] Look through invariant.group intrinsics

Turning on -fstrict-vtable-pointers in Chrome caused an extra global
initializer. Turns out that a llvm.strip.invariant.group intrinsic was
causin

[Evaluator] Look through invariant.group intrinsics

Turning on -fstrict-vtable-pointers in Chrome caused an extra global
initializer. Turns out that a llvm.strip.invariant.group intrinsic was
causing GlobalOpt to fail to step through some simple code.

We can treat *.invariant.group uses as simply their operand.
Value::stripPointerCastsForAliasAnalysis() does exactly this. This
should be safe because the Evaluator does not skip memory accesses due
to invariants or alias analysis.

However, we don't want to leak that we've stripped arbitrary pointer
casts to users of Evaluator, so we bail out if we evaluate a function to
any constant, since we may have looked through *.invariant.group calls
and aliasing pointers cannot be arbitrarily substituted.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D98843

show more ...

[Transforms/Utils] Use range-based for loops (NFC)

[CodeGen, Transforms] Use *Map::lookup (NFC)

Fix for Bug 48055.

Differential Revision: https://reviews.llvm.org/D92599

[CSSPGO] IR intrinsic for pseudo-probe block instrumentation

This change introduces a new IR intrinsic named `llvm.pseudoprobe` for pseudo-probe block instrumentation. Please refer to https://review

[CSSPGO] IR intrinsic for pseudo-probe block instrumentation

This change introduces a new IR intrinsic named `llvm.pseudoprobe` for pseudo-probe block instrumentation. Please refer to https://reviews.llvm.org/D86193 for the whole story.

A pseudo probe is used to collect the execution count of the block where the probe is instrumented. This requires a pseudo probe to be persisting. The LLVM PGO instrumentation also instruments in similar places by placing a counter in the form of atomic read/write operations or runtime helper calls. While these operations are very persisting or optimization-resilient, in theory we can borrow the atomic read/write implementation from PGO counters and cut it off at the end of compilation with all the atomics converted into binary data. This was our initial design and we’ve seen promising sample correlation quality with it. However, the atomics approach has a couple issues:

1. IR Optimizations are blocked unexpectedly. Those atomic instructions are not going to be physically present in the binary code, but since they are on the IR till very end of compilation, they can still prevent certain IR optimizations and result in lower code quality.
2. The counter atomics may not be fully cleaned up from the code stream eventually.
3. Extra work is needed for re-targeting.

We choose to implement pseudo probes based on a special LLVM intrinsic, which is expected to have most of the semantics that comes with an atomic operation but does not block desired optimizations as much as possible. More specifically the semantics associated with the new intrinsic enforces a pseudo probe to be virtually executed exactly the same number of times before and after an IR optimization. The intrinsic also comes with certain flags that are carefully chosen so that the places they are probing are not going to be messed up by the optimizer while most of the IR optimizations still work. The core flags given to the special intrinsic is `IntrInaccessibleMemOnly`, which means the intrinsic accesses memory and does have a side effect so that it is not removable, but is does not access memory locations that are accessible by any original instructions. This way the intrinsic does not alias with any original instruction and thus it does not block optimizations as much as an atomic operation does. We also assign a function GUID and a block index to an intrinsic so that they are uniquely identified and not merged in order to achieve good correlation quality.

Let's now look at an example. Given the following LLVM IR:

```
define internal void @foo2(i32 %x, void (i32)* %f) !dbg !4 {
bb0:
%cmp = icmp eq i32 %x, 0
br i1 %cmp, label %bb1, label %bb2
bb1:
br label %bb3
bb2:
br label %bb3
bb3:
ret void
}
```

The instrumented IR will look like below. Note that each `llvm.pseudoprobe` intrinsic call represents a pseudo probe at a block, of which the first parameter is the GUID of the probe’s owner function and the second parameter is the probe’s ID.

```
define internal void @foo2(i32 %x, void (i32)* %f) !dbg !4 {
bb0:
%cmp = icmp eq i32 %x, 0
call void @llvm.pseudoprobe(i64 837061429793323041, i64 1)
br i1 %cmp, label %bb1, label %bb2
bb1:
call void @llvm.pseudoprobe(i64 837061429793323041, i64 2)
br label %bb3
bb2:
call void @llvm.pseudoprobe(i64 837061429793323041, i64 3)
br label %bb3
bb3:
call void @llvm.pseudoprobe(i64 837061429793323041, i64 4)
ret void
}

```

Reviewed By: wmi

Differential Revision: https://reviews.llvm.org/D86490

show more ...

[IR] Replace all uses of CallBase::getCalledValue() with getCalledOperand().

This method has been commented as deprecated for a while. Remove
it and replace all uses with the equivalent getCalledOpe

[IR] Replace all uses of CallBase::getCalledValue() with getCalledOperand().

This method has been commented as deprecated for a while. Remove
it and replace all uses with the equivalent getCalledOperand().

I also made a few cleanups in here. For example, to removes use
of getElementType on a pointer when we could just use getFunctionType
from the call.

Differential Revision: https://reviews.llvm.org/D78882

show more ...

[llvm][NFC][CallSite] Remove CallSite from Evaluator.

Reviewers: craig.topper, dblaikie

Subscribers: hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D78395