[MLInliner] No need to invalidate everything post-inlining.

We really just need to invalidate loop info and the dominator tree, in
addition to the FunctionPropertiesInfo we were invalidating origina

[MLInliner] No need to invalidate everything post-inlining.

We really just need to invalidate loop info and the dominator tree, in
addition to the FunctionPropertiesInfo we were invalidating originally.
Doing more adds unnecessary compile time overhead.

show more ...

[MLInliner] Don't inline call sites in unreachable basic blocks

This requires DominatorTree be updated, which we do in the ml inliner
case, but not in the default case, and the cost of doing so is
n

[MLInliner] Don't inline call sites in unreachable basic blocks

This requires DominatorTree be updated, which we do in the ml inliner
case, but not in the default case, and the cost of doing so is
noticeable to compile time for the latter[1]. So the patch only affects
the ML inliner.

[1] https://llvm-compile-time-tracker.com/compare.php?from=9fc0aa45e3312944431ba7e1ca0cec99c613992b&to=7af461b1ce0d9138211ef5f883f35d5b9ddf47be&stat=wall-time

Differential Revision: https://reviews.llvm.org/D127899

show more ...

[mlgo] Fix accounting for SCC splits

Previously if the inliner split an SCC such that an empty one remained, the MLInlineAdvisor could potentially lose track of the EdgeCount if a subsequent CGSCC p

[mlgo] Fix accounting for SCC splits

Previously if the inliner split an SCC such that an empty one remained, the MLInlineAdvisor could potentially lose track of the EdgeCount if a subsequent CGSCC pass modified the calls of a function that was initially in the SCC pre-split. Saving the seen nodes in onPassEntry resolves this.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D127693

show more ...

FunctionPropertiesAnalysis: handle callsite BBs that lose edges

There could be successors that were reached before but now are only
reachable from elsewhere in the CFG.

Suppose the following diamon

FunctionPropertiesAnalysis: handle callsite BBs that lose edges

There could be successors that were reached before but now are only
reachable from elsewhere in the CFG.

Suppose the following diamond CFG (lines are arrows pointing down):
A
/ \
B C
\ /
D
There's a call site in C that is inlined. Upon doing that, it turns out
it expands to:
call void @llvm.trap()
unreachable
D isn't reachable from C anymore, but we did discount it when we set up
FunctionPropertiesUpdater, so we need to re-include it here.

The patch also updates loop accounting to use LoopInfo rather than
traverse BBs.

Differential Revision: https://reviews.llvm.org/D127353

show more ...

[mlgo] Update FunctionPropertyCache after invalidating analyses

The update depends on LoopInfo, so we need that refreshed first, not
after.

Differential Revision: https://reviews.llvm.org/D127467

[mlgo] Disable accounting upon ForceStop

Once ForceStop is set to true, we only return positive inlining advice when it is mandatory; There is no need for further node/edge accounting.

Reviewed By:

[mlgo] Disable accounting upon ForceStop

Once ForceStop is set to true, we only return positive inlining advice when it is mandatory; There is no need for further node/edge accounting.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D127245

show more ...

[mlgo] Incrementally update FunctionPropertiesInfo during inlining

Re-computing FunctionPropertiesInfo after each inlining may be very time
consuming: in certain cases, e.g. large caller with lots o

[mlgo] Incrementally update FunctionPropertiesInfo during inlining

Re-computing FunctionPropertiesInfo after each inlining may be very time
consuming: in certain cases, e.g. large caller with lots of callsites,
and when the overall IR doesn't increase (thus not tripping a size bloat
threshold).

This patch addresses this by incrementally updating
FunctionPropertiesInfo.

Differential Revision: https://reviews.llvm.org/D125841

show more ...

[mlgo] Support exposing more features than those supported by models

This allows the compiler to support more features than those supported by a
model. The only requirement (development mode only) i

[mlgo] Support exposing more features than those supported by models

This allows the compiler to support more features than those supported by a
model. The only requirement (development mode only) is that the new
features must be appended at the end of the list of features requested
from the model. The support is transparent to compiler code: for
unsupported features, we provide a valid buffer to copy their values;
it's just that this buffer is disconnected from the model, so insofar
as the model is concerned (AOT or development mode), these features don't
exist. The buffers are allocated at setup - meaning, at steady state,
there is no extra allocation (maintaining the current invariant). These
buffers has 2 roles: one, keep the compiler code simple. Second, allow
logging their values in development mode. The latter allows retraining
a model supporting the larger feature set starting from traces produced
with the old model.

For release mode (AOT-ed models), this decouples compiler evolution from
model evolution, which we want in scenarios where the toolchain is
frequently rebuilt and redeployed: we can first deploy the new features,
and continue working with the older model, until a new model is made
available, which can then be picked up the next time the compiler is built.

Differential Revision: https://reviews.llvm.org/D124565

show more ...

Fix build breaks on ml-* bots introduced by include cleanups

Cleanup includes: LLVMAnalysis

Number of lines output by preprocessor:
before: 1065940348
after: 1065307662

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Diff

Cleanup includes: LLVMAnalysis

Number of lines output by preprocessor:
before: 1065940348
after: 1065307662

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120659

show more ...

[NFC][MLGO] Simplify conditional compilation

Most of the code that's shared between 'release' and 'development'
modes doesn't depend on anything special.

[MLGO] Improved support for AOT cross-targeting scenarios

The tensorflow AOT compiler can cross-target, but it can't run on (for
example) arm64. We added earlier support where the AOT-ed header and

[MLGO] Improved support for AOT cross-targeting scenarios

The tensorflow AOT compiler can cross-target, but it can't run on (for
example) arm64. We added earlier support where the AOT-ed header and object
would be built on a separate builder and then passed at build time to
a build host where the AOT compiler can't run, but clang can be otherwise
built.

To simplify such scenarios given we now support more than one AOT-able
case (regalloc and inliner), we make the AOT scenario centered on whether
files are generated, case by case (this includes the "passed from a
different builder" scenario).
This means we shouldn't need an 'umbrella' LLVM_HAVE_TF_AOT, in favor of
case by case control. A builder can opt out of an AOT case by passing that case's
model path as `none`. Note that the overrides still take precedence.

This patch controls conditional compilation with case-specific flags,
which can be enabled locally, for the component where those are
available. We still keep an overall flag for some tests.

The 'development/training' mode is unchanged, because there the model is
passed from the command line and interpreted.

Differential Revision: https://reviews.llvm.org/D117752

show more ...

[mlgo][inline] Improve global state tracking

The global state refers to the number of the nodes currently in the
module, and the number of direct calls between nodes, across the
module.

Node counts

[mlgo][inline] Improve global state tracking

The global state refers to the number of the nodes currently in the
module, and the number of direct calls between nodes, across the
module.

Node counts are not a problem; edge counts are because we want strictly
the kind of edges that affect inlining (direct calls), and that is not
easily obtainable without iteration over the whole module.

This patch avoids relying on analysis invalidation because it turned out
to be too aggressive in some cases. It leverages the fact that Node
objects are stable - they do not get deleted while cgscc passes are
run over the module; and cgscc pass manager invariants.

Reviewed By: aeubanks

Differential Revision: https://reviews.llvm.org/D115847

show more ...

[NFC][MLGO] Use LazyCallGraph::Node to track functions.

This avoids the InlineAdvisor carrying the responsibility of deleting
Function objects. We use LazyCallGraph::Node objects instead, which are

[NFC][MLGO] Use LazyCallGraph::Node to track functions.

This avoids the InlineAdvisor carrying the responsibility of deleting
Function objects. We use LazyCallGraph::Node objects instead, which are
stable in memory for the duration of the Module-wide performance of CGSCC
passes started under the same ModuleToPostOrderCGSCCPassAdaptor (which
is the case here)

Differential Revision: https://reviews.llvm.org/D116964

show more ...

[NFC] Expose the ReleaseModeModelRunner

The type was pretty much generic, just needed a bit of parameterization.

Differential Revision: https://reviews.llvm.org/D115764

[NFC][mlgo] Generalize model runner interface

This prepares it for the regalloc work. Part of it is making model
evaluation accross 'development' and 'release' scenarios more reusable.
This patch:
-

[NFC][mlgo] Generalize model runner interface

This prepares it for the regalloc work. Part of it is making model
evaluation accross 'development' and 'release' scenarios more reusable.
This patch:
- extends support to tensors of any shape (not just scalars, like we had
in the inliner -Oz case). While the tensor shape can be anything, we
assume row-major layout and expose the tensor as a buffer.
- exposes the NoInferenceModelRunner, which we use in the 'development'
mode to keep the evaluation code path consistent and simplify logging,
as we'll want to reuse it in the regalloc case.

Differential Revision: https://reviews.llvm.org/D115306

show more ...

[NFC][InlineAdvisor] Inform advisor when the module is invalidated

This avoids unnecessary re-calculation of module-wide features in the
MLInlineAdvisor. In cases where function passes don't invalid

[NFC][InlineAdvisor] Inform advisor when the module is invalidated

This avoids unnecessary re-calculation of module-wide features in the
MLInlineAdvisor. In cases where function passes don't invalidate
functions (and, thus, don't invalidate the module), but we re-process a
CGSCC, we currently refreshed module features unnecessarily. The
overhead of fetching cached results (albeit they weren't themselves
invalidated) was noticeable in certain modules' compilations.

We don't want to just invalidate the advisor object, though, via the
analysis manager, because we'd then need to re-create expensive state
(like the model evaluator in the ML 'development' mode).

Reviewed By: phosek

Differential Revision: https://reviews.llvm.org/D113644

show more ...

Unpack the CostEstimate feature in ML inlining models.

This change yields an additional 2% size reduction on an internal search
binary, and an additional 0.5% size reduction on fuchsia.

Differentia

Unpack the CostEstimate feature in ML inlining models.

This change yields an additional 2% size reduction on an internal search
binary, and an additional 0.5% size reduction on fuchsia.

Differential Revision: https://reviews.llvm.org/D104751

show more ...

[MLGO] Fix use of AM.invalidate post D100519

The ML inline advisors more aggressively invalidate certain analyses
after each call site inlining, to more accurately capture the problem
state.

Reland "[NPM][Inliner] Factor ImportedFunctionStats in the InlineAdvisor"

This reverts commit d97f776be5f8cd3cd446fe73827cd355f6bab4e1.

The original problem was due to build failures in shared lib

Reland "[NPM][Inliner] Factor ImportedFunctionStats in the InlineAdvisor"

This reverts commit d97f776be5f8cd3cd446fe73827cd355f6bab4e1.

The original problem was due to build failures in shared lib builds. D95079
moved ImportedFunctionsInliningStatistics under Analysis, unblocking
this.

show more ...

Revert "[NPM][Inliner] Factor ImportedFunctionStats in the InlineAdvisor"

This reverts commit e8aec763a57e211420dfceb2a8dc6b88574924f3.

[NPM][Inliner] Factor ImportedFunctionStats in the InlineAdvisor

When using 2 InlinePass instances in the same CGSCC - one for other
mandatory inlinings, the other for the heuristic-driven ones - th

[NPM][Inliner] Factor ImportedFunctionStats in the InlineAdvisor

When using 2 InlinePass instances in the same CGSCC - one for other
mandatory inlinings, the other for the heuristic-driven ones - the order
in which the ImportedFunctionStats would be output-ed would depend on
the destruction order of the inline passes, which is not deterministic.

This patch moves the ImportedFunctionStats responsibility to the
InlineAdvisor to address this problem.

Differential Revision: https://reviews.llvm.org/D94982

show more ...

[NewPM][Inliner] Move the 'always inliner' case in the same CGSCC pass as 'regular' inliner

Expanding from D94808 - we ensure the same InlineAdvisor is used by both
InlinerPass instances. The notion

[NewPM][Inliner] Move the 'always inliner' case in the same CGSCC pass as 'regular' inliner

Expanding from D94808 - we ensure the same InlineAdvisor is used by both
InlinerPass instances. The notion of mandatory inlining is moved into
the core InlineAdvisor: advisors anyway have to handle that case, so
this change also factors out that a bit better.

Differential Revision: https://reviews.llvm.org/D94825

show more ...

[llvm][inliner] Reuse the inliner pass to implement 'always inliner'

Enable performing mandatory inlinings upfront, by reusing the same logic
as the full inliner, instead of the AlwaysInliner. This

[llvm][inliner] Reuse the inliner pass to implement 'always inliner'

Enable performing mandatory inlinings upfront, by reusing the same logic
as the full inliner, instead of the AlwaysInliner. This has the
following benefits:
- reduce code duplication - one inliner codebase
- open the opportunity to help the full inliner by performing additional
function passes after the mandatory inlinings, but before th full
inliner. Performing the mandatory inlinings first simplifies the problem
the full inliner needs to solve: less call sites, more contextualization, and,
depending on the additional function optimization passes run between the
2 inliners, higher accuracy of cost models / decision policies.

Note that this patch does not yet enable much in terms of post-always
inline function optimization.

Differential Revision: https://reviews.llvm.org/D91567

show more ...

Reapply "Rename InlineFeatureAnalysis to FunctionPropertiesAnalysis"

(This reverts commit a5e0194709c40212694370e0ea789a1ca14548b5, and
corrects author).

Rename the pass to be able to extend it to

Reapply "Rename InlineFeatureAnalysis to FunctionPropertiesAnalysis"

(This reverts commit a5e0194709c40212694370e0ea789a1ca14548b5, and
corrects author).

Rename the pass to be able to extend it to function properties other than inliner features.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D82044

show more ...