[Inline] Annotate inline pass name with link phase information for analysis.

The annotation is flag gated; flag is turned off by default.

Differential Revision: https://reviews.llvm.org/D125495

[llvm] Use value_or instead of getValueOr (NFC)

[mlgo] Fix accounting for SCC splits

Previously if the inliner split an SCC such that an empty one remained, the MLInlineAdvisor could potentially lose track of the EdgeCount if a subsequent CGSCC p

[mlgo] Fix accounting for SCC splits

Previously if the inliner split an SCC such that an empty one remained, the MLInlineAdvisor could potentially lose track of the EdgeCount if a subsequent CGSCC pass modified the calls of a function that was initially in the SCC pre-split. Saving the seen nodes in onPassEntry resolves this.

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D127693

show more ...

[inliner] Add per-SCC-pass InlineAdvisor printing option

Adds option to print the contents of the Inline Advisor after each SCC Inliner pass

Reviewed By: mtrofin

Differential Revision: https://rev

[inliner] Add per-SCC-pass InlineAdvisor printing option

Adds option to print the contents of the Inline Advisor after each SCC Inliner pass

Reviewed By: mtrofin

Differential Revision: https://reviews.llvm.org/D127689

show more ...

[llvm][Inline] Remove PriorityInlineOrder in SCC inliner

Since the size of most of SCC's is 1, the PriorityInlineOrder would not change the inline
order in SCC inliner.

Reviewed By: kazu

Different

[llvm][Inline] Remove PriorityInlineOrder in SCC inliner

Since the size of most of SCC's is 1, the PriorityInlineOrder would not change the inline
order in SCC inliner.

Reviewed By: kazu

Differential Revision: https://reviews.llvm.org/D123608

show more ...

Cleanup includes: Transforms/IPO

Preprocessor output diff: -238205 lines
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.ll

Cleanup includes: Transforms/IPO

Preprocessor output diff: -238205 lines
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D122183

show more ...

[NewPM][Inliner] Make inlined calls to functions in same SCC as callee exponentially expensive

Introduce a new attribute "function-inline-cost-multiplier" which
multiplies the inline cost of a call

[NewPM][Inliner] Make inlined calls to functions in same SCC as callee exponentially expensive

Introduce a new attribute "function-inline-cost-multiplier" which
multiplies the inline cost of a call site (or all calls to a callee) by
the multiplier.

When processing the list of calls created by inlining, check each call
to see if the new call's callee is in the same SCC as the original
callee. If so, set the "function-inline-cost-multiplier" attribute of
the new call site to double the original call site's attribute value.
This does not happen when the original call site is intra-SCC.

This is an alternative to D120584, which marks the call sites as
noinline.

Hopefully fixes PR45253.

Reviewed By: davidxl

Differential Revision: https://reviews.llvm.org/D121084

show more ...

Cleanup includes: LLVMAnalysis

Number of lines output by preprocessor:
before: 1065940348
after: 1065307662

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Diff

Cleanup includes: LLVMAnalysis

Number of lines output by preprocessor:
before: 1065940348
after: 1065307662

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120659

show more ...

[mlgo][inline] Improve global state tracking

The global state refers to the number of the nodes currently in the
module, and the number of direct calls between nodes, across the
module.

Node counts

[mlgo][inline] Improve global state tracking

The global state refers to the number of the nodes currently in the
module, and the number of direct calls between nodes, across the
module.

Node counts are not a problem; edge counts are because we want strictly
the kind of edges that affect inlining (direct calls), and that is not
easily obtainable without iteration over the whole module.

This patch avoids relying on analysis invalidation because it turned out
to be too aggressive in some cases. It leverages the fact that Node
objects are stable - they do not get deleted while cgscc passes are
run over the module; and cgscc pass manager invariants.

Reviewed By: aeubanks

Differential Revision: https://reviews.llvm.org/D115847

show more ...

[Inline] Attempt to delete any discardable if unused functions

Previously we limited ourselves to only internal/private functions. We
can also delete linkonce_odr functions.

Minor compile time wins

[Inline] Attempt to delete any discardable if unused functions

Previously we limited ourselves to only internal/private functions. We
can also delete linkonce_odr functions.

Minor compile time wins:
https://llvm-compile-time-tracker.com/compare.php?from=d51e3474e060cb0e90dc2e2487f778b0d3e6a8de&to=bccffe3f8d5dd4dda884c9ac1f93e51772519cad&stat=instructions

Major memory wins on tramp3d:
https://llvm-compile-time-tracker.com/compare.php?from=d51e3474e060cb0e90dc2e2487f778b0d3e6a8de&to=bccffe3f8d5dd4dda884c9ac1f93e51772519cad&stat=max-rss

Relanding with fix for compile times D117236.

Reviewed By: nikic, mtrofin

Differential Revision: https://reviews.llvm.org/D115545

show more ...

[Inliner] Don't removeDeadConstantUsers() when checking if a function is dead

If a function has many uses, this can take a good chunk of compile times.

Reviewed By: nikic

Differential Revision: ht

[Inliner] Don't removeDeadConstantUsers() when checking if a function is dead

If a function has many uses, this can take a good chunk of compile times.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D117236

show more ...

Revert "[Inline] Attempt to delete any discardable if unused functions"

Somehow this ends up causing an infinite loop in the inliner.

This reverts commit d5be48c66d3e5e8be21805c3a33dc67a20e258be.

[Inline] Attempt to delete any discardable if unused functions

Previously we limited ourselves to only internal/private functions. We
can also delete linkonce_odr functions.

Minor compile time wins

[Inline] Attempt to delete any discardable if unused functions

Previously we limited ourselves to only internal/private functions. We
can also delete linkonce_odr functions.

Minor compile time wins:
https://llvm-compile-time-tracker.com/compare.php?from=d51e3474e060cb0e90dc2e2487f778b0d3e6a8de&to=bccffe3f8d5dd4dda884c9ac1f93e51772519cad&stat=instructions

Major memory wins on tramp3d:
https://llvm-compile-time-tracker.com/compare.php?from=d51e3474e060cb0e90dc2e2487f778b0d3e6a8de&to=bccffe3f8d5dd4dda884c9ac1f93e51772519cad&stat=max-rss

Reviewed By: nikic, mtrofin

Differential Revision: https://reviews.llvm.org/D115545

show more ...

[NFC][MLGO] Use LazyCallGraph::Node to track functions.

This avoids the InlineAdvisor carrying the responsibility of deleting
Function objects. We use LazyCallGraph::Node objects instead, which are

[NFC][MLGO] Use LazyCallGraph::Node to track functions.

This avoids the InlineAdvisor carrying the responsibility of deleting
Function objects. We use LazyCallGraph::Node objects instead, which are
stable in memory for the duration of the Module-wide performance of CGSCC
passes started under the same ModuleToPostOrderCGSCCPassAdaptor (which
is the case here)

Differential Revision: https://reviews.llvm.org/D116964

show more ...

[llvm] Remove redundant member initialization (NFC)

Identified with readability-redundant-member-init.

Revert "[Inline] Attempt to delete any discardable if unused functions"

This reverts commit 335a3163aab4ab3a52c7c15d03bcf1ce9ba79fbb.

Causes crashes when building llvm-test-suite's kc under Release

Revert "[Inline] Attempt to delete any discardable if unused functions"

This reverts commit 335a3163aab4ab3a52c7c15d03bcf1ce9ba79fbb.

Causes crashes when building llvm-test-suite's kc under ReleaseLTO-g.

show more ...

[Inline] Attempt to delete any discardable if unused functions

Previously we limited ourselves to only internal/private functions. We
can also delete linkonce_odr functions.

Minor compile time wins

[Inline] Attempt to delete any discardable if unused functions

Previously we limited ourselves to only internal/private functions. We
can also delete linkonce_odr functions.

Minor compile time wins:
https://llvm-compile-time-tracker.com/compare.php?from=d51e3474e060cb0e90dc2e2487f778b0d3e6a8de&to=bccffe3f8d5dd4dda884c9ac1f93e51772519cad&stat=instructions

Major memory wins on tramp3d:
https://llvm-compile-time-tracker.com/compare.php?from=d51e3474e060cb0e90dc2e2487f778b0d3e6a8de&to=bccffe3f8d5dd4dda884c9ac1f93e51772519cad&stat=max-rss

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D115545

show more ...

[ModuleUtils] Remove dead arg from filterDeadComdatFunctions() (NFC)

The module argument is no longer used.

Revert "[llvm] Remove redundant member initialization (NFC)"

This reverts commit fd4808887ee47f3ec8a030e9211169ef4fb094c3.

This patch causes gcc to issue a lot of warnings like:

warning: base cl

Revert "[llvm] Remove redundant member initialization (NFC)"

This reverts commit fd4808887ee47f3ec8a030e9211169ef4fb094c3.

This patch causes gcc to issue a lot of warnings like:

warning: base class ‘class llvm::MCParsedAsmOperand’ should be
explicitly initialized in the copy constructor [-Wextra]

show more ...

[llvm] Remove redundant member initialization (NFC)

Identified with readability-redundant-member-init.

[Inliner] Add debug message for history skip (NFC)

[NewPM] Add option to prevent rerunning function pipeline on functions in CGSCC adaptor

In a CGSCC pass manager, we may visit the same function multiple times
due to SCC mutations. In the inliner pi

[NewPM] Add option to prevent rerunning function pipeline on functions in CGSCC adaptor

In a CGSCC pass manager, we may visit the same function multiple times
due to SCC mutations. In the inliner pipeline, this results in running
the function simplification pipeline on a function multiple times even
if it hasn't been changed since the last function simplification
pipeline run.

We use a newly introduced analysis to keep track of whether or not a
function has changed since the last time the function simplification
pipeline has run on it. If we see this analysis available for a function
in a CGSCCToFunctionPassAdaptor, we skip running the function passes on
the function. The analysis is queried at the end of the function passes
so that it's available after the first time the function simplification
pipeline runs on a function. This is a per-adaptor option so it doesn't
apply to every adaptor.

The goal of this is to improve compile times. However, currently we
can't turn this on by default at least for the higher optimization
levels since the function simplification pipeline is not robust enough
to be idempotent in many cases, resulting in performance regressions if
we stop running the function simplification pipeline on a function
multiple times. We may be able to turn this on for -O1 in the near
future, but turning this on for higher optimization levels would require
more investment in the function simplification pipeline.

Heavily inspired by D98103.

Example compile time improvements with flag turned on:
https://llvm-compile-time-tracker.com/compare.php?from=998dc4a5d3491d2ae8cbe742d2e13bc1b0cacc5f&to=5c27c913687d3d5559ef3ab42b5a3d513531d61c&stat=instructions

Reviewed By: asbirlea, nikic

Differential Revision: https://reviews.llvm.org/D113947

show more ...

[NewPM] Only invalidate modified functions' analyses in CGSCC passes + turn on eagerly invalidate analyses

Previously, any change in any function in an SCC would cause all
analyses for all functions

[NewPM] Only invalidate modified functions' analyses in CGSCC passes + turn on eagerly invalidate analyses

Previously, any change in any function in an SCC would cause all
analyses for all functions in the SCC to be invalidated. With this
change, we now manually invalidate analyses for functions we modify,
then let the pass manager know that all function analyses should be
preserved since we've already handled function analysis invalidation.

So far this only touches the inliner, argpromotion, function-attrs, and
updateCGAndAnalysisManager(), since they are the most used.

This is part of an effort to investigate running the function
simplification pipeline less on functions we visit multiple times in the
inliner pipeline.

However, this causes major memory regressions especially on larger IR.
To counteract this, turn on the option to eagerly invalidate function
analyses. This invalidates analyses on functions immediately after
they're processed in a module or scc to function adaptor for specific
parts of the pipeline.

Within an SCC, if a pass only modifies one function, other functions in
the SCC do not have their analyses invalidated, so in later function
passes in the SCC pass manager the analyses may still be cached. It is
only after the function passes that the eager invalidation takes effect.
For the default pipelines this makes sense because the inliner pipeline
runs the function simplification pipeline after all other SCC passes
(except CoroSplit which doesn't request any analyses).

Overall this has mostly positive effects on compile time and positive effects on memory usage.
https://llvm-compile-time-tracker.com/compare.php?from=7f627596977624730f9298a1b69883af1555765e&to=39e824e0d3ca8a517502f13032dfa67304841c90&stat=instructions
https://llvm-compile-time-tracker.com/compare.php?from=7f627596977624730f9298a1b69883af1555765e&to=39e824e0d3ca8a517502f13032dfa67304841c90&stat=max-rss

D113196 shows that we slightly regressed compile times in exchange for
some memory improvements when turning on eager invalidation. D100917
shows that we slightly improved compile times in exchange for major
memory regressions in some cases when invalidating less in SCC passes.
Turning these on at the same time keeps the memory improvements while
keeping compile times neutral/slightly positive.

Reviewed By: asbirlea, nikic

Differential Revision: https://reviews.llvm.org/D113304

show more ...

[InlineAdvisor] Add fallback/format switches and negative remark processing to Replay Inliner

Adds the following switches:

1. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback:

[InlineAdvisor] Add fallback/format switches and negative remark processing to Replay Inliner

Adds the following switches:

1. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback: controls what the replay advisor does for inline sites that are not present in the replay. Options are:

1. Original: defers to original advisor
2. AlwaysInline: inline all sites not in replay
3. NeverInline: inline no sites not in replay

2. --sample-profile-inline-replay-format/--cgscc-inline-replay-format: controls what format should be generated to match against the replay remarks. Options are:

1. Line
2. LineColumn
3. LineDiscriminator
4. LineColumnDiscriminator

Adds support for negative inlining decisions. These are denoted by "will not be inlined into" as compared to the positive "inlined into" in the remarks.

All of these together with the previous `--sample-profile-inline-replay-scope/--cgscc-inline-replay-scope` allow tweaking in how to apply replay. In my testing, I'm using:
1. --sample-profile-inline-replay-scope/--cgscc-inline-replay-scope = Function to only replay on a function
2. --sample-profile-inline-replay-fallback/--cgscc-inline-replay-fallback = NeverInline since I'm feeding in only positive remarks to the replay system
3. --sample-profile-inline-replay-format/--cgscc-inline-replay-format = Line since I'm generating the remarks from DWARF information from GCC which can conflict quite heavily in column number compared to Clang

An alternative configuration could be to do Function, AlwaysInline, Line fallback with negative remarks which closer matches the final call-sites. Note that this can lead to unbounded inlining if a negative remark doesn't match/exist for one reason or another.

Updated various tests to cover the new switches and negative remarks

Testing:
ninja check-all

Reviewed By: wenlei, mtrofin

Differential Revision: https://reviews.llvm.org/D112040

show more ...

[InlineAdvisor] Add -inline-replay-scope=<Function|Module> to control replay scope

The goal is to allow grafting an inline tree from Clang or GCC into a new compilation without affecting other funct

[InlineAdvisor] Add -inline-replay-scope=<Function|Module> to control replay scope

The goal is to allow grafting an inline tree from Clang or GCC into a new compilation without affecting other functions. For GCC, we're doing this by extracting the inline tree from dwarf information and generating the equivalent remarks.

This allows easier side-by-side asm analysis and a trial way to see if a particular inlining setup provides benefits by itself.

Testing:
ninja check-all

Reviewed By: wenlei, mtrofin

Differential Revision: https://reviews.llvm.org/D110658

show more ...