[NFC] FunctionSamples::getEntrySamples -> getHeadSamplesEstimate

The name `getEntrySamples` was misleading for 2 reasons. One, it's
close in name to `Function::getEntryCount`, but the equivalent her

[NFC] FunctionSamples::getEntrySamples -> getHeadSamplesEstimate

The name `getEntrySamples` was misleading for 2 reasons. One, it's
close in name to `Function::getEntryCount`, but the equivalent here is
`getHeadSamples`; second, as opposed to the other get* APIs in
`FunctionSamples`, it performs an estimate/heuristic rather than just
retrieving raw data (or a non-heuristic derivate off that data, like
`getMaxCountInside`)

The new name should more clearly communicate its intent; and, being
close (in name) to `getHeadSamples`, it should allow the reader discover
the relation between them.

Also updated the doc comments for both `getHeadSamples[Estimate]` so a
reader may better understand the relation between them.

Differential Revision: https://reviews.llvm.org/D130281

show more ...

[LegacyPM] Remove SampleProfileLoaderLegacyPass

Following recent changes removing non-core features of the legacy
PM/optimization pipeline (e.g. PGO), remove SamplePGO.

[llvm] Use value instead of getValue (NFC)

[CSSPGO][llvm-profgen] Reimplement SampleContextTracker using context trie

This is the followup patch to https://reviews.llvm.org/D125246 for the `SampleContextTracker` part. Before the promotion an

[CSSPGO][llvm-profgen] Reimplement SampleContextTracker using context trie

This is the followup patch to https://reviews.llvm.org/D125246 for the `SampleContextTracker` part. Before the promotion and merging of the context is based on the SampleContext(the array of frame), this causes a lot of cost to the memory. This patch detaches the tracker from using the array ref instead to use the context trie itself. This can save a lot of memory usage and benefit both the compiler's CS inliner and llvm-profgen's pre-inliner.

One structure needs to be specially treated is the `FuncToCtxtProfiles`, this is used to get all the functionSamples for one function to do the merging and promoting. Before it search each functions' context and traverse the trie to get the node of the context. Now we don't have the context inside the profile, instead we directly use an auxiliary map `ProfileToNodeMap` for profile , it initialize to create the FunctionSamples to TrieNode relations and keep updating it during promoting and merging the node.

Moreover, I was expecting the results before and after remain the same, but I found that the order of FuncToCtxtProfiles matter and affect the results. This can happen on recursive context case, but the difference should be small. Now we don't have the context, so I just used a vector for the order, the result is still deterministic.

Measured on one huge size(12GB) profile from one of our internal service. The profile similarity difference is 99.999%, and the running time is improved by 3X(debug mode) and the memory is reduced from 170GB to 90GB.

Reviewed By: hoy, wenlei

Differential Revision: https://reviews.llvm.org/D127031

show more ...

Revert "Don't use Optional::hasValue (NFC)"

This reverts commit aa8feeefd3ac6c78ee8f67bf033976fc7d68bc6d.

Don't use Optional::hasValue (NFC)

[Inline] Annotate inline pass name with link phase information for analysis.

The annotation is flag gated; flag is turned off by default.

Differential Revision: https://reviews.llvm.org/D125495

[SampleProfile][Inline] Annotate sample profile inline remarks with link phase (prelink/postlink) information.

Differential Revision: https://reviews.llvm.org/D126833

[llvm] Don't use Optional::hasValue (NFC)

Remove unneeded cl::ZeroOrMore for cl::opt/cl::list options

[llvm] Remove unneeded cl::ZeroOrMore for cl::opt options. NFC

Some cl::ZeroOrMore were added to avoid the `may only occur zero or one times!`
error. More were added due to cargo cult. Since the err

[llvm] Remove unneeded cl::ZeroOrMore for cl::opt options. NFC

Some cl::ZeroOrMore were added to avoid the `may only occur zero or one times!`
error. More were added due to cargo cult. Since the error has been removed,
cl::ZeroOrMore is unneeded.

Also remove cl::init(false) while touching the lines.

show more ...

[iwyu] Handle regressions in libLLVM header include

Running iwyu-diff on LLVM codebase since fa5a4e1b95c8f37796 detected a few
regressions, fixing them.

Differential Revision: https://reviews.llvm.

[iwyu] Handle regressions in libLLVM header include

Running iwyu-diff on LLVM codebase since fa5a4e1b95c8f37796 detected a few
regressions, fixing them.

Differential Revision: https://reviews.llvm.org/D124847

show more ...

[CSSPGO] Relax size limitation for priority inlining with preinlined profile

As a follow-up to D124632, I'm turning on unlimited size caps for inlining with preinlined profile. It should be safe as

[CSSPGO] Relax size limitation for priority inlining with preinlined profile

As a follow-up to D124632, I'm turning on unlimited size caps for inlining with preinlined profile. It should be safe as a preinlined profile has "bounded" inline contexts.

No noticeable size or perf delta was seen with two of our internal large services, but I think this is still a good change to be consistent with the other case.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D124793

show more ...

[CSSPGO][NFC] Replace SampleProfileLoader::ProfileIsCS with FunctionSamples::ProfileIsCS.

The two fields have the same meaning. Their values come from the reader. Therefore I'm removing one.

Review

[CSSPGO][NFC] Replace SampleProfileLoader::ProfileIsCS with FunctionSamples::ProfileIsCS.

The two fields have the same meaning. Their values come from the reader. Therefore I'm removing one.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D124788

show more ...

[CSSPGO] Turn on priority inlining for probe-only profile

We have seen that the prioirty inliner delivered on-par performance with the old inliner for probe-only CSSPGO profile, as long as without a

[CSSPGO] Turn on priority inlining for probe-only profile

We have seen that the prioirty inliner delivered on-par performance with the old inliner for probe-only CSSPGO profile, as long as without a size budget. I'm turning on the priority inliner for probe-only profile by default.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D124632

show more ...

[CSSPGO] Rename ProfileIsCSNested and ProfileIsCSFlat

To be more clear and definitive, I'm renaming `ProfileIsCSFlat` back to `ProfileIsCS` which stands for full context-sensitive flat profiles. `P

[CSSPGO] Rename ProfileIsCSNested and ProfileIsCSFlat

To be more clear and definitive, I'm renaming `ProfileIsCSFlat` back to `ProfileIsCS` which stands for full context-sensitive flat profiles. `ProfileIsCSNested` is now renamed to `ProfileIsPreInlined` and is extended to be applicable for CS flat profiles too. More specifically, `ProfileIsPreInlined` is for any kind of profiles (flat or nested) that contain 'ShouldBeInlined' contexts. The flag is encoded in the profile summary section for extbinary profiles and is computed on-the-fly for text profiles.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D122602

show more ...

[llvm][misexpect] Avoid division by 0 when using sample profiling

MisExpect diagnostics should not prevent compilation from succeeding, and the
assertion is insufficient to prevent division by zero

[llvm][misexpect] Avoid division by 0 when using sample profiling

MisExpect diagnostics should not prevent compilation from succeeding, and the
assertion is insufficient to prevent division by zero in release builds.

This patch addresses that by replacing the assert with an early return.

Additionally, it disables MisExpect diagnostics when using sample profiling,
since this is the only known case where this error has manifested.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D124302

show more ...

[misexpect] Re-implement MisExpect Diagnostics

Reimplements MisExpect diagnostics from D66324 to reconstruct its
original checking methodology only using MD_prof branch_weights
metadata.

New checks

[misexpect] Re-implement MisExpect Diagnostics

Reimplements MisExpect diagnostics from D66324 to reconstruct its
original checking methodology only using MD_prof branch_weights
metadata.

New checks rely on 2 invariants:

1) For frontend instrumentation, MD_prof branch_weights will always be
populated before llvm.expect intrinsics are lowered.

2) for IR and sample profiling, llvm.expect intrinsics will always be
lowered before branch_weights are populated from the IR profiles.

These invariants allow the checking to assume how the existing branch
weights are populated depending on the profiling method used, and emit
the correct diagnostics. If these invariants are ever invalidated, the
MisExpect related checks would need to be updated, potentially by
re-introducing MD_misexpect metadata, and ensuring it always will be
transformed the same way as branch_weights in other optimization passes.

Frontend based profiling is now enabled without using LLVM Args, by
introducing a new CodeGen option, and checking if the -Wmisexpect flag
has been passed on the command line.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D115907

show more ...

Revert "[misexpect] Re-implement MisExpect Diagnostics"

This reverts commit 46774df307159444d65083c2fd82f8574f0ab1d9.

[misexpect] Re-implement MisExpect Diagnostics

Reimplements MisExpect diagnostics from D66324 to reconstruct its
original checking methodology only using MD_prof branch_weights
metadata.

New checks

[misexpect] Re-implement MisExpect Diagnostics

Reimplements MisExpect diagnostics from D66324 to reconstruct its
original checking methodology only using MD_prof branch_weights
metadata.

New checks rely on 2 invariants:

1) For frontend instrumentation, MD_prof branch_weights will always be
populated before llvm.expect intrinsics are lowered.

2) for IR and sample profiling, llvm.expect intrinsics will always be
lowered before branch_weights are populated from the IR profiles.

These invariants allow the checking to assume how the existing branch
weights are populated depending on the profiling method used, and emit
the correct diagnostics. If these invariants are ever invalidated, the
MisExpect related checks would need to be updated, potentially by
re-introducing MD_misexpect metadata, and ensuring it always will be
transformed the same way as branch_weights in other optimization passes.

Frontend based profiling is now enabled without using LLVM Args, by
introducing a new CodeGen option, and checking if the -Wmisexpect flag
has been passed on the command line.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D115907

show more ...

Revert "[misexpect] Re-implement MisExpect Diagnostics"

This reverts commit 2add3fbd976d7b80a3a7fc14ef0deb9b1ca6beee.

[misexpect] Re-implement MisExpect Diagnostics

Reimplements MisExpect diagnostics from D66324 to reconstruct its
original checking methodology only using MD_prof branch_weights
metadata.

New checks

[misexpect] Re-implement MisExpect Diagnostics

Reimplements MisExpect diagnostics from D66324 to reconstruct its
original checking methodology only using MD_prof branch_weights
metadata.

New checks rely on 2 invariants:

1) For frontend instrumentation, MD_prof branch_weights will always be
populated before llvm.expect intrinsics are lowered.

2) for IR and sample profiling, llvm.expect intrinsics will always be
lowered before branch_weights are populated from the IR profiles.

These invariants allow the checking to assume how the existing branch
weights are populated depending on the profiling method used, and emit
the correct diagnostics. If these invariants are ever invalidated, the
MisExpect related checks would need to be updated, potentially by
re-introducing MD_misexpect metadata, and ensuring it always will be
transformed the same way as branch_weights in other optimization passes.

Frontend based profiling is now enabled without using LLVM Args, by
introducing a new CodeGen option, and checking if the -Wmisexpect flag
has been passed on the command line.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D115907

show more ...

[CSSPGO] Turn on profi and ext-tsp when using probe-based profile.

Probe-based profile leads to a better performance when combined with profi and ext-tsp block layout. I'm turning them on by default

[CSSPGO] Turn on profi and ext-tsp when using probe-based profile.

Probe-based profile leads to a better performance when combined with profi and ext-tsp block layout. I'm turning them on by default.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D122442

show more ...

[ProfSampleLoader] When disable-sample-loader-inlining is true, merge profiles of inlined instances to outlining versions.

When --disable-sample-loader-inlining is true, skip inline transformation,

[ProfSampleLoader] When disable-sample-loader-inlining is true, merge profiles of inlined instances to outlining versions.

When --disable-sample-loader-inlining is true, skip inline transformation, but merge profiles of inlined instances to outlining versions.

Differential Revision: https://reviews.llvm.org/D121862

show more ...

Cleanup includes: Transforms/IPO

Preprocessor output diff: -238205 lines
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.ll

Cleanup includes: Transforms/IPO

Preprocessor output diff: -238205 lines
Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D122183

show more ...