[Support] Change compression::zlib::{compress,uncompress} to use uint8_t *

It's more natural to use uint8_t * (std::byte needs C++17 and llvm has
too much uint8_t *) and most callers use uint8_t * i

[Support] Change compression::zlib::{compress,uncompress} to use uint8_t *

It's more natural to use uint8_t * (std::byte needs C++17 and llvm has
too much uint8_t *) and most callers use uint8_t * instead of char *.
The functions are recently moved into `llvm::compression::zlib::`, so
downstream projects need to make adaption anyway.

show more ...

[NFC] Refactor llvm::zlib namespace

* Refactor compression namespaces across the project, making way for a possible
introduction of alternatives to zlib compression.
Changes are as follows:
*

[NFC] Refactor llvm::zlib namespace

* Refactor compression namespaces across the project, making way for a possible
introduction of alternatives to zlib compression.
Changes are as follows:
* Relocate the `llvm::zlib` namespace to `llvm::compression::zlib`.

Reviewed By: MaskRay, leonardchan, phosek

Differential Revision: https://reviews.llvm.org/D128953

show more ...

[CSSPGO] Rename ProfileIsCSNested and ProfileIsCSFlat

To be more clear and definitive, I'm renaming `ProfileIsCSFlat` back to `ProfileIsCS` which stands for full context-sensitive flat profiles. `P

[CSSPGO] Rename ProfileIsCSNested and ProfileIsCSFlat

To be more clear and definitive, I'm renaming `ProfileIsCSFlat` back to `ProfileIsCS` which stands for full context-sensitive flat profiles. `ProfileIsCSNested` is now renamed to `ProfileIsPreInlined` and is extended to be applicable for CS flat profiles too. More specifically, `ProfileIsPreInlined` is for any kind of profiles (flat or nested) that contain 'ShouldBeInlined' contexts. The flag is encoded in the profile summary section for extbinary profiles and is computed on-the-fly for text profiles.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D122602

show more ...

Apply clang-tidy fixes for readability-redundant-smartptr-get in SampleProfReader.cpp (NFC)

Cleanup includes: ProfileData

Estimation of the impact on preprocessor output:

before: 1067349756
after: 1065940348

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-clea

Cleanup includes: ProfileData

Estimation of the impact on preprocessor output:

before: 1067349756
after: 1065940348

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120434

show more ...

[ProfileData] Remove unused and racy FunctionSamples::Format after D51643

The write may be racy if ThinLTO creates multiple `InProcessThinBackend` instances.

[CSSPGO] Do not merge a context that is already duplicated into the base profile.

Do not merge a context that is already duplicated into the base profile.

Also fixing a typo caused by previous refa

[CSSPGO] Do not merge a context that is already duplicated into the base profile.

Do not merge a context that is already duplicated into the base profile.

Also fixing a typo caused by previous refactoring.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D119735

show more ...

[CSSPGO] Print "context-nested" instead of "preilnined" for ProfileSummarySection.

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D117141

[CSSPGO] Use nested context-sensitive profile.

CSSPGO currently employs a flat profile format for context-sensitive profiles. Such a flat profile allows for precisely manipulating contexts that is e

[CSSPGO] Use nested context-sensitive profile.

CSSPGO currently employs a flat profile format for context-sensitive profiles. Such a flat profile allows for precisely manipulating contexts that is either inlined or not inlined. This is a benefit over the nested profile format used by non-CS AutoFDO. A downside of this is the longer build time due to parsing the indexing the full CS contexts.

For a CS flat profile, though only the context profiles relevant to a module are loaded when that module is compiled, the cost to figure out what profiles are relevant is noticeably high when there're many contexts, since the sample reader will need to scan all context strings anyway. On the contrary, a nested function profile has its related inline subcontexts isolated from other unrelated contexts. Therefore when compiling a set of functions, unrelated contexts will never need to be scanned.

In this change we are exploring using nested profile format for CSSPGO. This is expected to work based on an assumption that with a preinliner-computed profile all contexts are precomputed and expected to be inlined by the compiler. Contexts not expected to be inlined will be cut off and returned to corresponding base profiles (for top-level outlined functions). This naturally forms a nested profile where all nested contexts are expected to be inlined. The compiler will less likely optimize on derived contexts that are not precomputed.

A CS-nested profile will look exactly the same with regular nested profile except that each nested profile can come with an attributes. With pseudo probes, a nested profile shown as below can also have a CFG checksum.

```

main:1968679:12
2: 24
3: 28 _Z5funcAi:18
3.1: 28 _Z5funcBi:30
3: _Z5funcAi:1467398
0: 10
1: 10 _Z8funcLeafi:11
3: 24
1: _Z8funcLeafi:1467299
0: 6
1: 6
3: 287884
4: 287864 _Z3fibi:315608
15: 23
!CFGChecksum: 138828622701
!Attributes: 2
!CFGChecksum: 281479271677951
!Attributes: 2
```

Specific work included in this change:
- A recursive profile converter to convert CS flat profile to nested profile.
- Extend function checksum and attribute metadata to be stored in nested way for text profile and extbinary profile.
- Unifiy sample loader inliner path for CS and preinlined nested profile.
- Changes in the sample loader to support probe-based nested profile.

I've seen promising results regarding build time. A nested profile can result in a 20% shorter build time than a CS flat profile while keep an on-par performance. This is with -duplicate-contexts-into-base=1.

Test Plan:

Reviewed By: wenlei

Differential Revision: https://reviews.llvm.org/D115205

show more ...

[LLVM][NFC]Inclusive language: remove occurances of sanity check/test from llvm

Part of work to use more inclusive language in clang/llvm. Rewording
some comments and change function and variable na

[LLVM][NFC]Inclusive language: remove occurances of sanity check/test from llvm

Part of work to use more inclusive language in clang/llvm. Rewording
some comments and change function and variable names.

show more ...

[llvm-profdata] Print out section flags for FunctionMetadata section

As titled.

Reviewed By: wenlei, wlei

Differential Revision: https://reviews.llvm.org/D113064

Fix Wdocumentation unknown parameter warning. NFCI.

[CSSPGO] Sort function offset table to speed up profile loading.

With the context split work, the context-based (an array of strings) sorting performed at profile load time is way more expansive tha

[CSSPGO] Sort function offset table to speed up profile loading.

With the context split work, the context-based (an array of strings) sorting performed at profile load time is way more expansive than single-string-based sorting. This is likely due to auxiliary operations done on each array element, such as indirect references, std::min operations, also likely cache misses. In this change I'm presorting profiles during profile generation time to avoid sorting at compile time.

Compared to the previous context-split work, this effectively cuts down compile time by 20% for one of our large services and brings us closer to non-CS build, with still a small gap in build time.

Reviewed By: wenlei, wmi

Differential Revision: https://reviews.llvm.org/D109036

show more ...

[CSSPGO] Enable loading MD5 CS profile.

Adding the compiler support of MD5 CS profile based on pervious context split work D107299. A MD5 CS profile is about 40% smaller than the string-based extbin

[CSSPGO] Enable loading MD5 CS profile.

Adding the compiler support of MD5 CS profile based on pervious context split work D107299. A MD5 CS profile is about 40% smaller than the string-based extbinary profile. As a result, the compilation is 15% faster.

There are a few conversion from real names to md5 names that have been made on the sample loader and context tracker side to get it work.

Reviewed By: wenlei, wmi

Differential Revision: https://reviews.llvm.org/D108342

show more ...

[CSSPGO] Split context string to deduplicate function name used in the context.

Currently context strings contain a lot of duplicated function names and that significantly increase the profile size.

[CSSPGO] Split context string to deduplicate function name used in the context.

Currently context strings contain a lot of duplicated function names and that significantly increase the profile size. This change split the context into a series of {name, offset, discriminator} tuples so function names used in the context can be replaced by the index into the name table and that significantly reduce the size consumed by context.

A follow-up improvement made in the compiler and profiling tools is to avoid reconstructing full context strings which is time- and memory- consuming. Instead a context vector of `StringRef` is adopted to represent the full context in all scenarios. As a result, the previous prevalent profile map which was implemented as a `StringRef` is now engineered as an unordered map keyed by `SampleContext`. `SampleContext` is reshaped to using an `ArrayRef` to represent a full context for CS profile. For non-CS profile, it falls back to use `StringRef` to represent a contextless function name. Both the `ArrayRef` and `StringRef` objects are underpinned by real array and string objects that are stored in producer buffers. For compiler, they are maintained by the sample reader. For llvm-profgen, they are maintained in `ProfiledBinary` and `ProfileGenerator`. Full context strings can be generated only in those cases of debugging and printing.

When it comes to profile format, nothing has changed to the text format, though internally CS context is implemented as a vector. Extbinary format is only changed for CS profile, with an additional `SecCSNameTable` section which stores all full contexts logically in the form of `vector<int>`, which each element as an offset points to `SecNameTable`. All occurrences of contexts elsewhere are redirected to using the offset of `SecCSNameTable`.

Testing
This is no-diff change in terms of code quality and profile content (for text profile).

For our internal large service (aka ads), the profile generation is cut to half, with a 20x smaller string-based extbinary format generated.

The compile time of ads is dropped by 25%.

Differential Revision: https://reviews.llvm.org/D107299

show more ...

[CSSPGO] Add switch for sample loader to honor global pre-inliner decision from llvm-profgen

The change adds a switch to allow sample loader to use global pre-inliner's decision instead. The pre-inl

[CSSPGO] Add switch for sample loader to honor global pre-inliner decision from llvm-profgen

The change adds a switch to allow sample loader to use global pre-inliner's decision instead. The pre-inliner in llvm-profgen makes inline decision globally based on whole program profile and function byte size as cost proxy.

Since pre-inliner also adjusts/merges context profile based on its inline decision, honoring its inline decision in sample loader would lead to better post-inline profile quality especially for thinlto where cross module profile merging isn't possible without pre-inliner.

Minor fix in profile reader is also included. When pre-inliner is use, we now also turn off the default merging and trimming logic unless it's explicitly asked.

Differential Revision: https://reviews.llvm.org/D108677

show more ...

[SampleFDO] Set ProfileIsFS bit properly from the internal option

We have "-profile-isfs" internal option for text, binary, and
compactbinary format (mostly for debug and test purpose). We
need to s

[SampleFDO] Set ProfileIsFS bit properly from the internal option

We have "-profile-isfs" internal option for text, binary, and
compactbinary format (mostly for debug and test purpose). We
need to set the related flag in FunctionSamples so that ProfileIsFS
is written to the header in extbinary format.

Differential Revision: https://reviews.llvm.org/D108707

show more ...

[SamplePGO][NFC] Dump function profiles in order

Sample profiles are stored in a string map which is basically an unordered map. Printing out profiles by simply walking the string map doesn't enforc

[SamplePGO][NFC] Dump function profiles in order

Sample profiles are stored in a string map which is basically an unordered map. Printing out profiles by simply walking the string map doesn't enforce an order. I'm sorting the map in the decreasing order of total samples to enable a more stable dump, which is good for comparing two dumps.

Reviewed By: wenlei, wlei

Differential Revision: https://reviews.llvm.org/D108147

show more ...

Reapply commit b7425e956

The commit b7425e956: [NFC] fix typos
is harmless but was reverted by accident. Reapply.

Revert "[NFC] Fix typos"

This reverts commit b7425e956be60a73004d7ae5bb37da85872c29fb.

[NFC] Fix typos

s/senstive/senstive/g

[SampleFDO] New hierarchical discriminator for FS SampleFDO (llvm-profdata part)

This patch was split from https://reviews.llvm.org/D102246
[SampleFDO] New hierarchical discriminator for Flow Sensit

[SampleFDO] New hierarchical discriminator for FS SampleFDO (llvm-profdata part)

This patch was split from https://reviews.llvm.org/D102246
[SampleFDO] New hierarchical discriminator for Flow Sensitive SampleFDO
This is for llvm-profdata part of change. It sets the bit masks for the
profile reader in llvm-profdata. Also add an internal option
"-fs-discriminator-pass" for show and merge command to process the profile
offline.

This patch also moved setDiscriminatorMaskedBitFrom() to
SampleProfileReader::create() to simplify the interface.

Differential Revision: https://reviews.llvm.org/D103550

show more ...

[SampleFDO] New hierarchical discriminator for FS SampleFDO (ProfileData part)

This patch was split from https://reviews.llvm.org/D102246
[SampleFDO] New hierarchical discriminator for Flow Sensitiv

[SampleFDO] New hierarchical discriminator for FS SampleFDO (ProfileData part)

This patch was split from https://reviews.llvm.org/D102246
[SampleFDO] New hierarchical discriminator for Flow Sensitive SampleFDO
This is mainly for ProfileData part of change. It will load
FS Profile when such profile is detected. For an extbinary format profile,
create_llvm_prof tool will add a flag to profile summary section.
For other format profiles, the users need to use an internal option
(-profile-isfs) to tell the compiler that the profile uses FS discriminators.

This patch also simplified the bit API used by FS discriminators.

Differential Revision: https://reviews.llvm.org/D103041

show more ...

[SystemZ][z/OS] Add IsText Argument to GetFile and GetFileOrSTDIN

Add the `IsText` argument to `GetFile` and `GetFileOrSTDIN` which will help z/OS distinguish between text and binary correctly. This

[SystemZ][z/OS] Add IsText Argument to GetFile and GetFileOrSTDIN

Add the `IsText` argument to `GetFile` and `GetFileOrSTDIN` which will help z/OS distinguish between text and binary correctly. This is an extension to [this patch](https://reviews.llvm.org/D97785)

Reviewed By: abhina.sreeskantharajan, amccarth

Differential Revision: https://reviews.llvm.org/D100488

show more ...

[CSSPGO] Add attribute metadata for context profile

This changes adds attribute field for metadata of context profile. Currently we have an inline attribute that indicates whether the leaf frame cor

[CSSPGO] Add attribute metadata for context profile

This changes adds attribute field for metadata of context profile. Currently we have an inline attribute that indicates whether the leaf frame corresponding to a context profile was inlined in previous build.

This will be used to help estimating inlining and be taken into account when trimming context. Changes for that in llvm-profgen will follow. It will also help tuning.

Differential Revision: https://reviews.llvm.org/D98823

show more ...