[memprof] Deduplicate and outline frame storage in the memprof profile.

The current implementation of memprof information in the indexed profile
format stores the representation of each calling cont

[memprof] Deduplicate and outline frame storage in the memprof profile.

The current implementation of memprof information in the indexed profile
format stores the representation of each calling context fram inline.
This patch uses an interned representation where the frame contents are
stored in a separate on-disk hash table. The table is indexed via a hash
of the contents of the frame. With this patch, the compressed size of a
large memprof profile reduces by ~22%.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D123094

show more ...

Reland "[memprof] Store callsite metadata with memprof records."

This reverts commit f4b794427e8037a4e952cacdfe7201e961f31a6f.

Reland with underlying msan issue fixed in D122260.

Revert "[memprof] Store callsite metadata with memprof records."

This reverts commit 0d362c90d335509c57c0fbd01ae1829e2b9c3765.

Reason: Causes the MSan buildbot to fail (see comments on
https://revi

Revert "[memprof] Store callsite metadata with memprof records."

This reverts commit 0d362c90d335509c57c0fbd01ae1829e2b9c3765.

Reason: Causes the MSan buildbot to fail (see comments on
https://reviews.llvm.org/D121179 for more information

show more ...

[memprof] Store callsite metadata with memprof records.

To ease profile annotation, each of the callsites in a function can be
annotated with profile data - "IR metadata format for MemProf" [1]. Thi

[memprof] Store callsite metadata with memprof records.

To ease profile annotation, each of the callsites in a function can be
annotated with profile data - "IR metadata format for MemProf" [1]. This
patch extends the on-disk serialized record format to store the debug
information for allocation callsites incl inline frames. This change is
incompatible with the existing format i.e. indexed profiles must be
regenerated, raw profiles are unaffected.

[1] https://groups.google.com/g/llvm-dev/c/aWHsdMxKAfE/m/WtEmRqyhAgAJ

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D121179

show more ...

Cleanup includes: ProfileData

Estimation of the impact on preprocessor output:

before: 1067349756
after: 1065940348

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-clea

Cleanup includes: ProfileData

Estimation of the impact on preprocessor output:

before: 1067349756
after: 1065940348

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120434

show more ...

[instrprof] Rename the profile kind types to be more descriptive.

Based on the discussion in D115393, I've updated the names to be more
descriptive.

Reviewed By: ellis, MaskRay

Differential Revisi

[instrprof] Rename the profile kind types to be more descriptive.

Based on the discussion in D115393, I've updated the names to be more
descriptive.

Reviewed By: ellis, MaskRay

Differential Revision: https://reviews.llvm.org/D120092

show more ...

Reland "[memprof] Extend the index prof format to include memory profiles."

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profi

Reland "[memprof] Extend the index prof format to include memory profiles."

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header adds a new
field which points to the offset of the memory profile section (if
present) in the indexed profile. For users who do not utilize this
feature the only overhead is a 64-bit offset in the header.

The memory profile section contains (1) profile metadata describing the
information recorded for each entry (2) an on-disk hashtable containing
the profile records indexed via llvm::md5(function_name). We chose to
introduce a separate hash table instead of the existing one since the
indexing for the instrumented fdo hash table is based on a CFG hash
which itself is perturbed by memprof instrumentation.

This commit also includes the changes reviewed separately in D120093.

Differential Revision: https://reviews.llvm.org/D120103

show more ...

Revert "Reland "[memprof] Extend the index prof format to include memory profiles.""

This reverts commit 807ba7aace188ada83ddb4477265728e97346af1.

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 85355a560a33897453df2ef959e255ee725eebce.

This patch adds support for optional memory profile informa

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 85355a560a33897453df2ef959e255ee725eebce.

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header adds a new
field which points to the offset of the memory profile section (if
present) in the indexed profile. For users who do not utilize this
feature the only overhead is a 64-bit offset in the header.

The memory profile section contains (1) profile metadata describing the
information recorded for each entry (2) an on-disk hashtable containing
the profile records indexed via llvm::md5(function_name). We chose to
introduce a separate hash table instead of the existing one since the
indexing for the instrumented fdo hash table is based on a CFG hash
which itself is perturbed by memprof instrumentation.

Differential Revision: https://reviews.llvm.org/D118653

show more ...

Revert "Reland "[memprof] Extend the index prof format to include memory profiles.""

This reverts commit de54e4ab78ef09b60f870e8df6f8a87e56d6bd94 [1/4]

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 0f73fb18ca333e38cdb9ffa701a8db026c56041d.

Use llvm/Profile/MIBEntryDef.inc instead of relative path.

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 0f73fb18ca333e38cdb9ffa701a8db026c56041d.

Use llvm/Profile/MIBEntryDef.inc instead of relative path.

Generated the raw profile data with `-mllvm
-enable-name-compression=false` so that builbots where the reader is
built without zlib do not fail.

Also updated the test build instructions.

show more ...

Revert "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 43c2348c5b926df6bdbc5b70efaa35ecdefe12d5.

Buildbots are failing with an error on reading memprof test

Revert "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 43c2348c5b926df6bdbc5b70efaa35ecdefe12d5.

Buildbots are failing with an error on reading memprof testdata.
"Inputs/basic.profraw: profile uses zlib
compression but the profile reader was built without zlib support"

https://lab.llvm.org/buildbot/#/builders/16/builds/24490

show more ...

[memprof] Extend the index prof format to include memory profiles.

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header

[memprof] Extend the index prof format to include memory profiles.

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header adds a new
field which points to the offset of the memory profile section (if
present) in the indexed profile. For users who do not utilize this
feature the only overhead is a 64-bit offset in the header.

The memory profile section contains (1) profile metadata describing the
information recorded for each entry (2) an on-disk hashtable containing
the profile records indexed via llvm::md5(function_name). We chose to
introduce a separate hash table instead of the existing one since the
indexing for the instrumented fdo hash table is based on a CFG hash
which itself is perturbed by memprof instrumentation.

Differential Revision: https://reviews.llvm.org/D118653

show more ...

[InstrProf][NFC] Remove stray option in InstrProfWriter

This variable was added to `InstrProfWriter.cpp` in D115693 by mistake and it isn't needed.

Reviewed By: kyulee

Differential Revision: http

[InstrProf][NFC] Remove stray option in InstrProfWriter

This variable was added to `InstrProfWriter.cpp` in D115693 by mistake and it isn't needed.

Reviewed By: kyulee

Differential Revision: https://reviews.llvm.org/D118664

show more ...

[InstrProf] Add single byte coverage mode

Use the llvm flag `-pgo-function-entry-coverage` to create single byte "counters" to track functions coverage. This mode has significantly less size overhea

[InstrProf] Add single byte coverage mode

Use the llvm flag `-pgo-function-entry-coverage` to create single byte "counters" to track functions coverage. This mode has significantly less size overhead in both code and data because
* We mark a function as "covered" with a store instead of an increment which generally requires fewer assembly instructions
* We use a single byte per function rather than 8 bytes per block

The trade off of course is that this mode only tells you if a function has been covered. This is useful, for example, to detect dead code.

When combined with debug info correlation [0] we are able to create an instrumented Clang binary that is only 150M (the vanilla Clang binary is 143M). That is an overhead of 7M (4.9%) compared to the default instrumentation (without value profiling) which has an overhead of 31M (21.7%).

[0] https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4

Reviewed By: kyulee

Differential Revision: https://reviews.llvm.org/D116180

show more ...

[InstrProf][NFC] Refactor Profile kind into a bitset enum.

This change refactors the ProfileKind enum into a bitset enum to
represent the different attributes a profile can have. This change
simplif

[InstrProf][NFC] Refactor Profile kind into a bitset enum.

This change refactors the ProfileKind enum into a bitset enum to
represent the different attributes a profile can have. This change
simplifies the logic in the instrprof writer when multiple profiles are
merged together. In the future we plan on introducing a new memory
profile section which will extend the enum by one additional entry.
Without this change when accounting for memory profiles will have to be
maintained separately and will make the logic more complex.

Differential Revision: https://reviews.llvm.org/D115393

show more ...

[Try2][InstrProf] Attach debug info to counters

Add the llvm flag `-debug-info-correlate` to attach debug info to instrumentation counters so we can correlate raw profile data to their functions. Ra

[Try2][InstrProf] Attach debug info to counters

Add the llvm flag `-debug-info-correlate` to attach debug info to instrumentation counters so we can correlate raw profile data to their functions. Raw profiles are dumped as `.proflite` files. The next diff enables `llvm-profdata` to consume `.proflite` and debug info files to produce a normal `.profdata` profile.

Part of the "lightweight instrumentation" work: https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4

The original diff https://reviews.llvm.org/D114565 was reverted because of the `Instrumentation/InstrProfiling/debug-info-correlate.ll` test, which is fixed in this commit.

Reviewed By: kyulee

Differential Revision: https://reviews.llvm.org/D115693

show more ...

Revert "[InstrProf] Attach debug info to counters"

This reverts commit 800bf8ed29fbcaa9436540e83bc119ec92e7d40f.

The `Instrumentation/InstrProfiling/debug-info-correlate.ll` test was
failing becaus

Revert "[InstrProf] Attach debug info to counters"

This reverts commit 800bf8ed29fbcaa9436540e83bc119ec92e7d40f.

The `Instrumentation/InstrProfiling/debug-info-correlate.ll` test was
failing because I forgot the `llc` commands are architecture specific.
I'll follow up with a fix.

Differential Revision: https://reviews.llvm.org/D115689

show more ...

[InstrProf] Attach debug info to counters

Add the llvm flag `-debug-info-correlate` to attach debug info to instrumentation counters so we can correlate raw profile data to their functions. Raw prof

[InstrProf] Attach debug info to counters

Add the llvm flag `-debug-info-correlate` to attach debug info to instrumentation counters so we can correlate raw profile data to their functions. Raw profiles are dumped as `.proflite` files. The next diff enables `llvm-profdata` to consume `.proflite` and debug info files to produce a normal `.profdata` profile.

Part of the "lightweight instrumentation" work: https://groups.google.com/g/llvm-dev/c/r03Z6JoN7d4

Reviewed By: kyulee

Differential Revision: https://reviews.llvm.org/D114565

show more ...

[NFC] ProfileSummary: const most of the fields.

This simplifies readability / maintainability.

[llvm] Use StringRef::contains (NFC)

[llvm-profdata] Emit Error when Invalid MemOpSize Section is Created by llvm-profdata

Under certain (currently unknown) conditions, llvm-profdata is outputting
profiles that have two consecutive ent

[llvm-profdata] Emit Error when Invalid MemOpSize Section is Created by llvm-profdata

Under certain (currently unknown) conditions, llvm-profdata is outputting
profiles that have two consecutive entries in the MemOPSize section for the
value 0. This causes the PGOMemOPSizeOpt pass to output an invalid switch
instruction with two cases for 0. As mentioned, we’re not quite sure what’s
causing this to happen, but this patch prevents llvm-profdata from outputting a
profile that has this problem and gives an error with a request for a
reproducible.

Differential Revision: https://reviews.llvm.org/D92074

show more ...

Supplement instr profile with sample profile.

PGO profile is usually more precise than sample profile. However, PGO profile
needs to be collected from loadtest and loadtest may not be representative

Supplement instr profile with sample profile.

PGO profile is usually more precise than sample profile. However, PGO profile
needs to be collected from loadtest and loadtest may not be representative
enough to the production workload. Sample profile collected from production
can be used as a supplement -- for functions cold in loadtest but warm/hot
in production, we can scale up the related function in PGO profile if the
function is warm or hot in sample profile.

The implementation contains changes in compiler side and llvm-profdata side.
Given an instr profile and a sample profile, for a function cold in PGO
profile but warm/hot in sample profile, llvm-profdata will either mark
all the counters in the profile to be -1 or scale up the max count in the
function to be above hot threshold, depending on the zero counter ratio in
the profile. The assumption is if there are too many counters being zero
in the function profile, the profile is more likely to cause harm than good,
then llvm-profdata will mark all the counters to be -1 indicating the
function is hot but the profile is unaccountable. In compiler side, if a
function profile with all -1 counters is seen, the function entry count will
be set to be above hot threshold but its internal profile will be dropped.

In the long run, it may be useful to let compiler support using PGO profile
and sample profile at the same time, but that requires more careful design
and more substantial changes to make two profiles work seamlessly. The patch
here serves as a simple intermediate solution.

Differential Revision: https://reviews.llvm.org/D81981

show more ...

[PGO] Supporting code for always instrumenting entry block

This patch includes the supporting code that enables always
instrumenting the function entry block by default.

This patch will NOT the def

[PGO] Supporting code for always instrumenting entry block

This patch includes the supporting code that enables always
instrumenting the function entry block by default.

This patch will NOT the default behavior.

It adds a variant bit in the profile version, adds new directives in
text profile format, and changes llvm-profdata tool accordingly.

This patch is a split of D83024 (https://reviews.llvm.org/D83024)
Many test changes from D83024 are also included.

Differential Revision: https://reviews.llvm.org/D84261

show more ...

[PGO] Fix typos from r359612. NFC.

llvm-svn: 373369