[memprof] Return an error for unsupported symbolization.

Add a check to detect that the profiled binary was build with position
independent code. Add a test with a pie binary to which can be reused

[memprof] Return an error for unsupported symbolization.

Add a check to detect that the profiled binary was build with position
independent code. Add a test with a pie binary to which can be reused
later when support is added. Also clean up the error messages with
trailing colons.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D128564

show more ...

[ProfileData] Use llvm::erase_if (NFC)

[memprof] Print out the segment information in YAML format.

This change prints out the segment information in the raw profile in
YAML format for testing. Since we don't capture build ids yet, we pri

[memprof] Print out the segment information in YAML format.

This change prints out the segment information in the raw profile in
YAML format for testing. Since we don't capture build ids yet, we print
out <None> for now.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D126840

show more ...

[memprof] Update summary output.

Update the YAML format print out of the profile to include a summary
instead of displaying the headers in the raw file buffer. This allows us
to release the raw buff

[memprof] Update summary output.

Update the YAML format print out of the profile to include a summary
instead of displaying the headers in the raw file buffer. This allows us
to release the raw buffer early saving memory.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D126834

show more ...

[memprof] Keep and display symbol names in the RawMemProfReader.

Extend the Frame struct to hold the symbol name if requested
when a RawMemProfReader object is constructed. This change updates the
t

[memprof] Keep and display symbol names in the RawMemProfReader.

Extend the Frame struct to hold the symbol name if requested
when a RawMemProfReader object is constructed. This change updates the
tests and removes the need to pass --debug to obtain the mapping from
GUID to symbol names.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D126344

show more ...

[memprof] Deduplicate and outline frame storage in the memprof profile.

The current implementation of memprof information in the indexed profile
format stores the representation of each calling cont

[memprof] Deduplicate and outline frame storage in the memprof profile.

The current implementation of memprof information in the indexed profile
format stores the representation of each calling context fram inline.
This patch uses an interned representation where the frame contents are
stored in a separate on-disk hash table. The table is indexed via a hash
of the contents of the frame. With this patch, the compressed size of a
large memprof profile reduces by ~22%.

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D123094

show more ...

Cleanup includes: final pass

Cleanup a few extra files, this closes the work on libLLVM dependencies on my
side.

Impact on libLLVM preprocessed output: -35876 lines

Discourse thread: https://disco

Cleanup includes: final pass

Cleanup a few extra files, this closes the work on libLLVM dependencies on my
side.

Impact on libLLVM preprocessed output: -35876 lines

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D122576

show more ...

Reland "[memprof] Store callsite metadata with memprof records."

This reverts commit f4b794427e8037a4e952cacdfe7201e961f31a6f.

Reland with underlying msan issue fixed in D122260.

Revert "[memprof] Store callsite metadata with memprof records."

This reverts commit 0d362c90d335509c57c0fbd01ae1829e2b9c3765.

Reason: Causes the MSan buildbot to fail (see comments on
https://revi

Revert "[memprof] Store callsite metadata with memprof records."

This reverts commit 0d362c90d335509c57c0fbd01ae1829e2b9c3765.

Reason: Causes the MSan buildbot to fail (see comments on
https://reviews.llvm.org/D121179 for more information

show more ...

[memprof] Store callsite metadata with memprof records.

To ease profile annotation, each of the callsites in a function can be
annotated with profile data - "IR metadata format for MemProf" [1]. Thi

[memprof] Store callsite metadata with memprof records.

To ease profile annotation, each of the callsites in a function can be
annotated with profile data - "IR metadata format for MemProf" [1]. This
patch extends the on-disk serialized record format to store the debug
information for allocation callsites incl inline frames. This change is
incompatible with the existing format i.e. indexed profiles must be
regenerated, raw profiles are unaffected.

[1] https://groups.google.com/g/llvm-dev/c/aWHsdMxKAfE/m/WtEmRqyhAgAJ

Reviewed By: tejohnson

Differential Revision: https://reviews.llvm.org/D121179

show more ...

[memprof] Update the frame is inline logic and unittests.

Since DI frames are enumerated with the leaf function at index 0, this
patch fixes the logic when IsInlineFrame is set. Also update the
unit

[memprof] Update the frame is inline logic and unittests.

Since DI frames are enumerated with the leaf function at index 0, this
patch fixes the logic when IsInlineFrame is set. Also update the
unittests to check that only the last frame is marked as non-inline from
a set of DI Frames for a PC address.

Differential Revision: https://reviews.llvm.org/D121830

show more ...

[memprof] Add a test to verify callstack order.

We add to ensure that we are observing the correct callstack order in memprof
during symbolization. There was some confusion whether the order of
DIFr

[memprof] Add a test to verify callstack order.

We add to ensure that we are observing the correct callstack order in memprof
during symbolization. There was some confusion whether the order of
DIFrame objects were reversed but in reality the leaf function is at
index 0 so no code changes are required.

Differential Revision: https://reviews.llvm.org/D121759

show more ...

[memprof] Filter out callstack frames which cannot be symbolized.

This patch filters out callstack frames which can't be symbolized or if
the frames belong to the runtime. Symbolization may not be p

[memprof] Filter out callstack frames which cannot be symbolized.

This patch filters out callstack frames which can't be symbolized or if
the frames belong to the runtime. Symbolization may not be possible if
debug information is unavailable or if the addresses are from a shared
library. For now we only support optimization of the main binary which
is statically linked to the compiler runtime.

Differential Revision: https://reviews.llvm.org/D120860

show more ...

[memprof] Symbolize and cache stack frames.

Currently, symbolization of stack frames occurs on demand when the instrprof writer
iterates over all the records in the raw memprof reader. With this
cha

[memprof] Symbolize and cache stack frames.

Currently, symbolization of stack frames occurs on demand when the instrprof writer
iterates over all the records in the raw memprof reader. With this
change we symbolize and cache the frames immediately after reading the
raw profiles. For a large internal binary this results in a runtime
reduction of ~50% (2m -> 48s) when merging a memprof raw profile with a
raw instr profile to generate an indexed profile. This change also makes
it simpler in the future to generate additional calling context
metadata to attach to each memprof record.

Differential Revision: https://reviews.llvm.org/D120430

show more ...

Cleanup includes: ProfileData

Estimation of the impact on preprocessor output:

before: 1067349756
after: 1065940348

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-clea

Cleanup includes: ProfileData

Estimation of the impact on preprocessor output:

before: 1067349756
after: 1065940348

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120434

show more ...

Reland "[memprof] Extend the index prof format to include memory profiles."

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profi

Reland "[memprof] Extend the index prof format to include memory profiles."

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header adds a new
field which points to the offset of the memory profile section (if
present) in the indexed profile. For users who do not utilize this
feature the only overhead is a 64-bit offset in the header.

The memory profile section contains (1) profile metadata describing the
information recorded for each entry (2) an on-disk hashtable containing
the profile records indexed via llvm::md5(function_name). We chose to
introduce a separate hash table instead of the existing one since the
indexing for the instrumented fdo hash table is based on a CFG hash
which itself is perturbed by memprof instrumentation.

This commit also includes the changes reviewed separately in D120093.

Differential Revision: https://reviews.llvm.org/D120103

show more ...

Revert "Reland "[memprof] Extend the index prof format to include memory profiles.""

This reverts commit 807ba7aace188ada83ddb4477265728e97346af1.

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 85355a560a33897453df2ef959e255ee725eebce.

This patch adds support for optional memory profile informa

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 85355a560a33897453df2ef959e255ee725eebce.

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header adds a new
field which points to the offset of the memory profile section (if
present) in the indexed profile. For users who do not utilize this
feature the only overhead is a 64-bit offset in the header.

The memory profile section contains (1) profile metadata describing the
information recorded for each entry (2) an on-disk hashtable containing
the profile records indexed via llvm::md5(function_name). We chose to
introduce a separate hash table instead of the existing one since the
indexing for the instrumented fdo hash table is based on a CFG hash
which itself is perturbed by memprof instrumentation.

Differential Revision: https://reviews.llvm.org/D118653

show more ...

Reland "[memprof] Introduce a wrapper around MemInfoBlock."

This reverts commit e6999040f5758f89a64b6232119b775b7bd1c85b.

Update test to fix signed int comparison warning, fix whitespace in
compile

Reland "[memprof] Introduce a wrapper around MemInfoBlock."

This reverts commit e6999040f5758f89a64b6232119b775b7bd1c85b.

Update test to fix signed int comparison warning, fix whitespace in
compiler-rt MIBEntryDef.inc file.

Differential Revision: https://reviews.llvm.org/D117256

show more ...

Revert "[memprof] Introduce a wrapper around MemInfoBlock."

This reverts commit 9b67165285c5e752fce3b554769f5a22e7b38da8. [3/4]

Revert "Reland "[memprof] Extend the index prof format to include memory profiles.""

This reverts commit de54e4ab78ef09b60f870e8df6f8a87e56d6bd94 [1/4]

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 0f73fb18ca333e38cdb9ffa701a8db026c56041d.

Use llvm/Profile/MIBEntryDef.inc instead of relative path.

Reland "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 0f73fb18ca333e38cdb9ffa701a8db026c56041d.

Use llvm/Profile/MIBEntryDef.inc instead of relative path.

Generated the raw profile data with `-mllvm
-enable-name-compression=false` so that builbots where the reader is
built without zlib do not fail.

Also updated the test build instructions.

show more ...

Revert "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 43c2348c5b926df6bdbc5b70efaa35ecdefe12d5.

Buildbots are failing with an error on reading memprof test

Revert "[memprof] Extend the index prof format to include memory profiles."

This reverts commit 43c2348c5b926df6bdbc5b70efaa35ecdefe12d5.

Buildbots are failing with an error on reading memprof testdata.
"Inputs/basic.profraw: profile uses zlib
compression but the profile reader was built without zlib support"

https://lab.llvm.org/buildbot/#/builders/16/builds/24490

show more ...

[memprof] Extend the index prof format to include memory profiles.

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header

[memprof] Extend the index prof format to include memory profiles.

This patch adds support for optional memory profile information to be
included with and indexed profile. The indexed profile header adds a new
field which points to the offset of the memory profile section (if
present) in the indexed profile. For users who do not utilize this
feature the only overhead is a 64-bit offset in the header.

The memory profile section contains (1) profile metadata describing the
information recorded for each entry (2) an on-disk hashtable containing
the profile records indexed via llvm::md5(function_name). We chose to
introduce a separate hash table instead of the existing one since the
indexing for the instrumented fdo hash table is based on a CFG hash
which itself is perturbed by memprof instrumentation.

Differential Revision: https://reviews.llvm.org/D118653

show more ...

[memprof] Introduce a wrapper around MemInfoBlock.

Use the macro based format to add a wrapper around the MemInfoBlock
when stored in the MemProfRecord. This wrapped block can then be
serialized/des

[memprof] Introduce a wrapper around MemInfoBlock.

Use the macro based format to add a wrapper around the MemInfoBlock
when stored in the MemProfRecord. This wrapped block can then be
serialized/deserialized based on a schema specified by a list of enums.

Differential Revision: https://reviews.llvm.org/D117256

show more ...