[llvm] [Debuginfod] DebuginfodCollection and DebuginfodServer for tracking local debuginfo.

This library implements the class `DebuginfodCollection`, which scans a set of directories for binaries, c

[llvm] [Debuginfod] DebuginfodCollection and DebuginfodServer for tracking local debuginfo.

This library implements the class `DebuginfodCollection`, which scans a set of directories for binaries, classifying them according to whether they contain debuginfo. This also provides the `DebuginfodServer`, an `HTTPServer` which serves debuginfod's `/debuginfo` and `/executable` endpoints. This is intended as the final new supporting library required for `llvm-debuginfod`.

As implemented here, `DebuginfodCollection` only finds ELF binaries and DWARF debuginfo. All other files are ignored. However, the class interface is format-agnostic. Generalizing to support other platforms will require refactoring of LLVM's object parsing libraries to eliminate use of `report_fatal_error` ([[ https://github.com/llvm/llvm-project/blob/main/llvm/lib/Object/WasmObjectFile.cpp#L74 | e.g. when reading WASM files ]]), so that the debuginfod daemon does not crash when it encounters a malformed file on the disk.

The `DebuginfodCollection` is tested by end-to-end tests of the debuginfod server (D114846).

Reviewed By: mysterymath

Differential Revision: https://reviews.llvm.org/D114845

show more ...

Cleanup includes: DebugInfo & CodeGen

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121332

[Symbolizer] Move default ctor into .cpp file

Follow up to 1e396affca6a0d21247d960c93a415e8f6fe0301. On some standard
library configurations these have a dependency on the complete type of
Symboliz

[Symbolizer] Move default ctor into .cpp file

Follow up to 1e396affca6a0d21247d960c93a415e8f6fe0301. On some standard
library configurations these have a dependency on the complete type of
SymbolizableModule.

show more ...

[Symbolize] LRU cache binaries in llvm-symbolizer.

This change adds a simple LRU cache to the Symbolize class to put a cap
on llvm-symbolizer memory usage. Previously, the Symbolizer's virtual
memor

[Symbolize] LRU cache binaries in llvm-symbolizer.

This change adds a simple LRU cache to the Symbolize class to put a cap
on llvm-symbolizer memory usage. Previously, the Symbolizer's virtual
memory footprint would grow without bound as additional binaries were
referenced.

I'm putting this out there early for an informal review, since there may be
a dramatically different/better way to go about this. I still need to
figure out a good default constant for the memory cap and benchmark the
implementation against a large symbolization workload. Right now I've
pegged max memory usage at zero for testing purposes, which evicts the whole
cache every time.

Unfortunately, it looks like StringRefs in the returned DI objects can
directly refer to the contents of binaries. Accordingly, the cache
pruning must be explicitly requested by the caller, as the caller must
guarantee that none of the returned objects will be used afterwards.

For llvm-symbolizer this a light burden; symbolization occurs
line-by-line, and the returned objects are discarded after each.

Implementation wise, there are a number of nested caches that depend
on one another. I've implemented a simple Evictor callback system to
allow derived caches to register eviction actions to occur when the
underlying binaries are evicted.

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D119784

show more ...

[Symbolizer] Move ctor/dtor into .cpp file

On some standard library configurations these have a dependency on the
complete type of SymbolizableModule. They also do a lot of
copying/freeing so no poi

[Symbolizer] Move ctor/dtor into .cpp file

On some standard library configurations these have a dependency on the
complete type of SymbolizableModule. They also do a lot of
copying/freeing so no point in inlining them.

show more ...

Cleanup include: DebugInfo/Symbolize

Estimation of the impact on preprocessor output
after: 1067349756
before:1067487786

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-

Cleanup include: DebugInfo/Symbolize

Estimation of the impact on preprocessor output
after: 1067349756
before:1067487786

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D120433

show more ...

[Symbolize] Fix -Wunused-variable in -DLLVM_ENABLE_ASSERTIONS=off builds after D118633

[Symbolizer] Add Build ID flag to llvm-symbolizer.

This adds a --build-id=<hex build ID> flag to llvm-symbolizer. If --obj
is unspecified, this will attempt to look up the provided build ID using
wh

[Symbolizer] Add Build ID flag to llvm-symbolizer.

This adds a --build-id=<hex build ID> flag to llvm-symbolizer. If --obj
is unspecified, this will attempt to look up the provided build ID using
whatever mechanisms are available to the Symbolizer (typically,
debuginfod). The semantics are then as if the found binary were given
using the --obj flag.

Reviewed By: jhenderson, phosek

Differential Revision: https://reviews.llvm.org/D118633

show more ...

[Debuginfod] [Symbolizer] Break debuginfod out of libLLVM.

Debuginfod can pull in libcurl as a dependency, which isn't appropriate
for libLLVM. (See
https://gitlab.freedesktop.org/mesa/mesa/-/issues

[Debuginfod] [Symbolizer] Break debuginfod out of libLLVM.

Debuginfod can pull in libcurl as a dependency, which isn't appropriate
for libLLVM. (See
https://gitlab.freedesktop.org/mesa/mesa/-/issues/5732).

This change breaks out debuginfod into a separate non-component library
that can be used directly in llvm-symbolizer. The tool can inject
debuginfod into the Symbolizer library via an abstract DebugInfoFetcher
interface, breaking the dependency of Symbolizer on debuinfod.

See https://github.com/llvm/llvm-project/issues/52731

Reviewed By: phosek

Differential Revision: https://reviews.llvm.org/D118413

show more ...

[DebugInfo] Move the SymbolizableObjectFile header to include/llvm.

This change moves the SymbolizableObjectFile header to
include/llvm/DebugInfo/Symbolize. Making this header available to other
llv

[DebugInfo] Move the SymbolizableObjectFile header to include/llvm.

This change moves the SymbolizableObjectFile header to
include/llvm/DebugInfo/Symbolize. Making this header available to other
llvm libraries simplifies use cases where implicit caching, multiple
platform support and other features of the Symbolizer class are not
required. This also makes the dependent libraries easier to unit test
by having mocks which derive from SymbolizableModule.

Differential Revision: https://reviews.llvm.org/D116781

show more ...

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.
Fixed a cast of Erorr::success() to Expected<> i

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.
Fixed a cast of Erorr::success() to Expected<> in debuginfod library.
Added Debuginfod to Symbolize deps in gn.
Updates compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh to include Debuginfod library to fix sanitizer-x86_64-linux breakage.

Reviewed By: jhenderson, vitalybuka

Differential Revision: https://reviews.llvm.org/D113717

show more ...

Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit 5bba0fe12b2971a9cbc859f48ee6e6c1356c88b8.
Makes lld depend on libcurl, see comments on https://reviews.

Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit 5bba0fe12b2971a9cbc859f48ee6e6c1356c88b8.
Makes lld depend on libcurl, see comments on https://reviews.llvm.org/D113717

show more ...

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.
Fixed a cast of Erorr::success() to Expected<> i

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.
Fixed a cast of Erorr::success() to Expected<> in debuginfod library.
Added Debuginfod to Symbolize deps in gn.
Updates compiler-rt/lib/sanitizer_common/symbolizer/scripts/build_symbolizer.sh to include Debuginfod library to fix sanitizer-x86_64-linux breakage.

Reviewed By: jhenderson, vitalybuka

Differential Revision: https://reviews.llvm.org/D113717

show more ...

Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit e2ad4f1756027cd27f6c82db620042e9877f900c because it
does not correctly fix the sanitizer buildbot break

Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit e2ad4f1756027cd27f6c82db620042e9877f900c because it
does not correctly fix the sanitizer buildbot breakage.

show more ...

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.
Fixed a cast of Erorr::success() to Expected<> i

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.
Fixed a cast of Erorr::success() to Expected<> in debuginfod library.
Added Debuginfod to Symbolize deps in gn.
Adds new symbolizer symbols to `global_symbols.txt`.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D113717

show more ...

Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit 02cc8d698c4941f8f0120ea1a5d7205fb33a312d because it
caused buildbot failures. The issue appears to be s

Revert "[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer."

This reverts commit 02cc8d698c4941f8f0120ea1a5d7205fb33a312d because it
caused buildbot failures. The issue appears to be simply that we need to
only enable debuginfod when the HTTPClient has been initialized by the
running tool, since InitLLVM does not do the initialization step anymore.

show more ...

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.

Reviewed By: jhenderson

Differential Revision:

[Symbolizer][Debuginfo] Add debuginfod client to llvm-symbolizer.

Adds a fallback to use the debuginfod client library (386655) in `findDebugBinary`.

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D113717

show more ...

Support: Use Expected<T>::moveInto() in a few places

These are some usage examples for `Expected<T>::moveInto()`.

Differential Revision: https://reviews.llvm.org/D112280

[Symbolize] Demangle Rust symbols

Add support for demangling Rust v0 symbols to LLVM symbolizer by reusing
nonMicrosoftDemangle which supports both Itanium and Rust mangling.

Reviewed By: dblaikie,

[Symbolize] Demangle Rust symbols

Add support for demangling Rust v0 symbols to LLVM symbolizer by reusing
nonMicrosoftDemangle which supports both Itanium and Rust mangling.

Reviewed By: dblaikie, jhenderson

Part of https://reviews.llvm.org/D110664

show more ...

[DWARF] Don't process .debug_info relocations for DWO Context

When we build with split dwarf in single mode the .o files that contain both "normal" debug sections and dwo sections, along with reloca

[DWARF] Don't process .debug_info relocations for DWO Context

When we build with split dwarf in single mode the .o files that contain both "normal" debug sections and dwo sections, along with relocaiton sections for "normal" debug sections.
When we create DWARF context in DWARFObjInMemory we process relocations and store them in the map for .debug_info, etc section.
For DWO Context we also do it for non dwo dwarf sections. Which I believe is not necessary. This leads to a lot of memory being wasted. We observed 70GB extra memory being used.

I went with context sensitive approach, flag is passed in. I am not sure if it's always safe not to process relocations for regular debug sections if Obj contains .dwo sections.
If it is alternatvie might be just to scan, in constructor, sections and if there are .dwo sections not to process regular debug ones.

Reviewed By: dblaikie

Differential Revision: https://reviews.llvm.org/D106624

show more ...

Reformat files.

Differential Revision: https://reviews.llvm.org/D105982

[Symbolize] Teach symbolizer to work directly on object file.

This patch intended to provide additional interface to LLVMsymbolizer
such that they work directly on object files. There is an existing

[Symbolize] Teach symbolizer to work directly on object file.

This patch intended to provide additional interface to LLVMsymbolizer
such that they work directly on object files. There is an existing
method - symbolizecode which takes an object file, this patch provides
similar overloads for symbolizeInlinedCode, symbolizeData,
symbolizeFrame. This can be useful for clients who already have a
in-memory object files to symbolize for.

Patch By: pvellien (praveen velliengiri)

Reviewed By: scott.linder

Differential Revision: https://reviews.llvm.org/D95232

show more ...

[llvm] Use isDigit (NFC)

[llvm] Use llvm::all_of (NFC)

[lib/Object, tools] - Make ELFObjectFile::getELFFile return reference.

We always have an object, so we don't have to return a pointer.

Differential revision: https://reviews.llvm.org/D92560