Introduce new symbol on-demand for debug info

This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant perfo

Introduce new symbol on-demand for debug info

This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant performance improvement
for application with dynamic linking mode which has large
number of modules.
The feature can be turned on with:
"settings set symbols.load-on-demand true"

The feature works by creating a new SymbolFileOnDemand class for
each module which wraps the actual SymbolFIle subclass as member
variable. By default, most virtual methods on SymbolFileOnDemand are
skipped so that it looks like there is no debug info for that module.
But once the module's debug info is explicitly requested to
be enabled (in the conditions mentioned below) SymbolFileOnDemand
will allow all methods to pass through and forward to the actual SymbolFile
which would hydrate module's debug info on-demand.

In an internal benchmark, we are seeing more than 95% improvement
for a 3000 modules application.

Currently we are providing several ways to on demand hydrate
a module's debug info:
* Source line breakpoint: matching in supported files
* Stack trace: resolving symbol context for an address
* Symbolic breakpoint: symbol table match guided promotion
* Global variable: symbol table match guided promotion

In all above situations the module's debug info will be on-demand
parsed and indexed.

Some follow-ups for this feature:
* Add a command that allows users to load debug info explicitly while using a
new or existing command when this feature is enabled
* Add settings for "never load any of these executables in Symbols On Demand"
that takes a list of globs
* Add settings for "always load the the debug info for executables in Symbols
On Demand" that takes a list of globs
* Add a new column in "image list" that shows up by default when Symbols On
Demand is enable to show the status for each shlib like "not enabled for
this", "debug info off" and "debug info on" (with a single character to
short string, not the ones I just typed)

Differential Revision: https://reviews.llvm.org/D121631

show more ...

Refactor protected virtual functions from SymbolFile into new SymbolFileCommon class.

This is a preparatory patch for https://reviews.llvm.org/D121631.
It refactors protected virtual members of Symb

Refactor protected virtual functions from SymbolFile into new SymbolFileCommon class.

This is a preparatory patch for https://reviews.llvm.org/D121631.
It refactors protected virtual members of SymbolFile
into a new SymbolFileCommon class per suggestion in:
https://reviews.llvm.org/D121631

This will avoid the friendship declaration in that patch.

Differential Revision: https://reviews.llvm.org/D124110

show more ...

Remove redundant return and continue statements (NFC)

Identified with readability-redundant-control-flow.

[lldb][NFC] clang-format some files as preparation for https://reviews.llvm.org/D114627

Reviewed By: werat

Differential Revision: https://reviews.llvm.org/D115110

Add new key/value pairs to the module statistics for "statistics dump".

The new key/value pairs that are added to each module's stats are:
"debugInfoByteSize": The size in bytes of debug info for ea

Add new key/value pairs to the module statistics for "statistics dump".

The new key/value pairs that are added to each module's stats are:
"debugInfoByteSize": The size in bytes of debug info for each module.
"debugInfoIndexTime": The time in seconds that it took to index the debug info.
"debugInfoParseTime": The time in seconds that debug info had to be parsed.

At the top level we add up all of the debug info size, parse time and index time with the following keys:
"totalDebugInfoByteSize": The size in bytes of all debug info in all modules.
"totalDebugInfoIndexTime": The time in seconds that it took to index all debug info if it was indexed for all modules.
"totalDebugInfoParseTime": The time in seconds that debug info was parsed for all modules.

Differential Revision: https://reviews.llvm.org/D112501

show more ...

[lldb] Refactor argument group by SourceLocationSpec (NFCI)

This patch refactors a good part of the code base turning the usual
FileSpec, Line, Column, CheckInlines, ExactMatch arguments into a
Sour

[lldb] Refactor argument group by SourceLocationSpec (NFCI)

This patch refactors a good part of the code base turning the usual
FileSpec, Line, Column, CheckInlines, ExactMatch arguments into a
SourceLocationSpec object.

This change is required for a following patch that will add handling of the
column line information when doing symbol resolution.

Differential Revision: https://reviews.llvm.org/D100965

Signed-off-by: Med Ismail Bennani <[email protected]>

show more ...

[lldb][NFC] Make all CompilerDeclContext parameters references instead of pointers

Summary:
All of our lookup APIs either use `CompilerDeclContext &` or `CompilerDeclContext *` semi-randomly it seem

[lldb][NFC] Make all CompilerDeclContext parameters references instead of pointers

Summary:
All of our lookup APIs either use `CompilerDeclContext &` or `CompilerDeclContext *` semi-randomly it seems.
This leads to us constantly converting between those two types (and doing nullptr checks when going from
pointer to reference). It also leads to the confusing situation where we have two possible ways to express
that we don't have a CompilerDeclContex: either a nullptr or an invalid CompilerDeclContext (aka a default
constructed CompilerDeclContext).

This moves all APIs to use references and gets rid of all the nullptr checks and conversions.

Reviewers: labath, mib, shafik

Reviewed By: labath, shafik

Subscribers: shafik, arphaman, abidh, JDevlieghere, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D74607

show more ...

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp ----------------------------------------

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp -------------------------------------------------===//
```
However in LLDB most of our source files have arbitrary changes to this format and
these changes are spreading through LLDB as folks usually just use the existing
source files as templates for their new files (most notably the unnecessary
editor language indicator `-*- C++ -*-` is spreading and in every review
someone is pointing out that this is wrong, resulting in people pointing out that this
is done in the same way in other files).

This patch removes most of these inconsistencies including the editor language indicators,
all the different missing/additional '-' characters, files that center the file name, missing
trailing `===//` (mostly caused by clang-format breaking the line).

Reviewers: aprantl, espindola, jfb, shafik, JDevlieghere

Reviewed By: JDevlieghere

Subscribers: dexonsmith, wuzish, emaste, sdardis, nemanjai, kbarton, MaskRay, atanasyan, arphaman, jfb, abidh, jsji, JDevlieghere, usaxena95, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D73258

show more ...

Add RTTI support to the SymbolFile class hierarchy

Differential Revision: https://reviews.llvm.org/D70322

Performance: Add a set of visited SymbolFiles to the other FindFiles variant.

This is basically the same bug as in r260434.

SymbolFileDWARF::FindTypes has exponential worst-case when digging
throug

Performance: Add a set of visited SymbolFiles to the other FindFiles variant.

This is basically the same bug as in r260434.

SymbolFileDWARF::FindTypes has exponential worst-case when digging
through dependency DAG of .pcm files because each object file and .pcm
file may depend on an already-visited .pcm file, which may again have
dependencies. Fixed here by carrying a set of already visited
SymbolFiles around.

rdar://problem/56993424

Differential Revision: https://reviews.llvm.org/D70106

show more ...

Modernize the rest of the Find.* API (NFC)

This patch removes the size_t return value and the append parameter
from the remainder of the Find.* functions in LLDB's internal API. As
in the previous p

Modernize the rest of the Find.* API (NFC)

This patch removes the size_t return value and the append parameter
from the remainder of the Find.* functions in LLDB's internal API. As
in the previous patches, this is motivated by the fact that these
parameters aren't really used, and in the case of the append parameter
were frequently implemented incorrectly.

Differential Revision: https://reviews.llvm.org/D69119

llvm-svn: 375160

show more ...

Remove size_t return parameter from FindTypes

In r368345 I accidentally introduced a regression that would
over-report the number of matches found by FindTypes if the
DeclContext Filter was hit.

Th

Remove size_t return parameter from FindTypes

In r368345 I accidentally introduced a regression that would
over-report the number of matches found by FindTypes if the
DeclContext Filter was hit.

This patch simply removes the size_t return parameter altogether —
it's not that useful.

rdar://problem/55500457

Differential Revision: https://reviews.llvm.org/D68169

llvm-svn: 373344

show more ...

Remove unused "append" parameter from FindTypes API

I noticed that SymbolFileDWARFDebugMap::FindTypes was implementing it
incorrectly (passing append=false in a for-loop to recursive calls to
FindTy

Remove unused "append" parameter from FindTypes API

I noticed that SymbolFileDWARFDebugMap::FindTypes was implementing it
incorrectly (passing append=false in a for-loop to recursive calls to
FindTypes would yield only the very last set of results), but instead
of fixing it, removing it seemed like an even better option.

rdar://problem/54412692

Differential Revision: https://reviews.llvm.org/D68171

llvm-svn: 373224

show more ...

Extend FindTypes with CompilerContext to allow filtering by language.

This patch is also motivated by the Swift branch and is effectively NFC for the single-TypeSystem llvm.org branch.

In multi-lan

Extend FindTypes with CompilerContext to allow filtering by language.

This patch is also motivated by the Swift branch and is effectively NFC for the single-TypeSystem llvm.org branch.

In multi-language projects it is extremely common to have, e.g., a
Clang type and a similarly-named rendition of that same type in
another language. When searching for a type It is much cheaper to pass
a set of supported languages to the SymbolFile than having it
materialize every result and then rejecting the materialized types
that have the wrong language.

Differential Revision: https://reviews.llvm.org/D66546

<rdar://problem/54471165>

This reapplies r369690 with a previously missing constructor for LanguageSet.

llvm-svn: 369710

show more ...

Revert Extend FindTypes with CompilerContext to allow filtering by language.

This reverts r369690 (git commit aa3a564efa6b5fff2129f81a4041069a0233168f)

llvm-svn: 369702

Extend FindTypes with CompilerContext to allow filtering by language.

This patch is also motivated by the Swift branch and is effectively NFC for the single-TypeSystem llvm.org branch.

In multi-lan

Extend FindTypes with CompilerContext to allow filtering by language.

This patch is also motivated by the Swift branch and is effectively NFC for the single-TypeSystem llvm.org branch.

In multi-language projects it is extremely common to have, e.g., a
Clang type and a similarly-named rendition of that same type in
another language. When searching for a type It is much cheaper to pass
a set of supported languages to the SymbolFile than having it
materialize every result and then rejecting the materialized types
that have the wrong language.

Differential Revision: https://reviews.llvm.org/D66546

<rdar://problem/54471165>

llvm-svn: 369690

show more ...

Generalize FindTypes with CompilerContext to support fuzzy lookup

This patch generalizes the FindTypes with CompilerContext interface to
support looking up a type of unknown kind by name, as well as

Generalize FindTypes with CompilerContext to support fuzzy lookup

This patch generalizes the FindTypes with CompilerContext interface to
support looking up a type of unknown kind by name, as well as looking
up a type inside an unspecified submodule. These features are
motivated by the Swift branch, but are fully tested via unit tests and
lldb-test on llvm.org. Specifically, this patch adds an AnyModule and
an AnyType CompilerContext kind.

Differential Revision: https://reviews.llvm.org/D66507

rdar://problem/54471165

llvm-svn: 369555

show more ...

Fix PDB tests after r367820

The commit changed Module dumping code to call SymbolFile::Dump
directly, which meant that we were no longer showing the plugin name in
the output (as that was done in th

Fix PDB tests after r367820

The commit changed Module dumping code to call SymbolFile::Dump
directly, which meant that we were no longer showing the plugin name in
the output (as that was done in the SymbolVendor).

This adds the plugin name printing code to the SymbolFile dump method,
and tweak the assertions in the PDB tests to match it correctly.

llvm-svn: 367835

show more ...

Remove SymbolVendor::GetSymtab

Summary:
This patch removes the GetSymtab method from the SymbolVendor, which is
a no-op as it's implementation just forwards to the relevant SymbolFile.
Instead it cr

Remove SymbolVendor::GetSymtab

Summary:
This patch removes the GetSymtab method from the SymbolVendor, which is
a no-op as it's implementation just forwards to the relevant SymbolFile.
Instead it creates a Module::GetSymtab, which calls the SymbolFile
method directly.

All callers have been updated to use the Module method directly instead
of a two phase GetSymbolVendor->GetSymtab search, which leads to reduced
intentation in a lot of deeply nested code.

Reviewers: clayborg, JDevlieghere, jingham

Subscribers: lldb-commits

Differential Revision: https://reviews.llvm.org/D65569

llvm-svn: 367820

show more ...

SymbolVendor: Remove the object file member variable

Summary:
The last responsibility of the SymbolVendor was to hold an owning
reference to the object file (in case symbols are being read from a
di

SymbolVendor: Remove the object file member variable

Summary:
The last responsibility of the SymbolVendor was to hold an owning
reference to the object file (in case symbols are being read from a
different file than the main module). As SymbolFile classes already hold
a non-owning reference to the object file, we can easily remove this
responsibility of the SymbolVendor by making the SymbolFile reference
owning.

Reviewers: JDevlieghere, clayborg, jingham

Subscribers: lldb-commits

Differential Revision: https://reviews.llvm.org/D65401

llvm-svn: 367392

show more ...

[Symbol] Use llvm::Expected when getting TypeSystems

Summary:
This commit achieves the following:
- Functions used to return a `TypeSystem *` return an
`llvm::Expected<TypeSystem *>` now. This mea

[Symbol] Use llvm::Expected when getting TypeSystems

Summary:
This commit achieves the following:
- Functions used to return a `TypeSystem *` return an
`llvm::Expected<TypeSystem *>` now. This means that the result of a call
is always checked, forcing clients to move more carefully.
- `TypeSystemMap::GetTypeSystemForLanguage` will either return an Error or a
non-null pointer to a TypeSystem.

Reviewers: JDevlieghere, davide, compnerd

Subscribers: jdoerfert, lldb-commits

Differential Revision: https://reviews.llvm.org/D65122

llvm-svn: 367360

show more ...

SymbolVendor: Move locking into the Symbol Files

Summary:
The last bit of functionality in SymbolVendor passthrough functions is
the locking the module mutex. While it may be nice doing the locking

SymbolVendor: Move locking into the Symbol Files

Summary:
The last bit of functionality in SymbolVendor passthrough functions is
the locking the module mutex. While it may be nice doing the locking in
a central place, we weren't really succesful in doing that right now,
because some SymbolFile function could still be called without going
through the SymbolVendor. This meant in SymbolFileDWARF (the only
battle-tested symbol file implementation) roughly a half of the
functions was taking additional locks and another half was asserting
that the lock is already held. By making the SymbolFile responsible for
locking, we can at least make the situation in SymbolFileDWARF more
consistent.

Reviewers: clayborg, JDevlieghere, jingham, jdoerfert

Subscribers: aprantl, lldb-commits

Differential Revision: https://reviews.llvm.org/D65329

llvm-svn: 367298

show more ...

SymbolVendor: Make SectionAddressesChanged a passthrough

Summary:
This moves the implementation of the function into the SymbolFile class,
making it possible to excise the SymbolVendor passthrough f

SymbolVendor: Make SectionAddressesChanged a passthrough

Summary:
This moves the implementation of the function into the SymbolFile class,
making it possible to excise the SymbolVendor passthrough functions in
follow-up patches.

Reviewers: clayborg, jingham, JDevlieghere

Subscribers: lldb-commits

Differential Revision: https://reviews.llvm.org/D65266

llvm-svn: 367231

show more ...

SymbolVendor: Move Symtab construction into the SymbolFile

Summary:
Instead of having SymbolVendor coordinate Symtab construction between
Symbol and Object files, make the SymbolVendor function a pa

SymbolVendor: Move Symtab construction into the SymbolFile

Summary:
Instead of having SymbolVendor coordinate Symtab construction between
Symbol and Object files, make the SymbolVendor function a passthrough,
and put all of the logic into the SymbolFile.

Reviewers: clayborg, JDevlieghere, jingham, espindola

Subscribers: emaste, mgorny, arichardson, MaskRay, lldb-commits

Differential Revision: https://reviews.llvm.org/D65208

llvm-svn: 367086

show more ...

SymbolFile: Fix -Wunused-variable in -DLLVM_ENABLE_ASSERTIONS=off builds after D65089/r366791

llvm-svn: 367001