Introduce new symbol on-demand for debug info

This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant perfo

Introduce new symbol on-demand for debug info

This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant performance improvement
for application with dynamic linking mode which has large
number of modules.
The feature can be turned on with:
"settings set symbols.load-on-demand true"

The feature works by creating a new SymbolFileOnDemand class for
each module which wraps the actual SymbolFIle subclass as member
variable. By default, most virtual methods on SymbolFileOnDemand are
skipped so that it looks like there is no debug info for that module.
But once the module's debug info is explicitly requested to
be enabled (in the conditions mentioned below) SymbolFileOnDemand
will allow all methods to pass through and forward to the actual SymbolFile
which would hydrate module's debug info on-demand.

In an internal benchmark, we are seeing more than 95% improvement
for a 3000 modules application.

Currently we are providing several ways to on demand hydrate
a module's debug info:
* Source line breakpoint: matching in supported files
* Stack trace: resolving symbol context for an address
* Symbolic breakpoint: symbol table match guided promotion
* Global variable: symbol table match guided promotion

In all above situations the module's debug info will be on-demand
parsed and indexed.

Some follow-ups for this feature:
* Add a command that allows users to load debug info explicitly while using a
new or existing command when this feature is enabled
* Add settings for "never load any of these executables in Symbols On Demand"
that takes a list of globs
* Add settings for "always load the the debug info for executables in Symbols
On Demand" that takes a list of globs
* Add a new column in "image list" that shows up by default when Symbols On
Demand is enable to show the status for each shlib like "not enabled for
this", "debug info off" and "debug info on" (with a single character to
short string, not the ones I just typed)

Differential Revision: https://reviews.llvm.org/D121631

show more ...

[LLDB] Applying clang-tidy modernize-use-default-member-init over LLDB

Applied modernize-use-default-member-init clang-tidy check over LLDB.
It appears in many files we had already switched to in cl

[LLDB] Applying clang-tidy modernize-use-default-member-init over LLDB

Applied modernize-use-default-member-init clang-tidy check over LLDB.
It appears in many files we had already switched to in class member init but
never updated the constructors to reflect that. This check is already present in
the lldb/.clang-tidy config.

Differential Revision: https://reviews.llvm.org/D121481

show more ...

[lldb] Rename Logging.h to LLDBLog.h and clean up includes

Most of our code was including Log.h even though that is not where the
"lldb" log channel is defined (Log.h defines the generic logging
inf

[lldb] Rename Logging.h to LLDBLog.h and clean up includes

Most of our code was including Log.h even though that is not where the
"lldb" log channel is defined (Log.h defines the generic logging
infrastructure). This worked because Log.h included Logging.h, even
though it should.

After the recent refactor, it became impossible the two files include
each other in this direction (the opposite inclusion is needed), so this
patch removes the workaround that was put in place and cleans up all
files to include the right thing. It also renames the file to LLDBLog to
better reflect its purpose.

show more ...

[lldb] Convert "LLDB" log channel to the new API

Added the ability to cache the finalized symbol tables subsequent debug sessions to start faster.

This is an updated version of the https://reviews.llvm.org/D113789 patch with the following changes:

Added the ability to cache the finalized symbol tables subsequent debug sessions to start faster.

This is an updated version of the https://reviews.llvm.org/D113789 patch with the following changes:
- We no longer modify modification times of the cache files
- Use LLVM caching and cache pruning instead of making a new cache mechanism (See DataFileCache.h/.cpp)
- Add signature to start of each file since we are not using modification times so we can tell when caches are stale and remove and re-create the cache file as files are changed
- Add settings to control the cache size, disk percentage and expiration in days to keep cache size under control

This patch enables symbol tables to be cached in the LLDB index cache directory. All cache files are in a single directory and the files use unique names to ensure that files from the same path will re-use the same file as files get modified. This means as files change, their cache files will be deleted and updated. The modification time of each of the cache files is not modified so that access based pruning of the cache can be implemented.

The symbol table cache files start with a signature that uniquely identifies a file on disk and contains one or more of the following items:
- object file UUID if available
- object file mod time if available
- object name for BSD archive .o files that are in .a files if available

If none of these signature items are available, then the file will not be cached. This keeps temporary object files from expressions from being cached.

When the cache files are loaded on subsequent debug sessions, the signature is compare and if the file has been modified (uuid changes, mod time changes, or object file mod time changes) then the cache file is deleted and re-created.

Module caching must be enabled by the user before this can be used:

symbols.enable-lldb-index-cache (boolean) = false

(lldb) settings set symbols.enable-lldb-index-cache true

There is also a setting that allows the user to specify a module cache directory that defaults to a directory that defaults to being next to the symbols.clang-modules-cache-path directory in a temp directory:

(lldb) settings show symbols.lldb-index-cache-path
/var/folders/9p/472sr0c55l9b20x2zg36b91h0000gn/C/lldb/IndexCache

If this setting is enabled, the finalized symbol tables will be serialized and saved to disc so they can be quickly loaded next time you debug.

Each module can cache one or more files in the index cache directory. The cache file names must be unique to a file on disk and its architecture and object name for .o files in BSD archives. This allows universal mach-o files to support caching multuple architectures in the same module cache directory. Making the file based on the this info allows this cache file to be deleted and replaced when the file gets updated on disk. This keeps the cache from growing over time during the compile/edit/debug cycle and prevents out of space issues.

If the cache is enabled, the symbol table will be loaded from the cache the next time you debug if the module has not changed.

The cache also has settings to control the size of the cache on disk. Each time LLDB starts up with the index cache enable, the cache will be pruned to ensure it stays within the user defined settings:

(lldb) settings set symbols.lldb-index-cache-expiration-days <days>

A value of zero will disable cache files from expiring when the cache is pruned. The default value is 7 currently.

(lldb) settings set symbols.lldb-index-cache-max-byte-size <size>

A value of zero will disable pruning based on a total byte size. The default value is zero currently.
(lldb) settings set symbols.lldb-index-cache-max-percent <percentage-of-disk-space>

A value of 100 will allow the disc to be filled to the max, a value of zero will disable percentage pruning. The default value is zero.

Reviewed By: labath, wallace

Differential Revision: https://reviews.llvm.org/D115324

show more ...

[lldb][NFC] avoid unnecessary roundtrips between different string types

The amount of roundtrips between StringRefs, ConstStrings and std::strings is
getting a bit out of hand, this patch avoid the

[lldb][NFC] avoid unnecessary roundtrips between different string types

The amount of roundtrips between StringRefs, ConstStrings and std::strings is
getting a bit out of hand, this patch avoid the unnecessary roundtrips.

Reviewed By: wallace, aprantl

Differential Revision: https://reviews.llvm.org/D112863

show more ...

[lldb][NFC] Modernize for-loops in ModuleList

Reviewed By: mib

Differential Revision: https://reviews.llvm.org/D112379

[lldb] Use a struct to pass function search options to Module::FindFunction

Rather than passing two booleans around, which is especially error prone
with them being next to each other, use a struct

[lldb] Use a struct to pass function search options to Module::FindFunction

Rather than passing two booleans around, which is especially error prone
with them being next to each other, use a struct with named fields
instead.

Differential revision: https://reviews.llvm.org/D107295

show more ...

[lldb] Use C++11 default member initializers

This converts a default constructor's member initializers into C++11
default member initializers. This patch was automatically generated with
clang-tidy

[lldb] Use C++11 default member initializers

This converts a default constructor's member initializers into C++11
default member initializers. This patch was automatically generated with
clang-tidy and the modernize-use-default-member-init check.

$ run-clang-tidy.py -header-filter='lldb' -checks='-*,modernize-use-default-member-init' -fix

This is a mass-refactoring patch and this commit will be added to
.git-blame-ignore-revs.

Differential revision: https://reviews.llvm.org/D103483

show more ...

[lldb] Access the ModuleList through iterators where possible (NFC)

Replace uses of GetModuleAtIndexUnlocked and
GetModulePointerAtIndexUnlocked with the ModuleIterable and
ModuleIterableNoLocking w

[lldb] Access the ModuleList through iterators where possible (NFC)

Replace uses of GetModuleAtIndexUnlocked and
GetModulePointerAtIndexUnlocked with the ModuleIterable and
ModuleIterableNoLocking where applicable.

Differential revision: https://reviews.llvm.org/D94271

show more ...

[lldb] Fix that symbols.clang-modules-cache-path is never initialized

LLDB is supposed to ask the Clang Driver what the default module cache path is
and then use that value as the default for the
`s

[lldb] Fix that symbols.clang-modules-cache-path is never initialized

LLDB is supposed to ask the Clang Driver what the default module cache path is
and then use that value as the default for the
`symbols.clang-modules-cache-path` setting. However, we use the property type
`String` to change `symbols.clang-modules-cache-path` even though the type of
that setting is `FileSpec`, so the setter will simply do nothing and return
`false`. We also don't check the return value of the setter, so this whole code
ends up not doing anything at all.

This changes the setter to use the correct property type and adds an assert that
we actually successfully set the default path. Also adds a test that checks that
the default value for this setting is never unset/empty path as this would
effectively disable the import-std-module feature from working by default.

Reviewed By: JDevlieghere, shafik

Differential Revision: https://reviews.llvm.org/D92772

show more ...

[lldb] Report old modules from ModuleList::ReplaceEquivalent

This allows the Target to update its module list when loading a shared
module replaces an equivalent one.

A testcase is added which hits

[lldb] Report old modules from ModuleList::ReplaceEquivalent

This allows the Target to update its module list when loading a shared
module replaces an equivalent one.

A testcase is added which hits this codepath -- without the fix, the
target reports libbreakpad.so twice in its module list.

Reviewed By: jingham

Differential Revision: https://reviews.llvm.org/D89157

show more ...

[lldb] GetSharedModule: Collect old modules in SmallVector

The various GetSharedModule methods have an optional out parameter for
the old module when a file has changed or been replaced, which the
T

[lldb] GetSharedModule: Collect old modules in SmallVector

The various GetSharedModule methods have an optional out parameter for
the old module when a file has changed or been replaced, which the
Target uses to keep its module list current/correct. We've been using
a single ModuleSP to track "the" old module, and this change switches
to using a SmallVector of ModuleSP, which has a couple benefits:
- There are multiple codepaths which may discover an old module, and
this centralizes the code for how to handle multiples in one place,
in the Target code. With the single ModuleSP, each place that may
discover an old module is responsible for how it handles multiples,
and the current code is inconsistent (some code paths drop the first
old module, others drop the second).
- The API will be more natural for identifying old modules in routines
that work on sets, like ModuleList::ReplaceEquivalent (which I plan
on updating to report old module(s) in a subsequent change to fix a
bug).

I'm not convinced we can ever actually run into the case that multiple
old modules are found in the same GetOrCreateModule call, but I think
this change makes sense regardless, in light of the above.

When an old module is reported, Target::GetOrCreateModule calls
m_images.ReplaceModule, which doesn't allow multiple "old" modules; the
new code calls ReplaceModule for the first "old" module, and for any
subsequent old modules it logs the event and calls m_images.Remove.

Reviewed By: jingham

Differential Revision: https://reviews.llvm.org/D89156

show more ...

[lldb] Remove redundant WithFormat suffixes (NFC)

Replace calls to FooWithFormat() with calls to Foo() when only one
argument is provided and the given string doesn't need to be formatted.

[lldb] Remove orphaned modules in a loop

Summary:

When modules reference each other (which happens for example with the different
modules LLDB loads when debugging -gmodules-compiled binaries), jus

[lldb] Remove orphaned modules in a loop

Summary:

When modules reference each other (which happens for example with the different
modules LLDB loads when debugging -gmodules-compiled binaries), just iterating
over the module list once isn't good enough to find all orphans. Any removed
modules in the module list will also clear up the shared pointers they hold to
other modules, so after any module was removed from the list, LLDB should
iterate again and check if any additional modules can no be safely deleted.

This is currently causing that many gmodules tests are not cleaning up all
allocated modules which causes cleanup asserts to fail (right now these asserts
just mark the test as unsupported, but after D83865 the tests will start
failing).

Reviewers: aprantl, clayborg, JDevlieghere

Reviewed By: JDevlieghere

Differential Revision: https://reviews.llvm.org/D84015

show more ...

[lldb][NFC] Make all CompilerDeclContext parameters references instead of pointers

Summary:
All of our lookup APIs either use `CompilerDeclContext &` or `CompilerDeclContext *` semi-randomly it seem

[lldb][NFC] Make all CompilerDeclContext parameters references instead of pointers

Summary:
All of our lookup APIs either use `CompilerDeclContext &` or `CompilerDeclContext *` semi-randomly it seems.
This leads to us constantly converting between those two types (and doing nullptr checks when going from
pointer to reference). It also leads to the confusing situation where we have two possible ways to express
that we don't have a CompilerDeclContex: either a nullptr or an invalid CompilerDeclContext (aka a default
constructed CompilerDeclContext).

This moves all APIs to use references and gets rid of all the nullptr checks and conversions.

Reviewers: labath, mib, shafik

Reviewed By: labath, shafik

Subscribers: shafik, arphaman, abidh, JDevlieghere, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D74607

show more ...

[lldb][NFC] Remove several inefficient ConstString -> const char * -> StringRef conversions

StringRef will call strlen on the C string which is inefficient (as ConstString already
knows the string l

[lldb][NFC] Remove several inefficient ConstString -> const char * -> StringRef conversions

StringRef will call strlen on the C string which is inefficient (as ConstString already
knows the string lenght and so does StringRef). This patch replaces all those calls
with GetStringRef() which doesn't recompute the length.

show more ...

[lldb][NFCI] Delete commented out code

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp ----------------------------------------

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp -------------------------------------------------===//
```
However in LLDB most of our source files have arbitrary changes to this format and
these changes are spreading through LLDB as folks usually just use the existing
source files as templates for their new files (most notably the unnecessary
editor language indicator `-*- C++ -*-` is spreading and in every review
someone is pointing out that this is wrong, resulting in people pointing out that this
is done in the same way in other files).

This patch removes most of these inconsistencies including the editor language indicators,
all the different missing/additional '-' characters, files that center the file name, missing
trailing `===//` (mostly caused by clang-format breaking the line).

Reviewers: aprantl, espindola, jfb, shafik, JDevlieghere

Reviewed By: JDevlieghere

Subscribers: dexonsmith, wuzish, emaste, sdardis, nemanjai, kbarton, MaskRay, atanasyan, arphaman, jfb, abidh, jsji, JDevlieghere, usaxena95, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D73258

show more ...

[lldb] Don't process symlinks deep inside DWARFUnit

Summary:
This code is handling debug info paths starting with /proc/self/cwd,
which is one of the mechanisms people use to obtain "relocatable" de

[lldb] Don't process symlinks deep inside DWARFUnit

Summary:
This code is handling debug info paths starting with /proc/self/cwd,
which is one of the mechanisms people use to obtain "relocatable" debug
info (the idea being that one starts the debugger with an appropriate
cwd and things "just work").

Instead of resolving the symlinks inside DWARFUnit, we can do the same
thing more elegantly by hooking into the existing Module path remapping
code. Since llvm::DWARFUnit does not support any similar functionality,
doing things this way is also a step towards unifying llvm and lldb
dwarf parsers.

Reviewers: JDevlieghere, aprantl, clayborg, jdoerfert

Subscribers: lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D71770

show more ...

Add a testcase for Clang modules being updated within one LLDB session.

This actually works as expected, but wasn't explicitly tested before.

Modernize the rest of the Find.* API (NFC)

This patch removes the size_t return value and the append parameter
from the remainder of the Find.* functions in LLDB's internal API. As
in the previous p

Modernize the rest of the Find.* API (NFC)

This patch removes the size_t return value and the append parameter
from the remainder of the Find.* functions in LLDB's internal API. As
in the previous patches, this is motivated by the fact that these
parameters aren't really used, and in the case of the append parameter
were frequently implemented incorrectly.

Differential Revision: https://reviews.llvm.org/D69119

llvm-svn: 375160

show more ...

Remove size_t return parameter from FindTypes

In r368345 I accidentally introduced a regression that would
over-report the number of matches found by FindTypes if the
DeclContext Filter was hit.

Th

Remove size_t return parameter from FindTypes

In r368345 I accidentally introduced a regression that would
over-report the number of matches found by FindTypes if the
DeclContext Filter was hit.

This patch simply removes the size_t return parameter altogether —
it's not that useful.

rdar://problem/55500457

Differential Revision: https://reviews.llvm.org/D68169

llvm-svn: 373344

show more ...

[lldb] Qualify includes of Properties[Enum].inc files. NFC

Summary:
This is a bit more explicit, and makes it possible to build LLDB without
varying the -I lines per-directory.
(The latter is useful

[lldb] Qualify includes of Properties[Enum].inc files. NFC

Summary:
This is a bit more explicit, and makes it possible to build LLDB without
varying the -I lines per-directory.
(The latter is useful because many build systems only allow this to be
configured per-library, and LLDB is insufficiently layered to be split into
multiple libraries on stricter build systems).

(My comment on D65185 has some more context)

Reviewers: JDevlieghere, labath, chandlerc, jdoerfert

Reviewed By: labath

Subscribers: mgorny, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D65397

Patch by Sam McCall!

llvm-svn: 367241

show more ...

[lldb] Also include the array definition in Properties.inc

Right now our Properties.inc only generates the initializer for the
options list but not the array declaration boilerplate around it. As th

[lldb] Also include the array definition in Properties.inc

Right now our Properties.inc only generates the initializer for the
options list but not the array declaration boilerplate around it. As the
array definition is identical for all arrays, we might as well also let
the Properties.inc generate it alongside the initializers.

Unfortunately we cannot do the same for enums, as there's this magic
ePropertyExperimental, which needs to come at the end to be interpreted
correctly. Hopefully we can get rid of this in the future and do the
same for the property enums.

Differential revision: https://reviews.llvm.org/D65353

llvm-svn: 367238

show more ...