Introduce new symbol on-demand for debug info

This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant perfo

Introduce new symbol on-demand for debug info

This diff introduces a new symbol on-demand which skips
loading a module's debug info unless explicitly asked on
demand. This provides significant performance improvement
for application with dynamic linking mode which has large
number of modules.
The feature can be turned on with:
"settings set symbols.load-on-demand true"

The feature works by creating a new SymbolFileOnDemand class for
each module which wraps the actual SymbolFIle subclass as member
variable. By default, most virtual methods on SymbolFileOnDemand are
skipped so that it looks like there is no debug info for that module.
But once the module's debug info is explicitly requested to
be enabled (in the conditions mentioned below) SymbolFileOnDemand
will allow all methods to pass through and forward to the actual SymbolFile
which would hydrate module's debug info on-demand.

In an internal benchmark, we are seeing more than 95% improvement
for a 3000 modules application.

Currently we are providing several ways to on demand hydrate
a module's debug info:
* Source line breakpoint: matching in supported files
* Stack trace: resolving symbol context for an address
* Symbolic breakpoint: symbol table match guided promotion
* Global variable: symbol table match guided promotion

In all above situations the module's debug info will be on-demand
parsed and indexed.

Some follow-ups for this feature:
* Add a command that allows users to load debug info explicitly while using a
new or existing command when this feature is enabled
* Add settings for "never load any of these executables in Symbols On Demand"
that takes a list of globs
* Add settings for "always load the the debug info for executables in Symbols
On Demand" that takes a list of globs
* Add a new column in "image list" that shows up by default when Symbols On
Demand is enable to show the status for each shlib like "not enabled for
this", "debug info off" and "debug info on" (with a single character to
short string, not the ones I just typed)

Differential Revision: https://reviews.llvm.org/D121631

show more ...

[LLDB] Remove cases of using namespace llvm:: from header file

We have using namespace llvm::dwarf in dwarf.h header globally. Replacing that
with a using namespace within lldb_private::dwarf and mo

[LLDB] Remove cases of using namespace llvm:: from header file

We have using namespace llvm::dwarf in dwarf.h header globally. Replacing that
with a using namespace within lldb_private::dwarf and moving to a
using namespace lldb_private::dwarf in .cpp files and fully qualified names
in the few header files.

Differential Revision: https://reviews.llvm.org/D120836

show more ...

[lldb] Convert DWARF log to the new API

This also deletes some dead log statements (log initialization commented
out).

[nfc] [lldb] Assertions for D106270 - [DWARF5] Fix offset check when using .debug_names

Skeleton vs. DWO units mismatch has been fixed in D106270. As they both
have type DWARFUnit it is a bit diffic

[nfc] [lldb] Assertions for D106270 - [DWARF5] Fix offset check when using .debug_names

Skeleton vs. DWO units mismatch has been fixed in D106270. As they both
have type DWARFUnit it is a bit difficult to debug. So it is better to
make it safe against future changes.

Reviewed By: kimanh, clayborg

Differential Revision: https://reviews.llvm.org/D107659

show more ...

[DWARF5] Fix offset check when using .debug_names

When going through the CU entries in the name index,
make sure to compare the name entry's CU
offset against the skeleton CU's offset.

Previously t

[DWARF5] Fix offset check when using .debug_names

When going through the CU entries in the name index,
make sure to compare the name entry's CU
offset against the skeleton CU's offset.

Previously there would be a mismatch, since the
wrong offset was compared, and thus no suitable
entry was found.

Reviewed By: jankratochvil

Differential Revision: https://reviews.llvm.org/D106270

show more ...

[DWARF5] Only fallback to manual index if no entry was found

If we succeed at gathering global variables for a compile
unit, there is no need to fallback to generating a manual index.

Reviewed By:

[DWARF5] Only fallback to manual index if no entry was found

If we succeed at gathering global variables for a compile
unit, there is no need to fallback to generating a manual index.

Reviewed By: jankratochvil

Differential Revision: https://reviews.llvm.org/D106355

show more ...

[nfc] [lldb] DWARF callbacks: DIERef -> DWARFDIE

Pavel Labath wrote in D73206:
The internal representation of DebugNames and Apple indexes is fixed by
the relevant (pseudo-)standards, so we can't re

[nfc] [lldb] DWARF callbacks: DIERef -> DWARFDIE

Pavel Labath wrote in D73206:
The internal representation of DebugNames and Apple indexes is fixed by
the relevant (pseudo-)standards, so we can't really change it. The
question is how to efficiently (and cleanly) convert from the internal
representation to some common thing. The conversion from AppleIndex to
DIERef is trivial (which is not surprising as it was the first and the
overall design was optimized for that). With debug_names, the situation
gets more tricky. The internal representation of debug_names uses
CU-relative DIE offsets, but DIERef wants an absolute offset. That means
the index has to do more work to produce the common representation. And
it needs to do that for all results, even though a lot of the index
users are really interested only in a single entry. With the switch to
user_id_t, _all_ indexes would have to do some extra work to encode it,
only for their users to have to immediately decode it back. Having
a iterator/callback based api would allow us to minimize the impact of
that, as it would only need to happen for the entries that are really
used. And /I think/ we could make it interface returns DWARFDies
directly, and each index converts to that using the most direct approach
available.

Jan Kratochvil:
It also makes all the callers shorter as they no longer need to fetch
DWARFDIE from DIERef (and handling if not found by ReportInvalidDIERef)
but the callers are already served DWARFDIE which they need.
In some cases the DWARFDIE had to be fetched both by callee (DWARFIndex
implementation) and caller.

Differential Revision: https://reviews.llvm.org/D77970

show more ...

[nfc] [lldb] Introduce DWARF callbacks (fixed-up)

As requested by @labath in https://reviews.llvm.org/D73206#1949516
providing DWARF index callbacks refactorization.

It is a re-landing of the patch

[nfc] [lldb] Introduce DWARF callbacks (fixed-up)

As requested by @labath in https://reviews.llvm.org/D73206#1949516
providing DWARF index callbacks refactorization.

It is a re-landing of the patch with a regression fix the previous
commit had.

Differential Revision: https://reviews.llvm.org/D77327

show more ...

Revert "[nfc] [lldb] Introduce DWARF callbacks"

This reverts commit bd47c470d13b1c57ecf37c1faf0324833d3a4542.

It broke Green Dragon, reason is unknown to me so far:
http://green.lab.llvm.org/gree

Revert "[nfc] [lldb] Introduce DWARF callbacks"

This reverts commit bd47c470d13b1c57ecf37c1faf0324833d3a4542.

It broke Green Dragon, reason is unknown to me so far:
http://green.lab.llvm.org/green/view/LLDB/job/lldb-cmake/15323/consoleFull

Differential Revision: https://reviews.llvm.org/D77327

show more ...

[nfc] [lldb] Introduce DWARF callbacks

As requested by @labath in https://reviews.llvm.org/D73206#1949516
providing DWARF index callbacks refactorization.

Differential Revision: https://reviews.llv

[nfc] [lldb] Introduce DWARF callbacks

As requested by @labath in https://reviews.llvm.org/D73206#1949516
providing DWARF index callbacks refactorization.

Differential Revision: https://reviews.llvm.org/D77327

show more ...

[nfc] [lldb] Unindent code - obvious part

It is an obvious part of D77326.

It removes some needless deep indentation and some redundant statements.
It prepares the code for a more clean next patch

[nfc] [lldb] Unindent code - obvious part

It is an obvious part of D77326.

It removes some needless deep indentation and some redundant statements.
It prepares the code for a more clean next patch - DWARF index callbacks
in D77327.

show more ...

[lldb/DWARF] Don't index dwp file multiple times

Summary:
When we added support for type units in dwo files, we changed the
"manual" dwarf index to index _all_ dwarf units in the dwo file instead
of

[lldb/DWARF] Don't index dwp file multiple times

Summary:
When we added support for type units in dwo files, we changed the
"manual" dwarf index to index _all_ dwarf units in the dwo file instead
of just the split unit belonging to our skeleton unit. This was fine for
dwo files, as they contain only a single compile units and type units do
not have a split type unit which would point to them.

However, this does not work for dwp files because, these files do
contain multiple split compile units, and the current approach means
that each unit gets indexed multiple times (once for each split unit =>
n^2 complexity).

This patch teaches the manual dwarf index to treat dwp files specially.
Any type units in the dwp file added to the main list of compile units
and indexed with them in a single batch. Split compile units in dwp
files are still indexed as a part of their skeleton unit -- this is done
because we need the DW_AT_language attribute from the skeleton unit to
index them properly.

Handling of dwo files remains unchanged -- all units (type and skeleton)
are indexed when we reach the dwo file through the split unit.

Reviewers: clayborg, JDevlieghere, aprantl

Subscribers: arphaman, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D74964

show more ...

[lldb/DWARF] Always construct a DWARFDebugInfo object

Change the return value of SymbolFileDWARF::DebugInfo from a pointer to
a reference, and remove all null checks.

Previously, we were not constr

[lldb/DWARF] Always construct a DWARFDebugInfo object

Change the return value of SymbolFileDWARF::DebugInfo from a pointer to
a reference, and remove all null checks.

Previously, we were not constructing the DebugInfo object when the
debug_info section was empty. Now we always construct the object but
it will return an empty list of dwarf units (a thing which it already
supported).

show more ...

[lldb/DWARF] Remove a workaround from DebugNamesDWARFIndex

This was needed when asking a compile unit for its dwo component
triggered a infinite recursion if the dwo unit has not been already
parsed

[lldb/DWARF] Remove a workaround from DebugNamesDWARFIndex

This was needed when asking a compile unit for its dwo component
triggered a infinite recursion if the dwo unit has not been already
parsed.

This has since been fixed.

show more ...

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp ----------------------------------------

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp -------------------------------------------------===//
```
However in LLDB most of our source files have arbitrary changes to this format and
these changes are spreading through LLDB as folks usually just use the existing
source files as templates for their new files (most notably the unnecessary
editor language indicator `-*- C++ -*-` is spreading and in every review
someone is pointing out that this is wrong, resulting in people pointing out that this
is done in the same way in other files).

This patch removes most of these inconsistencies including the editor language indicators,
all the different missing/additional '-' characters, files that center the file name, missing
trailing `===//` (mostly caused by clang-format breaking the line).

Reviewers: aprantl, espindola, jfb, shafik, JDevlieghere

Reviewed By: JDevlieghere

Subscribers: dexonsmith, wuzish, emaste, sdardis, nemanjai, kbarton, MaskRay, atanasyan, arphaman, jfb, abidh, jsji, JDevlieghere, usaxena95, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D73258

show more ...

[LLDB] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of

[LLDB] Migrate llvm::make_unique to std::make_unique

Now that we've moved to C++14, we no longer need the llvm::make_unique
implementation from STLExtras.h. This patch is a mechanical replacement
of (hopefully) all the llvm::make_unique instances across the monorepo.

Differential revision: https://reviews.llvm.org/D66259

llvm-svn: 368933

show more ...

Update LLDB to follow changes in llvm::DWARFDebugNames::NameIndex (4/5)

Differential Revision: https://reviews.llvm.org/D65640

llvm-svn: 368033

Add convenience methods to convert LLDB to LLVM data structures.

This patch adds two convenience methods named GetAsLLVM to the LLDB
counterparts of the DWARF DataExtractor and the DWARF context. Th

Add convenience methods to convert LLDB to LLVM data structures.

This patch adds two convenience methods named GetAsLLVM to the LLDB
counterparts of the DWARF DataExtractor and the DWARF context. The
DWARFContext, once created, is cached for future usage.

Differential revision: https://reviews.llvm.org/D64535

llvm-svn: 365819

show more ...

DWARF: Add "dwo_num" field to the DIERef class

Summary:
When dwo support was introduced, it used a trick where debug info
entries were referenced by the offset of the compile unit in the main
file,

DWARF: Add "dwo_num" field to the DIERef class

Summary:
When dwo support was introduced, it used a trick where debug info
entries were referenced by the offset of the compile unit in the main
file, but the die offset was relative to the dwo file. Although there
was some elegance to it, this representation was starting to reach its
breaking point:
- the fact that the skeleton compile unit owned the DWO file meant that
it was impossible (or at least hard and unintuitive) to support DWO
files containing more than one compile unit. These kinds of files are
produced by LTO for example.
- it made it impossible to reference any DIEs in the skeleton compile
unit (although the skeleton units are generally empty, clang still
puts some info into them with -fsplit-dwarf-inlining).
- (current motivation) it made it very hard to support type units placed
in DWO files, as type units don't have any skeleton units which could
be referenced in the main file

This patch addresses this problem by introducing an new
"dwo_num" field to the DIERef class, whose purpose is to identify the
dwo file. It's kind of similar to the dwo_id field in DWARF5 unit
headers, but while this is a 64bit hash whose main purpose is to catch
file mismatches, this is just a smaller integer used to indentify a
loaded dwo file. Currently, this is based on the index of the skeleton
compile unit which owns the dwo file, but it is intended to be
eventually independent of that (to support the LTO use case).

Simultaneously the cu_offset is dropped to conserve space, as it is no
longer necessary. This means we can remove the "BaseObjectOffset" field
from the DWARFUnit class. It also means we can remove some of the
workarounds put in place to support the skeleton-unit+dwo-die combo.
More work is needed to remove all of them, which is out of scope of this
patch.

Reviewers: JDevlieghere, clayborg, aprantl

Subscribers: mehdi_amini, dexonsmith, arphaman, lldb-commits

Differential Revision: https://reviews.llvm.org/D63428

llvm-svn: 364009

show more ...

DWARF: Provide accessors to DIERef fields

Summary:
Instead of accessing the fields directly, use accessor functions to
provide access to the DIERef components. This allows us to decouple the
externa

DWARF: Provide accessors to DIERef fields

Summary:
Instead of accessing the fields directly, use accessor functions to
provide access to the DIERef components. This allows us to decouple the
external interface, from the internal representation. The external
interface can use llvm::Optional and similar goodies, while the data can
still be stored internally in a more compact representation.

I also document the purpose of the existing DIERef fields.

The main motivation for this change is a need to introduce an additional
field to the DIERef class, but I believe the change has its own merit.

Reviewers: JDevlieghere, aprantl, clayborg

Subscribers: arphaman, lldb-commits

Differential Revision: https://reviews.llvm.org/D63400

llvm-svn: 363910

show more ...

DWARF: Make DIERefs always valid

Summary:
This patch makes the DIERef class always valid by default constructor
and operator bool. This allows one to express the validity of a DIERef
in the type sys

DWARF: Make DIERefs always valid

Summary:
This patch makes the DIERef class always valid by default constructor
and operator bool. This allows one to express the validity of a DIERef
in the type system. Places which are working with potentially-invalid
DIERefs have been updated to use Optional<DIERef> instead.

The constructor taking a DWARFFormValue was not needed, as all places
which were constructing a DIERef this way were immediately converting it
into a DWARFDIE or a user_id. This can be done without constructing an
intermediate DIERef.

Reviewers: JDevlieghere, clayborg, aprantl

Subscribers: arphaman, lldb-commits

Differential Revision: https://reviews.llvm.org/D63399

llvm-svn: 363767

show more ...

DWARF: Add ability to reference debug info coming from multiple sections

Summary:
This patch adds the ability to precisely address debug info in
situations when a single file can have more than one

DWARF: Add ability to reference debug info coming from multiple sections

Summary:
This patch adds the ability to precisely address debug info in
situations when a single file can have more than one debug-info-bearing
sections (as is the case with type units in DWARF v4).

The changes here can be classified into roughly three categories:
- the code which addresses a debug info by offset gets an additional
argument, which specifies the section one should look into.
- the DIERef class also gets an additional member variable specifying
the section. This way, code dealing with DIERefs can know which
section is the object referring to.
- the user_id_t encoding steals one bit from the dwarf_id field to store
the section. This means the total number of separate object files
(apple .o, or normal .dwo) is limited to 2 billion, but that is fine
as it's not possible to hit that number without switching to DWARF64
anyway.

This patch is functionally equivalent to (and inspired by) the two
patches (D61503 and D61504) by Jan Kratochvil, but there are differences
in the implementation:
- it uses an enum instead of a bool flag to differentiate the sections
- it increases the size of DIERef struct instead of reducing the amount
of addressable debug info
- it sets up DWARFDebugInfo to store the units in a single vector
instead of two. This sets us up for the future in which type units can
also live in the debug_info section, and I believe it's cleaner
because there's no need for unit index remapping

There are no tests with this patch as this is essentially NFC until
we start parsing type units from the debug_types section.

Reviewers: JDevlieghere, clayborg, aprantl

Subscribers: arphaman, jankratochvil, lldb-commits

Differential Revision: https://reviews.llvm.org/D61908

llvm-svn: 360872

show more ...

Finish renaming CompileUnit -> Unit

D42892 changed a lot of code to use superclass DWARFUnit instead of its
subclass DWARFCompileUnit.

Finish this change more thoroughly for any *CompileUnit* -> *U

Finish renaming CompileUnit -> Unit

D42892 changed a lot of code to use superclass DWARFUnit instead of its
subclass DWARFCompileUnit.

Finish this change more thoroughly for any *CompileUnit* -> *Unit* names.
Later patch will introduce DWARFTypeUnit which needs to be sometimes different
from DWARFCompileUnit and it would be confusing without this renaming.

Differential Revision: https://reviews.llvm.org/D61501

llvm-svn: 360443

show more ...

[DWARF] Reimplement/simplify DWARFUnit::GetID

Summary:
The implementation of GetID used a relatively complicated algorithm,
which returned some kind of an offset of the unit in some file
(depending

[DWARF] Reimplement/simplify DWARFUnit::GetID

Summary:
The implementation of GetID used a relatively complicated algorithm,
which returned some kind of an offset of the unit in some file
(depending on the debug info flavour). The only thing this ID was used
for was to enable subseqent retrieval of the unit from the SymbolFile.

This can be made simpler if we just make the "ID" of the unit an index
into the list of the units belonging to the symbol file. We already
support indexed access to the units, so each unit already has a well
"index" -- this just makes it accessible from within the unit.

To make the distincion between "id" and "offset" clearer (and help catch
any misuses), I also rename DWARFDebugInfo::GetCompileUnit (which
accesses by offset) into DWARFDebugInfo::GetCompileUnitAtOffset.

On its own, this only brings a minor simplification, but it enables
further simplifications in the DIERef class (coming in a follow-up
patch).

Reviewers: JDevlieghere, clayborg, aprantl

Subscribers: arphaman, jdoerfert, lldb-commits, tberghammer, jankratochvil

Differential Revision: https://reviews.llvm.org/D61481

llvm-svn: 360014

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...