Revert "[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute." and follow-ups

This reverts commit 9429b67b8e300e638d7828bbcb95585f85c4df4d.

It broke

Revert "[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute." and follow-ups

This reverts commit 9429b67b8e300e638d7828bbcb95585f85c4df4d.

It broke the build on Windows, see comments on https://reviews.llvm.org/D130309

It also reverts these follow-ups:

Revert "Fix buildbot breakage after https://reviews.llvm.org/D130309."
This reverts commit f959d815f4637890ebbacca379f1c38ab47e4e14.

Revert "Fix buildbot breakage after https://reviews.llvm.org/D130309."
This reverts commit 0bbce7a4c2d2bff622bdadd4323f93f5d90e6d24.

Revert "Cache the value for absolute path in FileSpec."
This reverts commit dabe877248b85b34878e75d5510339325ee087d0.

show more ...

[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute.

The FileSpect APIs allow users to modify instance variables directly by getting a non const ref

[NFC] Improve FileSpec internal APIs and usage in preparation for adding caching of resolved/absolute.

The FileSpect APIs allow users to modify instance variables directly by getting a non const reference to the directory and filename instance variables. This makes it impossibly to control all of the times the FileSpec object is modified so we can clear the cache. This patch modifies the APIs of FileSpec so no one can modify the directory or filename directly by adding set accessors and by removing the get accessors that are non const.

Many clients were using FileSpec::GetCString(...) which returned a unique C string from a ConstString'ified version of the result of GetPath() which returned a std::string. This caused many locations to use this convenient function incorrectly and could cause many strings to be added to the constant string pool that didn't need to. Most clients were converted to using FileSpec::GetPath().c_str() when possible. Other clients were modified to use the newly renamed version of this function which returns an actualy ConstString:
ConstString FileSpec::GetPathAsConstString(bool denormalize = true) const;

This avoids the issue where people were getting an already uniqued "const char *" that came from a ConstString only to put the "const char *" back into a "ConstString" object. By returning the ConstString instead of a "const char *" clients can be more efficient with the result.

The patch:
- Removes the non const GetDirectory() and GetFilename() get accessors
- Adds set accessors to replace the above functions: SetDirectory() and SetFilename().
- Adds ClearDirectory() and ClearFilename() to replace usage of the FileSpec::GetDirectory().Clear()/FileSpec::GetFilename().Clear() call sites
- Fixed all incorrect usage of FileSpec::GetCString() to use FileSpec::GetPath().c_str() where appropriate, and updated other call sites that wanted a ConstString to use the newly returned ConstString appropriately and efficiently.

Differential Revision: https://reviews.llvm.org/D130309

show more ...

[lldb] Refactor DataBuffer so we can map files as read-only

Currently, all data buffers are assumed to be writable. This is a
problem on macOS where it's not allowed to load unsigned binaries in
mem

[lldb] Refactor DataBuffer so we can map files as read-only

Currently, all data buffers are assumed to be writable. This is a
problem on macOS where it's not allowed to load unsigned binaries in
memory as writable. To be more precise, MAP_RESILIENT_CODESIGN and
MAP_RESILIENT_MEDIA need to be set for mapped (unsigned) binaries on our
platform.

Binaries are mapped through FileSystem::CreateDataBuffer which returns a
DataBufferLLVM. The latter is backed by a llvm::WritableMemoryBuffer
because every DataBuffer in LLDB is considered to be writable. In order
to use a read-only llvm::MemoryBuffer I had to split our abstraction
around it.

This patch distinguishes between a DataBuffer (read-only) and
WritableDataBuffer (read-write) and updates LLDB to use the appropriate
one.

rdar://74890607

Differential revision: https://reviews.llvm.org/D122856

show more ...

[lldb] Return a DataBuffer from FileSystem::CreateDataBuffer (NFC)

The concrete class (DataBufferLLVM) is an implementation detail.

revert "[lldb/Host] Fix crash in FileSystem::IsLocal"

This reverts commit 2dc6e906b09deb092c15a726c06d0ecbeec1eb18 following
changes introduced in 59eb70527741594fe3c92d0a1b6704ed45111437.

[lldb] Remove FileSystem::Initialize from FileCollector

This patch removes the ability to instantiate the LLDB FileSystem class
with a FileCollector. It keeps the ability to collect files, but uses

[lldb] Remove FileSystem::Initialize from FileCollector

This patch removes the ability to instantiate the LLDB FileSystem class
with a FileCollector. It keeps the ability to collect files, but uses
the FileCollectorFileSystem to do that transparently.

Because the two are intertwined, this patch also removes the
finalization logic which copied the files over out of process.

show more ...

[lldb] Remove FileSystem::Initialize from VFS mapping

This patch removes the ability to instantiate the LLDB FileSystem class
based on a VFS overlay. This also removes the "hack" where we cast the
V

[lldb] Remove FileSystem::Initialize from VFS mapping

This patch removes the ability to instantiate the LLDB FileSystem class
based on a VFS overlay. This also removes the "hack" where we cast the
VFS to a RedirectingFileSystem to obtain the external path. You can
still instantiate a FileSystem with a VFS, but with the caveat that
operations that rely on the external path won't work.

Differential revision: https://reviews.llvm.org/D120923

show more ...

[lldb/Host] Fix crash in FileSystem::IsLocal

This checks `m_fs` before dereferencing it to access its`isLocal` method.

rdar://67410058

Signed-off-by: Med Ismail Bennani <[email protected]

[lldb/Host] Fix crash in FileSystem::IsLocal

This checks `m_fs` before dereferencing it to access its`isLocal` method.

rdar://67410058

Signed-off-by: Med Ismail Bennani <[email protected]>

show more ...

Added the ability to cache the finalized symbol tables subsequent debug sessions to start faster.

This is an updated version of the https://reviews.llvm.org/D113789 patch with the following changes:

Added the ability to cache the finalized symbol tables subsequent debug sessions to start faster.

This is an updated version of the https://reviews.llvm.org/D113789 patch with the following changes:
- We no longer modify modification times of the cache files
- Use LLVM caching and cache pruning instead of making a new cache mechanism (See DataFileCache.h/.cpp)
- Add signature to start of each file since we are not using modification times so we can tell when caches are stale and remove and re-create the cache file as files are changed
- Add settings to control the cache size, disk percentage and expiration in days to keep cache size under control

This patch enables symbol tables to be cached in the LLDB index cache directory. All cache files are in a single directory and the files use unique names to ensure that files from the same path will re-use the same file as files get modified. This means as files change, their cache files will be deleted and updated. The modification time of each of the cache files is not modified so that access based pruning of the cache can be implemented.

The symbol table cache files start with a signature that uniquely identifies a file on disk and contains one or more of the following items:
- object file UUID if available
- object file mod time if available
- object name for BSD archive .o files that are in .a files if available

If none of these signature items are available, then the file will not be cached. This keeps temporary object files from expressions from being cached.

When the cache files are loaded on subsequent debug sessions, the signature is compare and if the file has been modified (uuid changes, mod time changes, or object file mod time changes) then the cache file is deleted and re-created.

Module caching must be enabled by the user before this can be used:

symbols.enable-lldb-index-cache (boolean) = false

(lldb) settings set symbols.enable-lldb-index-cache true

There is also a setting that allows the user to specify a module cache directory that defaults to a directory that defaults to being next to the symbols.clang-modules-cache-path directory in a temp directory:

(lldb) settings show symbols.lldb-index-cache-path
/var/folders/9p/472sr0c55l9b20x2zg36b91h0000gn/C/lldb/IndexCache

If this setting is enabled, the finalized symbol tables will be serialized and saved to disc so they can be quickly loaded next time you debug.

Each module can cache one or more files in the index cache directory. The cache file names must be unique to a file on disk and its architecture and object name for .o files in BSD archives. This allows universal mach-o files to support caching multuple architectures in the same module cache directory. Making the file based on the this info allows this cache file to be deleted and replaced when the file gets updated on disk. This keeps the cache from growing over time during the compile/edit/debug cycle and prevents out of space issues.

If the cache is enabled, the symbol table will be loaded from the cache the next time you debug if the module has not changed.

The cache also has settings to control the size of the cache on disk. Each time LLDB starts up with the index cache enable, the cache will be pruned to ensure it stays within the user defined settings:

(lldb) settings set symbols.lldb-index-cache-expiration-days <days>

A value of zero will disable cache files from expiring when the cache is pruned. The default value is 7 currently.

(lldb) settings set symbols.lldb-index-cache-max-byte-size <size>

A value of zero will disable pruning based on a total byte size. The default value is zero currently.
(lldb) settings set symbols.lldb-index-cache-max-percent <percentage-of-disk-space>

A value of 100 will allow the disc to be filled to the max, a value of zero will disable percentage pruning. The default value is zero.

Reviewed By: labath, wallace

Differential Revision: https://reviews.llvm.org/D115324

show more ...

[lldb] [gdb-remote] Add eOpenOptionReadWrite for future gdb compat

Modify OpenOptions enum to open the future path into synchronizing
vFile:open bits with GDB. Currently, LLDB and GDB use different

[lldb] [gdb-remote] Add eOpenOptionReadWrite for future gdb compat

Modify OpenOptions enum to open the future path into synchronizing
vFile:open bits with GDB. Currently, LLDB and GDB use different flag
models effectively making it impossible to match bits. Notably, LLDB
uses two bits to indicate read and write status, and uses union of both
for read/write. GDB uses a value of 0 for read-only, 1 for write-only
and 2 for read/write.

In order to future-proof the code for the GDB variant:

1. Add a distinct eOpenOptionReadWrite constant to be used instead
of (eOpenOptionRead | eOpenOptionWrite) when R/W access is required.

2. Rename eOpenOptionRead and eOpenOptionWrite to eOpenOptionReadOnly
and eOpenOptionWriteOnly respectively, to make it clear that they
do not mean to be combined and require update to all call sites.

3. Use the intersection of all three flags when matching against
the three possible values.

This commit does not change the actual bits used by LLDB.

Differential Revision: https://reviews.llvm.org/D106984

show more ...

[lldb][NFC] Use C++ versions of the deprecated C standard library headers

The C headers are deprecated so as requested in D102845, this is replacing them
all with their (not deprecated) C++ equivale

[lldb][NFC] Use C++ versions of the deprecated C standard library headers

The C headers are deprecated so as requested in D102845, this is replacing them
all with their (not deprecated) C++ equivalent.

Reviewed By: shafik

Differential Revision: https://reviews.llvm.org/D103084

show more ...

[NFC] Reordering parameters in getFile and getFileOrSTDIN

In future patches I will be setting the IsText parameter frequently so I will refactor the args to be in the following order. I have removed

[NFC] Reordering parameters in getFile and getFileOrSTDIN

In future patches I will be setting the IsText parameter frequently so I will refactor the args to be in the following order. I have removed the FileSize parameter because it is never used.

```
static ErrorOr<std::unique_ptr<MemoryBuffer>>
getFile(const Twine &Filename, bool IsText = false,
bool RequiresNullTerminator = true, bool IsVolatile = false);

static ErrorOr<std::unique_ptr<MemoryBuffer>>
getFileOrSTDIN(const Twine &Filename, bool IsText = false,
bool RequiresNullTerminator = true);

static ErrorOr<std::unique_ptr<MB>>
getFileAux(const Twine &Filename, uint64_t MapSize, uint64_t Offset,
bool IsText, bool RequiresNullTerminator, bool IsVolatile);

static ErrorOr<std::unique_ptr<WritableMemoryBuffer>>
getFile(const Twine &Filename, bool IsVolatile = false);
```

Reviewed By: jhenderson

Differential Revision: https://reviews.llvm.org/D99182

show more ...

[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

Previously file entries in the -ivfsoverlay yaml could map to a file in the
external file system

[VFS] Add support to RedirectingFileSystem for mapping a virtual directory to one in the external FS.

Previously file entries in the -ivfsoverlay yaml could map to a file in the
external file system, but directories had to list their contents in the form of
other file entries or directories. Allowing directory entries to map to a
directory in the external file system makes it possible to present an external
directory's contents in a different location and (in combination with the
'fallthrough' option) overlay one directory's contents on top of another.

rdar://problem/72485443
Differential Revision: https://reviews.llvm.org/D94844

show more ...

[VFS] Combine VFSFromYamlDirIterImpl and OverlayFSDirIterImpl into a single implementation (NFC)

As a fixme notes, both of these directory iterator implementations are
conceptually similar and dupli

[VFS] Combine VFSFromYamlDirIterImpl and OverlayFSDirIterImpl into a single implementation (NFC)

As a fixme notes, both of these directory iterator implementations are
conceptually similar and duplicate the functionality of returning and uniquing
entries across two or more directories. This patch combines them into a single
class 'CombiningDirIterImpl'.

This also drops the 'Redirecting' prefix from RedirectingDirEntry and
RedirectingFileEntry to save horizontal space. There's no loss of clarity as
they already have to be prefixed with 'RedirectingFileSystem::' whenever
they're referenced anyway.

rdar://problem/72485443
Differential Revision: https://reviews.llvm.org/D94857

show more ...

[lldb] FixFileSystem::GetExternalPath for VFS API change

[lldb] Move copying of files into reproducer out of process

For performance reasons the reproducers don't copy the files captured by
the file collector eagerly, but wait until the reproducer needs t

[lldb] Move copying of files into reproducer out of process

For performance reasons the reproducers don't copy the files captured by
the file collector eagerly, but wait until the reproducer needs to be
generated.

This is a problematic when LLDB crashes and we have to do all this
signal-unsafe work in the signal handler. This patch uses a similar
trick to clang, which has the driver invoke a new cc1 instance to do all
this work out-of-process.

This patch moves the writing of the mapping file as well as copying over
the reproducers into a separate process spawned when lldb crashes.

Differential revision: https://reviews.llvm.org/D89600

show more ...

[lldb] Capture and load home directory from the reproducer.

When replaying the reproducer, lldb should source the .lldbinit file
that was captured by the reproducer and not the one in the current ho

[lldb] Capture and load home directory from the reproducer.

When replaying the reproducer, lldb should source the .lldbinit file
that was captured by the reproducer and not the one in the current home
directory. This requires that we store the home directory as part of the
reproducer. By returning the virtual home directory during replay, we
ensure the correct virtual path gets constructed which the VFS can then
find and remap to the correct file in the reproducer root.

This patch adds a new HomeDirectoryProvider, similar to the existing
WorkingDirectoryProvider. As the home directory is not part of the VFS,
it is stored in LLDB's FileSystem instance.

show more ...

[lldb] Provide GetHomeDirectory wrapper in Host::FileSystem (NFC)

Provider a wrapper around llvm::sys::path::home_directory in the
FileSystem class. This will make it possible for the reproducers to

[lldb] Provide GetHomeDirectory wrapper in Host::FileSystem (NFC)

Provider a wrapper around llvm::sys::path::home_directory in the
FileSystem class. This will make it possible for the reproducers to
intercept the call in a central place.

show more ...

[lldb/Reproducers] Also record directories FileSystem::Collect.

Now that the FileCollector knows how to deal with directories we no
longer have to ignore them in the FileSystem class.

[lldb/Support] Treat empty FileSpec as an invalid file.

LLDB relies on empty FileSpecs being invalid files, for example, they
don't exists. Currently this assumption does not always hold during
repr

[lldb/Support] Treat empty FileSpec as an invalid file.

LLDB relies on empty FileSpecs being invalid files, for example, they
don't exists. Currently this assumption does not always hold during
reproducer replay, because we pass the result of GetPath to the VFS.
This is an empty string, which the VFS converts to an absolute directory
by prepending the current working directory, before looking it up in the
YAML mapping. This means that an empty FileSpec will exist when the
current working directory does. This breaks at least one test
(TestAddDsymCommand.py) when ran from replay.

This patch special cases empty FileSpecs and returns a sensible result
before calling GetPath and forwarding the call.

Differential revision: https://reviews.llvm.org/D77351

show more ...

[lldb/Reproducers] Collect files imported by command script import

Files imported by the script interpreter aren't opened by LLDB so they
don't end up in the reproducer. The solution is to explicitl

[lldb/Reproducers] Collect files imported by command script import

Files imported by the script interpreter aren't opened by LLDB so they
don't end up in the reproducer. The solution is to explicitly add them
to the FileCollector.

Differential revision: https://reviews.llvm.org/D76626

show more ...

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp ----------------------------------------

[lldb][NFC] Fix all formatting errors in .cpp file headers

Summary:
A *.cpp file header in LLDB (and in LLDB) should like this:
```
//===-- TestUtilities.cpp -------------------------------------------------===//
```
However in LLDB most of our source files have arbitrary changes to this format and
these changes are spreading through LLDB as folks usually just use the existing
source files as templates for their new files (most notably the unnecessary
editor language indicator `-*- C++ -*-` is spreading and in every review
someone is pointing out that this is wrong, resulting in people pointing out that this
is done in the same way in other files).

This patch removes most of these inconsistencies including the editor language indicators,
all the different missing/additional '-' characters, files that center the file name, missing
trailing `===//` (mostly caused by clang-format breaking the line).

Reviewers: aprantl, espindola, jfb, shafik, JDevlieghere

Reviewed By: JDevlieghere

Subscribers: dexonsmith, wuzish, emaste, sdardis, nemanjai, kbarton, MaskRay, atanasyan, arphaman, jfb, abidh, jsji, JDevlieghere, usaxena95, lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D73258

show more ...

[lldb/Utility] Don't forward directories to the file collector

The VFS mapping writer assumes that all the paths it gets are files.
When passed a directory, it ends up as a file in the VFS mapping t

[lldb/Utility] Don't forward directories to the file collector

The VFS mapping writer assumes that all the paths it gets are files.
When passed a directory, it ends up as a file in the VFS mapping twice,
once as a file and once as a directory.

{
'type': 'file',
'name': "Output",
'external-contents': "/root/path/to/Output"
},
{
'type': 'directory',
'name': "Output",
'contents': [ ... ]
}

show more ...

uint32_t options -> File::OpenOptions options

Summary:
This patch re-types everywhere that passes a File::OpenOptions
as a uint32_t so it actually uses File::OpenOptions.

It also converts some Open

uint32_t options -> File::OpenOptions options

Summary:
This patch re-types everywhere that passes a File::OpenOptions
as a uint32_t so it actually uses File::OpenOptions.

It also converts some OpenOptions related functions that fail
by returning 0 or NULL into llvm::Expected

split off from https://reviews.llvm.org/D68737

Reviewers: JDevlieghere, jasonmolenda, labath

Reviewed By: labath

Subscribers: lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D68853

llvm-svn: 374817

show more ...

factor out an abstract base class for File

Summary:
This patch factors out File as an abstract base
class and moves most of its actual functionality into
a subclass called NativeFile. In the next

factor out an abstract base class for File

Summary:
This patch factors out File as an abstract base
class and moves most of its actual functionality into
a subclass called NativeFile. In the next patch,
I'm going to be adding subclasses of File that
don't necessarily have any connection to actual OS files,
so they will not inherit from NativeFile.

This patch was split out as a prerequisite for
https://reviews.llvm.org/D68188

Reviewers: JDevlieghere, jasonmolenda, labath

Reviewed By: labath

Subscribers: lldb-commits

Tags: #lldb

Differential Revision: https://reviews.llvm.org/D68317

llvm-svn: 373564

show more ...