LLVM Driver Multicall tool

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER opti

LLVM Driver Multicall tool

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER option to the add_llvm_executable CMake
call, and changing the tool's main function to a canonicalized
tool_name_main format (i.e. llvm_ar_main, clang_main, etc...).

As currently implemented llvm-driver contains dsymutil, llvm-ar,
llvm-cxxfilt, llvm-objcopy, and clang (if clang is included in the
build).

llvm-driver can be enabled from builds by setting
LLVM_TOOL_LLVM_DRIVER_BUILD=On.

There are several limitations in the current implementation, which can
be addressed in subsequent patches:

(1) the multicall binary cannot currently properly handle
multi-dispatch tools. This means symlinking llvm-ranlib to llvm-driver
will not properly result in llvm-ar's main being called.
(2) the multicall binary cannot be comprised of tools containing
conflicting cl::opt options as the global cl::opt option list cannot
contain duplicates.

These limitations can be addressed in subsequent patches.

Differential revision: https://reviews.llvm.org/D109977

show more ...

Fix a buglet in remove_dots().

The function promises to canonicalize the path, but neglected to do so
for the root component.

For example, calling remove_dots("/tmp/foo.c", Style::windows_backslash

Fix a buglet in remove_dots().

The function promises to canonicalize the path, but neglected to do so
for the root component.

For example, calling remove_dots("/tmp/foo.c", Style::windows_backslash)
resulted in "/tmp\foo.c". Now it produces "\tmp\foo.c".

Also fix FIXME in the corresponding test.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D126412

show more ...

Cleanup include: TableGen

This also includes a few cleanup from Support.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.l

Cleanup include: TableGen

This also includes a few cleanup from Support.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121331

show more ...

[Support] [Windows] Don't cancel delete if we failed to set delete

Following up on commit 177176f75c6fa3f624d6d964b9d340ce39511565, if we
failed to setDeleteDisposition(true) during TempFile creatio

[Support] [Windows] Don't cancel delete if we failed to set delete

Following up on commit 177176f75c6fa3f624d6d964b9d340ce39511565, if we
failed to setDeleteDisposition(true) during TempFile creation, then
don't try to setDeleteDisposition(false) during TempFile::keep, since it
will likely fail as well.

Instead of letting TempFile::keep just fail, we should let it go ahead
and try renaming the file.

This fixes an issue we are seeing when running clang-cl.exe through the
Incredibuild distributed build system. We're seeing that renaming
temporary object files would fail here:
https://github.com/llvm/llvm-project/blob/5c1f7b296ac0dddeca02891976e6ab5cfc006719/clang/lib/Frontend/CompilerInstance.cpp#L789

Reviewed By: mstorsjo

Differential Revision: https://reviews.llvm.org/D118212

show more ...

Support: Extract sys::fs::readNativeFileToEOF() from MemoryBuffer

Extract the `readNativeFile()` loop from
`MemoryBuffer::getMemoryBufferForStream()` into `readNativeFileToEOF()`
to allow reuse. The

Support: Extract sys::fs::readNativeFileToEOF() from MemoryBuffer

Extract the `readNativeFile()` loop from
`MemoryBuffer::getMemoryBufferForStream()` into `readNativeFileToEOF()`
to allow reuse. The chunk size is configurable; the default of `4*4096`
is exposed as `sys::fs::DefaultReadChunkSize` to allow sizing of
SmallVectors.

There's somewhere I'd like to read a usually-small file without overhead
of a MemoryBuffer; extracting existing logic rather than duplicating it.

Differential Revision: https://reviews.llvm.org/D115397

show more ...

Support: Avoid using SmallVector::set_size() in sys::path

Stop using `SmallVector::set_size()` in sys::path APIs. In both cases,
use `truncate()` instead.

Differential Revision: https://reviews.llv

Support: Avoid using SmallVector::set_size() in sys::path

Stop using `SmallVector::set_size()` in sys::path APIs. In both cases,
use `truncate()` instead.

Differential Revision: https://reviews.llvm.org/D115391

show more ...

[Support] Allow configuring the preferred type of slashes on Windows

Default to preferring forward slashes when built for MinGW, as
many usecases, when e.g. Clang is used as a drop-in replacement
fo

[Support] Allow configuring the preferred type of slashes on Windows

Default to preferring forward slashes when built for MinGW, as
many usecases, when e.g. Clang is used as a drop-in replacement
for GCC, requires the compiler to output paths with forward slashes.

Not all tests pass yet, if configuring to prefer forward slashes though.

Differential Revision: https://reviews.llvm.org/D112787

show more ...

[Support] Add a new path style for Windows with forward slashes

This behaves just like the regular Windows style, with both separator
forms accepted, but with get_separator() returning forward slash

[Support] Add a new path style for Windows with forward slashes

This behaves just like the regular Windows style, with both separator
forms accepted, but with get_separator() returning forward slashes.

Add a more descriptive name for the existing style, keeping the old
name around as an alias initially.

Add a new function `make_preferred()` (like the C++17
`std::filesystem::path` function with the same name), which converts
windows paths to the preferred separator form (while this one works on
any platform and takes a `path::Style` argument).

Contrary to `native()` (just like `make_preferred()` in `std::filesystem`),
this doesn't do anything at all on Posix, it doesn't try to reinterpret
backslashes into forward slashes there.

Differential Revision: https://reviews.llvm.org/D111879

show more ...

[Support] [Windows] Use RemoveFileOnSignal if unable to use the delete-on-close flag

This takes care of cleaning up the temp files on crashes. It doesn't
handle cleanup when explicitly killed though

[Support] [Windows] Use RemoveFileOnSignal if unable to use the delete-on-close flag

This takes care of cleaning up the temp files on crashes. It doesn't
handle cleanup when explicitly killed though.

Differential Revision: https://reviews.llvm.org/D112710

show more ...

Support: Use sys::path::is_style_{posix,windows}() in a few places

Use the new sys::path::is_style_posix() and is_style_windows() in a few
places that need to detect the system's native path style.

Support: Use sys::path::is_style_{posix,windows}() in a few places

Use the new sys::path::is_style_posix() and is_style_windows() in a few
places that need to detect the system's native path style.

In llvm/lib/Support/Path.cpp, this patch removes most uses of the
private `real_style()`, where is_style_posix() and is_style_windows()
are just a little tidier.

Elsewhere, this removes `_WIN32` macro checks. Added a FIXME to a
FileManagerTest that seemed fishy, but maintained the existing
behaviour.

Differential Revision: https://reviews.llvm.org/D112289

show more ...

Support: Expose sys::path::is_style_{posix,windows,native}()

Expose three helpers in namespace llvm::sys::path to detect the
path rules followed by sys::path::Style.

- is_style_posix()
- is_style_w

Support: Expose sys::path::is_style_{posix,windows,native}()

Expose three helpers in namespace llvm::sys::path to detect the
path rules followed by sys::path::Style.

- is_style_posix()
- is_style_windows()
- is_style_native()

This are constexpr functions that that will allow a bunch of
path-related code to stop checking `_WIN32`.

Originally I looked at adding system_style(), analogous to
sys::endian::system_endianness(), but future patches (from others) will
add more Windows style variants for slash preferences. These helpers
should be resilient to that change, allowing callers to detect basic
path rules.

Differential Revision: https://reviews.llvm.org/D112288

show more ...

[Support] [Windows] Manually clean up temp files if not setting delete disposition

Since D81803 / 79657e2339b58bc01fe1b85a448bb073d57d90bb, temp files
created on network shares don't set "Dispositio

[Support] [Windows] Manually clean up temp files if not setting delete disposition

Since D81803 / 79657e2339b58bc01fe1b85a448bb073d57d90bb, temp files
created on network shares don't set "Disposition.DeleteFile = true".
This flag normally takes care of removing the temp file both if the
process exits abnormally (either crashing or killed externally), and
when the file is closed cleanly.

For network shares, we voluntarily choose to not set the flag, and
if the operation to inspect the file handle (as a prerequisite to
setting the flag since 79657e2339b58bc01fe1b85a448bb073d57d90bb)
fails we also error out. In both of these cases, we can at least make
sure to remove the temp files when they are closed cleanly.

Adjust the semantics of "OF_Delete" to not set the delete
disposition, but only set the access mode for allowing deletion.
Move the call to setDeleteDisposition into TempFile::create,
where we can check if it failed, and if it did, set a flag noting
that the file should be removed manually at the end.

This does leak files on crash, but at least doesn't leak files
in regular successful runs. (Technically, the alternative codepath
could use the RemoveFileOnSignal function, but that might complicate
the TempFile implementation further.)

This fixes https://github.com/mstorsjo/llvm-mingw/issues/233 and
https://bugs.llvm.org/show_bug.cgi?id=52080.

Differential Revision: https://reviews.llvm.org/D111875

show more ...

[Support] [Path] Use std::replace instead of an explicit comparison loop. NFC.

After 8fc7a907b93a8e9eef96e872f8f926db3ebfe9b6, this loop does
the same as a plain `std::replace`.

Also clarify the co

[Support] [Path] Use std::replace instead of an explicit comparison loop. NFC.

After 8fc7a907b93a8e9eef96e872f8f926db3ebfe9b6, this loop does
the same as a plain `std::replace`.

Also clarify the comment about what this function does.

Differential Revision: https://reviews.llvm.org/D111730

show more ...

[SystemZ][z/OS] Pass OpenFlags when creating tmp files

This patch https://reviews.llvm.org/D102876 caused some lit regressions on z/OS because tmp files were no longer being opened based on binary/t

[SystemZ][z/OS] Pass OpenFlags when creating tmp files

This patch https://reviews.llvm.org/D102876 caused some lit regressions on z/OS because tmp files were no longer being opened based on binary/text mode. This patch passes OpenFlags when creating tmp files so we can open files in different modes.

Reviewed By: amccarth

Differential Revision: https://reviews.llvm.org/D103806

show more ...

Recommit "Fix tmp files being left on Windows builds." with a fix for
incorrect std::string use. (Also remove redundant call to
RemoveFileOnSignal.)

Clang writes object files by first writing to a .

Recommit "Fix tmp files being left on Windows builds." with a fix for
incorrect std::string use. (Also remove redundant call to
RemoveFileOnSignal.)

Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the .tmp files don't get deleted.

Currently it seems like RemoveFileOnSignal takes care of deleting the
tmp files on Linux, but on Windows we need to call
setDeleteDisposition on tmp files so that they are deleted when
closed.

This patch switches to using TempFile to create the .tmp files we write
when creating object files, since it uses setDeleteDisposition on Windows.
This change applies to both Linux and Windows for consistency.

Differential Revision: https://reviews.llvm.org/D102876

This reverts commit 20797b129f844d4b12ffb2b12cf33baa2d42985c.

show more ...

Revert "Fix tmp files being left on Windows builds." for now;
causing some asan test failures.

This reverts commit 7daa18215905c831e130c7542f17619e9d936dfc.

Fix tmp files being left on Windows builds.

Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the

Fix tmp files being left on Windows builds.

Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the .tmp files don't get deleted.

Currently it seems like RemoveFileOnSignal takes care of deleting the
tmp files on Linux, but on Windows we need to call
setDeleteDisposition on tmp files so that they are deleted when
closed.

This patch switches to using TempFile to create the .tmp files we write
when creating object files, since it uses setDeleteDisposition on Windows.
This change applies to both Linux and Windows for consistency.

Differential Revision: https://reviews.llvm.org/D102876

show more ...

Support: Remove code duplication for mapped_file_region accessors, NFC

[SystemZ][z/OS] Distinguish between text and binary files on z/OS

This patch consists of the initial changes to help distinguish between text and binary content correctly on z/OS. I would like to ge

[SystemZ][z/OS] Distinguish between text and binary files on z/OS

This patch consists of the initial changes to help distinguish between text and binary content correctly on z/OS. I would like to get feedback from Windows users on setting OF_None for all ToolOutputFiles. This seems to have been done as an optimization to prevent CRLF translation on Windows in the past.

Reviewed By: zibi

Differential Revision: https://reviews.llvm.org/D97785

show more ...

[llvm] Use early exits and get rid of if-return-else-return pattern; NFC

https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code

Reviewed By: kadircet

Differential

[llvm] Use early exits and get rid of if-return-else-return pattern; NFC

https://llvm.org/docs/CodingStandards.html#use-early-exits-and-continue-to-simplify-code

Reviewed By: kadircet

Differential Revision: https://reviews.llvm.org/D89857

show more ...

Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""

This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is fail

Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""

This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is failing on Windows.

show more ...

Reland "[Modules] Add stats to measure performance of building and loading modules."

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header

Reland "[Modules] Add stats to measure performance of building and loading modules."

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

On relanding drop stats in MemoryBuffer.cpp as their value is pretty low
but affects a lot of clients and many of those aren't interested in
modules and header search.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895

show more ...

Revert "[Modules] Add stats to measure performance of building and loading modules."

This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is

Revert "[Modules] Add stats to measure performance of building and loading modules."

This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is failing. Reverting now
and will recommit after making the test not fail with the added stats.

show more ...

[Modules] Add stats to measure performance of building and loading modules.

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search.

[Modules] Add stats to measure performance of building and loading modules.

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895

show more ...

[Support/Path] Add path::is_absolute_gnu

Implements IS_ABSOLUTE_PATH from GNU tools.

C++17 is_absolute behavior is different the from the behavior defined by GNU
tools.

According to cppreference.c

[Support/Path] Add path::is_absolute_gnu

Implements IS_ABSOLUTE_PATH from GNU tools.

C++17 is_absolute behavior is different the from the behavior defined by GNU
tools.

According to cppreference.com, C++17 states: "An absolute path is a path
that unambiguously identifies the location of a file without reference
to an additional starting location."

In other words, the rules are:
1. POSIX style paths with nonempty root directory are absolute.
2. Windows style paths with nonempty root name and root directory are
absolute.
3. No other paths are absolute.

GNU rules are:
1. Paths starting with a path separator are absolute.
2. Windows style paths are also absolute if they start with a character
followed by ':'.
3. No other paths are absolute.

On Windows style the path "C:\Users\Default" has "C:" as root name and "\"
as root directory.

Hence "C:" on Windows is absolute under GNU rules and not absolute under
C++17 because it has no root directory. Likewise "/" and "\" on Windows are
absolute under GNU and are not absolute under C++17 due to empty root name.

Related to PR46368.

Differential Revision: https://reviews.llvm.org/D87667

show more ...