LLVM Driver Multicall tool

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER opti

LLVM Driver Multicall tool

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER option to the add_llvm_executable CMake
call, and changing the tool's main function to a canonicalized
tool_name_main format (i.e. llvm_ar_main, clang_main, etc...).

As currently implemented llvm-driver contains dsymutil, llvm-ar,
llvm-cxxfilt, llvm-objcopy, and clang (if clang is included in the
build).

llvm-driver can be enabled from builds by setting
LLVM_TOOL_LLVM_DRIVER_BUILD=On.

There are several limitations in the current implementation, which can
be addressed in subsequent patches:

(1) the multicall binary cannot currently properly handle
multi-dispatch tools. This means symlinking llvm-ranlib to llvm-driver
will not properly result in llvm-ar's main being called.
(2) the multicall binary cannot be comprised of tools containing
conflicting cl::opt options as the global cl::opt option list cannot
contain duplicates.

These limitations can be addressed in subsequent patches.

Differential revision: https://reviews.llvm.org/D109977

show more ...

Revert "Rename llvm::array_lengthof into llvm::size to match std::size from C++17"

This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2.

- It conflicts with the existing llvm::size in STLEx

Revert "Rename llvm::array_lengthof into llvm::size to match std::size from C++17"

This reverts commit ef8206320769ad31422a803a0d6de6077fd231d2.

- It conflicts with the existing llvm::size in STLExtras, which will now
never be called.
- Calling it without llvm:: breaks C++17 compat

show more ...

Rename llvm::array_lengthof into llvm::size to match std::size from C++17

As a conquence move llvm::array_lengthof from STLExtras.h to
STLForwardCompat.h (which is included by STLExtras.h so no buil

Rename llvm::array_lengthof into llvm::size to match std::size from C++17

As a conquence move llvm::array_lengthof from STLExtras.h to
STLForwardCompat.h (which is included by STLExtras.h so no build
breakage expected).

show more ...

Support: Avoid SmallVector::set_size() in Windows code

Replace a few `reserve()` / `set_size()` pairs with
`resize_for_overwrite()` / `truncate()` in the platform-specific
code for Windows.

Differe

Support: Avoid SmallVector::set_size() in Windows code

Replace a few `reserve()` / `set_size()` pairs with
`resize_for_overwrite()` / `truncate()` in the platform-specific
code for Windows.

Differential Revision: https://reviews.llvm.org/D115390

show more ...

[Support] Add MemoryBuffer::dontNeedIfMmap

On *NIX systems, this API calls madvise(MADV_DONTNEED) on read-only file mappings.
It should not be used on a writable buffer.
The API is used to implement

[Support] Add MemoryBuffer::dontNeedIfMmap

On *NIX systems, this API calls madvise(MADV_DONTNEED) on read-only file mappings.
It should not be used on a writable buffer.
The API is used to implement ld.lld LTO memory saving trick (D116367).

Note: on read-only file mappings, Linux's MADV_DONTNEED semantics match POSIX
POSIX_MADV_DONTNEED and BSD systems' MADV_DONTNEED.

On Windows, VirtualAllocEx MEM_COMMIT/MEM_RESET have similar semantics
but are unfortunately not drop-in replacements. dontNeedIfMmap is currently a no-op.

Reviewed By: aganea

Differential Revision: https://reviews.llvm.org/D116366

show more ...

[Support] [Windows] Convert paths to the preferred form

This normalizes most paths (except ones input from the user as command
line arguments) into the preferred form, if `real_style()` evaluates to

[Support] [Windows] Convert paths to the preferred form

This normalizes most paths (except ones input from the user as command
line arguments) into the preferred form, if `real_style()` evaluates to
`windows_forward`.

Differential Revision: https://reviews.llvm.org/D111880

show more ...

[Support] [Windows] Manually clean up temp files if not setting delete disposition

Since D81803 / 79657e2339b58bc01fe1b85a448bb073d57d90bb, temp files
created on network shares don't set "Dispositio

[Support] [Windows] Manually clean up temp files if not setting delete disposition

Since D81803 / 79657e2339b58bc01fe1b85a448bb073d57d90bb, temp files
created on network shares don't set "Disposition.DeleteFile = true".
This flag normally takes care of removing the temp file both if the
process exits abnormally (either crashing or killed externally), and
when the file is closed cleanly.

For network shares, we voluntarily choose to not set the flag, and
if the operation to inspect the file handle (as a prerequisite to
setting the flag since 79657e2339b58bc01fe1b85a448bb073d57d90bb)
fails we also error out. In both of these cases, we can at least make
sure to remove the temp files when they are closed cleanly.

Adjust the semantics of "OF_Delete" to not set the delete
disposition, but only set the access mode for allowing deletion.
Move the call to setDeleteDisposition into TempFile::create,
where we can check if it failed, and if it did, set a flag noting
that the file should be removed manually at the end.

This does leak files on crash, but at least doesn't leak files
in regular successful runs. (Technically, the alternative codepath
could use the RemoveFileOnSignal function, but that might complicate
the TempFile implementation further.)

This fixes https://github.com/mstorsjo/llvm-mingw/issues/233 and
https://bugs.llvm.org/show_bug.cgi?id=52080.

Differential Revision: https://reviews.llvm.org/D111875

show more ...

[llvm] Rename StringRef _lower() method calls to _insensitive()

This is a mechanical change. This actually also renames the
similarly named methods in the SmallString class, however these
methods do

[llvm] Rename StringRef _lower() method calls to _insensitive()

This is a mechanical change. This actually also renames the
similarly named methods in the SmallString class, however these
methods don't seem to be used outside of the llvm subproject, so
this doesn't break building of the rest of the monorepo.

show more ...

Recommit "Fix tmp files being left on Windows builds." with a fix for
incorrect std::string use. (Also remove redundant call to
RemoveFileOnSignal.)

Clang writes object files by first writing to a .

Recommit "Fix tmp files being left on Windows builds." with a fix for
incorrect std::string use. (Also remove redundant call to
RemoveFileOnSignal.)

Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the .tmp files don't get deleted.

Currently it seems like RemoveFileOnSignal takes care of deleting the
tmp files on Linux, but on Windows we need to call
setDeleteDisposition on tmp files so that they are deleted when
closed.

This patch switches to using TempFile to create the .tmp files we write
when creating object files, since it uses setDeleteDisposition on Windows.
This change applies to both Linux and Windows for consistency.

Differential Revision: https://reviews.llvm.org/D102876

This reverts commit 20797b129f844d4b12ffb2b12cf33baa2d42985c.

show more ...

Revert "Fix tmp files being left on Windows builds." for now;
causing some asan test failures.

This reverts commit 7daa18215905c831e130c7542f17619e9d936dfc.

Fix tmp files being left on Windows builds.

Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the

Fix tmp files being left on Windows builds.

Clang writes object files by first writing to a .tmp file and then
renaming to the final .obj name. On Windows, if a compile is killed
partway through the .tmp files don't get deleted.

Currently it seems like RemoveFileOnSignal takes care of deleting the
tmp files on Linux, but on Windows we need to call
setDeleteDisposition on tmp files so that they are deleted when
closed.

This patch switches to using TempFile to create the .tmp files we write
when creating object files, since it uses setDeleteDisposition on Windows.
This change applies to both Linux and Windows for consistency.

Differential Revision: https://reviews.llvm.org/D102876

show more ...

Support: Add move semantics to mapped_file_region

Update llvm::sys::fs::mapped_file_region to have a move constructor and
a move assignment operator, allowing it to be used as an Optional. Also,
upd

Support: Add move semantics to mapped_file_region

Update llvm::sys::fs::mapped_file_region to have a move constructor and
a move assignment operator, allowing it to be used as an Optional. Also,
update FileOutputBuffer's OnDiskBuffer to take advantage of this,
avoiding an extra allocation from the unique_ptr.

A nice follow-up would be to make the mapped_file_region constructor
private and replace its use with a factory function, such as
mapped_file_region::create(), that returns an Expected (or ErrorOr). I
don't plan on doing that immediately, but I might swing back later.

No functionality change, besides the saved allocation in OnDiskBuffer.

Differential Revision: https://reviews.llvm.org/D100159

show more ...

Support: Remove code duplication for mapped_file_region accessors, NFC

[SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text

Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a C

[SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text

Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a CRLF '\r\n' which may not be desirable.

Solution:
This patch adds two new flags

- OF_CRLF which indicates that CRLF translation is used.
- OF_TextWithCRLF = OF_Text | OF_CRLF indicates that the file is text and uses CRLF translation.

Developers should now use either the OF_Text or OF_TextWithCRLF for text files and OF_None for binary files. If the developer doesn't want carriage returns on Windows, they should use OF_Text, if they do want carriage returns on Windows, they should use OF_TextWithCRLF.

So this is the behaviour per platform with my patch:

z/OS:
OF_None: open in binary mode
OF_Text : open in text mode
OF_TextWithCRLF: open in text mode

Windows:
OF_None: open file with no carriage return
OF_Text: open file with no carriage return
OF_TextWithCRLF: open file with carriage return

The Major change is in llvm/lib/Support/Windows/Path.inc to only set text mode if the OF_CRLF is set.
```
if (Flags & OF_CRLF)
CrtOpenFlags |= _O_TEXT;
```

These following files are the ones that still use OF_Text which I left unchanged. I modified all these except raw_ostream.cpp in recent patches so I know these were previously in Binary mode on Windows.
./llvm/lib/Support/raw_ostream.cpp
./llvm/lib/TableGen/Main.cpp
./llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
./llvm/unittests/Support/Path.cpp
./clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp
./clang/lib/Frontend/CompilerInstance.cpp
./clang/lib/Driver/Driver.cpp
./clang/lib/Driver/ToolChains/Clang.cpp

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D99426

show more ...

[Support][Windows] Make sure only executables are found by sys::findProgramByName

The function utilizes Windows' SearchPathW function, which as I found out today, may also return directories. After

[Support][Windows] Make sure only executables are found by sys::findProgramByName

The function utilizes Windows' SearchPathW function, which as I found out today, may also return directories. After looking at the Unix implementation of the file I found that it contains a check whether the found path is also executable. While fixing the Windows implementation, I also learned that sys::fs::access returns successfully when querying whether directories are executable, which the Unix version does not.

This patch makes both of these functions equivalent to their Unix implementation and insures that any path returned by sys::findProgramByName on Windows may only be executable, just like the Unix implementation.

The equivalent additions I have made to the Windows implementation, in the Unix implementation are here:
sys::findProgramByName: https://github.com/llvm/llvm-project/blob/39ecfe614350fa5db7b8f13f81212f8e3831a390/llvm/lib/Support/Unix/Program.inc#L90
sys::fs::access: https://github.com/llvm/llvm-project/blob/c2a84771bb63947695ea50b89160c02b36fb634d/llvm/lib/Support/Unix/Path.inc#L608

I encountered this issue when running the LLVM testsuite. Commands of the form not test ... would fail to correctly execute test.exe, which is part of GnuWin32, as it actually tried to execute a folder called test, which happened to be in a directory on my PATH.

Differential Revision: https://reviews.llvm.org/D99357

show more ...

[Support] Fix 'keeping' temporary files on Windows 7

As reported here: https://bugs.llvm.org/show_bug.cgi?id=48378#c0
and here: https://github.com/rust-lang/rust/issues/81051
since 79657e2339b58bc01

[Support] Fix 'keeping' temporary files on Windows 7

As reported here: https://bugs.llvm.org/show_bug.cgi?id=48378#c0
and here: https://github.com/rust-lang/rust/issues/81051
since 79657e2339b58bc01fe1b85a448bb073d57d90bb, some programs such as llvm-ar
don't work properly on Windows 7.

The issue is shown in the snippet by Oleksandr Prodan:
https://pastebin.com/v51m3uBU

In essence, once the 'DeleteFile' flag has been set on FILE_DISPOSITION_INFO,
the file path can't be queried anymore with GetFinalPathNameByHandleW. This
however works on Windows 10, GetFinalPathNameByHandleW would return sucessfully.

To workaround the issue, we simply reset the 'DeleteFile' flag before even
checking if we're dealing with a network file.

Tested with `llvm-ar r empty.a a.obj` ran on a network mount. At the moment, we
cannot specifically add a test coverage for this, since it requres mounting a
network drive.

show more ...

[Support] PR42623: Avoid setting the delete-on-close bit if a TempFile doesn't reside on a local drive

On Windows, after commit 881ba104656c40098d4bc90c52613c08136f0fe1, tools
using TempFile would e

[Support] PR42623: Avoid setting the delete-on-close bit if a TempFile doesn't reside on a local drive

On Windows, after commit 881ba104656c40098d4bc90c52613c08136f0fe1, tools
using TempFile would error with "bad file descriptor" when writing the
file on a network drive. It appears that setting the delete-on-close bit via
SetFileInformationByHandle/FileDispositionInfo prevented it from
accessing the file on network drives, and although using
FILE_DISPOSITION_INFO seems to work, it causes other troubles.

Differential Revision: https://reviews.llvm.org/D81803

show more ...

Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""

This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is fail

Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""

This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is failing on Windows.

show more ...

Reland "[Modules] Add stats to measure performance of building and loading modules."

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header

Reland "[Modules] Add stats to measure performance of building and loading modules."

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

On relanding drop stats in MemoryBuffer.cpp as their value is pretty low
but affects a lot of clients and many of those aren't interested in
modules and header search.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895

show more ...

Revert "[Modules] Add stats to measure performance of building and loading modules."

This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is

Revert "[Modules] Add stats to measure performance of building and loading modules."

This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is failing. Reverting now
and will recommit after making the test not fail with the added stats.

show more ...

[Modules] Add stats to measure performance of building and loading modules.

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search.

[Modules] Add stats to measure performance of building and loading modules.

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895

show more ...

[Support][Windows] Fix incorrect GetFinalPathNameByHandleW() return value check in realPathFromHandle()

`GetFinalPathNameByHandleW(,,N,)` returns:
- `< N` on success (this value does not include the

[Support][Windows] Fix incorrect GetFinalPathNameByHandleW() return value check in realPathFromHandle()

`GetFinalPathNameByHandleW(,,N,)` returns:
- `< N` on success (this value does not include the size of the terminating null character)
- `>= N` if buffer is too small (this value includes the size of the terminating null character)

So, when `N == Buffer.capacity() - 1`, we need to resize buffer if return value is > `Buffer.capacity() - 2`.
Also, we can set `N` to `Buffer.capacity()`.

Thus, without this patch `realPathFromHandle()` returns unfilled buffer when length of the final path of the file is equal to `Buffer.capacity()` or `Buffer.capacity() - 1`.

Reviewed By: andrewng, amccarth

Differential Revision: https://reviews.llvm.org/D86564

show more ...

[NFC] Use value initializer for OVERLAPPED

To fix
../llvm/lib/Support/Windows/Path.inc(1265,21): warning: missing field
'InternalHigh' initializer [-Wmissing-field-initializers]
OVERLAPPED OV = {0

[NFC] Use value initializer for OVERLAPPED

To fix
../llvm/lib/Support/Windows/Path.inc(1265,21): warning: missing field
'InternalHigh' initializer [-Wmissing-field-initializers]
OVERLAPPED OV = {0};

Differential Revision: https://reviews.llvm.org/D85480

show more ...

[Support] Add file lock/unlock functions

This is recommit of f51bc4fb60fb, reverted in 8577595e03fa, because
the function `flock` is not available on Solaris. In this variant
`flock` was replaced wi

[Support] Add file lock/unlock functions

This is recommit of f51bc4fb60fb, reverted in 8577595e03fa, because
the function `flock` is not available on Solaris. In this variant
`flock` was replaced with `fcntl`, which is a POSIX function.

New functions `lockFile`, `tryLockFile` and `unlockFile` implement
simple file locking. They lock or unlock entire file. This must be
enough to support simulataneous writes to log files in parallel builds.

Differential Revision: https://reviews.llvm.org/D78896

show more ...

[Support] Fix Windows directory_iterator_construct out of bounds

Fix incorrect use of the size of Path when accessing PathUTF16, as the
UTF-16 path can be shorter. Added unit test for coverage of th

[Support] Fix Windows directory_iterator_construct out of bounds

Fix incorrect use of the size of Path when accessing PathUTF16, as the
UTF-16 path can be shorter. Added unit test for coverage of this test
case.

Thanks to Ding Fei (danix800) for the code fix, see
https://reviews.llvm.org/D83321.

Differential Revision: https://reviews.llvm.org/D83689

show more ...