LLVM Driver Multicall tool

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER opti

LLVM Driver Multicall tool

This patch adds an llvm-driver multicall tool that can combine multiple
LLVM-based tools. The build infrastructure is enabled for a tool by
adding the GENERATE_DRIVER option to the add_llvm_executable CMake
call, and changing the tool's main function to a canonicalized
tool_name_main format (i.e. llvm_ar_main, clang_main, etc...).

As currently implemented llvm-driver contains dsymutil, llvm-ar,
llvm-cxxfilt, llvm-objcopy, and clang (if clang is included in the
build).

llvm-driver can be enabled from builds by setting
LLVM_TOOL_LLVM_DRIVER_BUILD=On.

There are several limitations in the current implementation, which can
be addressed in subsequent patches:

(1) the multicall binary cannot currently properly handle
multi-dispatch tools. This means symlinking llvm-ranlib to llvm-driver
will not properly result in llvm-ar's main being called.
(2) the multicall binary cannot be comprised of tools containing
conflicting cl::opt options as the global cl::opt option list cannot
contain duplicates.

These limitations can be addressed in subsequent patches.

Differential revision: https://reviews.llvm.org/D109977

show more ...

[Support] Skip attempts to access /proc/self/fd on FreeBSD

In contrast to Linux it does not provide entries which can be readlinked
-- these are just regular files, not giving the expected outcome.

[Support] Skip attempts to access /proc/self/fd on FreeBSD

In contrast to Linux it does not provide entries which can be readlinked
-- these are just regular files, not giving the expected outcome. That's
on top of procfs not being mounted by default to begin with.

This is probably the case on other BSDs as well, so I expect there will
be more ifdefs added down the road.

Reviewed By: emaste, dim

Differential Revision: https://reviews.llvm.org/D122545

show more ...

[Support] Use posix_madvise() if available

This is a followup to D119695 using the suggestion by joerg. Rather
than manually declaring madvise() on __sun__, this uses
posix_madvise() if available, w

[Support] Use posix_madvise() if available

This is a followup to D119695 using the suggestion by joerg. Rather
than manually declaring madvise() on __sun__, this uses
posix_madvise() if available, which does get declared properly on
Illumos.

Differential Revision: https://reviews.llvm.org/D119856

show more ...

[Support] Fix build on illumos

D116366 added a call to madvise() in Path.inc. Unfortunately,
Illumos does not declare this function if _XOPEN_SOURCE is defined
(which it is by default) and we need t

[Support] Fix build on illumos

D116366 added a call to madvise() in Path.inc. Unfortunately,
Illumos does not declare this function if _XOPEN_SOURCE is defined
(which it is by default) and we need to provide the declaration
manually. This is the same workaround used in sanitizers:
https://github.com/llvm/llvm-project/blob/ee423d93ead39e94c2970b3cc7ef6e6faa75d10b/compiler-rt/lib/sanitizer_common/sanitizer_solaris.cpp#L77-L85

Differential Revision: https://reviews.llvm.org/D119695

show more ...

Support: Avoid SmallVector::set_size() in Unix code

Replace a `reserve()`/`set_size()` pair with `resize_for_overwrite()`
and `truncate()`. The out parameter also needs a `clear()` call on the
error

Support: Avoid SmallVector::set_size() in Unix code

Replace a `reserve()`/`set_size()` pair with `resize_for_overwrite()`
and `truncate()`. The out parameter also needs a `clear()` call on the
error path.

Differential Revision: https://reviews.llvm.org/D115389

show more ...

[AIX][z/OS][Support] Provide alternate no-op mapped_file_region::dontNeedImpl implementation

mapped_file_region::dontNeedImpl added in D116366 calls madvise, which
causes problems for z/OS and AIX.

[AIX][z/OS][Support] Provide alternate no-op mapped_file_region::dontNeedImpl implementation

mapped_file_region::dontNeedImpl added in D116366 calls madvise, which
causes problems for z/OS and AIX.

For z/OS, we don't have either madvise, so treat this as a no-op, same
as Windows does.

For AIX, it doesn't have any effect, doesn't have a standardized
signature, and it needs certain feature test macros (i.e. _ALL_SOURCE)
we don't set by default for LLVM on AIX, so just make it a no-op too.

Differential Revision: https://reviews.llvm.org/D116603

show more ...

[Support] Add MemoryBuffer::dontNeedIfMmap

On *NIX systems, this API calls madvise(MADV_DONTNEED) on read-only file mappings.
It should not be used on a writable buffer.
The API is used to implement

[Support] Add MemoryBuffer::dontNeedIfMmap

On *NIX systems, this API calls madvise(MADV_DONTNEED) on read-only file mappings.
It should not be used on a writable buffer.
The API is used to implement ld.lld LTO memory saving trick (D116367).

Note: on read-only file mappings, Linux's MADV_DONTNEED semantics match POSIX
POSIX_MADV_DONTNEED and BSD systems' MADV_DONTNEED.

On Windows, VirtualAllocEx MEM_COMMIT/MEM_RESET have similar semantics
but are unfortunately not drop-in replacements. dontNeedIfMmap is currently a no-op.

Reviewed By: aganea

Differential Revision: https://reviews.llvm.org/D116366

show more ...

[llvm] Use nullptr instead of 0 (NFC)

Identified with modernize-use-nullptr.

[Support] Revert posix_fallocate in resize_file

This reverts 3816c53f040cc6aa06425978dd504b0bd5b7899c and removes follow-up
fixups.

The original intention was to show error earlier (posix_fallocate

[Support] Revert posix_fallocate in resize_file

This reverts 3816c53f040cc6aa06425978dd504b0bd5b7899c and removes follow-up
fixups.

The original intention was to show error earlier (posix_fallocate time) than
later for ld.lld but it appears to cause some problems which make it not free.

* FreeBSD ZFS: EINVAL, not too bad.
* FreeBSD UFS: according to khng "devastatingly slow on freebsd because UFS on freebsd does not have preallocation support like illumos. It zero-fills."
* NetBSD: maybe EOPNOTSUPP
* Linux tmpfs: unless tmpfs is set up to use huge pages (requires CONFIG_TRANSPARENT_HUGE_PAGECACHE=y), I can consistently demonstrate ~300ms delay for a 1.4GiB output.
* Linux ext4: I don't measure any benefit, either backed by a hard disk or by a file in tmpfs.
* The current code organization of `defined(HAVE_POSIX_FALLOCATE)` costs us a macro dispatch for AIX.

I think we should just remove it. I think if posix_fallocate ever finds demonstrable benefit,
it is likely Linux specific and will not need HAVE_POSIX_FALLOCATE, and possibly opt-in by some specific programs.

In a filesystem with CoW and compression, the ENOSPC benefit may be lost as well.

Reviewed By: khng300

Differential Revision: https://reviews.llvm.org/D115957

show more ...

Support: Reduce stats in fs::copy_file on Darwin

fs::copy_file() on Darwin has a nice optimization to clone the file when
possible. Change the implementation to use clonefile() directly, instead
of

Support: Reduce stats in fs::copy_file on Darwin

fs::copy_file() on Darwin has a nice optimization to clone the file when
possible. Change the implementation to use clonefile() directly, instead
of the higher-level copyfile(). The latter does the wrong thing for
symlinks, which requires calling `stat` first...

With that out of the way, optimistically call clonefile() all the time,
and then for any error that's recoverable try again with copyfile()
(without the COPYFILE_CLONE flag, as before).

Differential Revision: https://reviews.llvm.org/D112250

show more ...

[Support] Implement getMainExecutable on Solaris

Many `flang` tests currently `FAIL` on Solaris because the module files
aren't found. I could trace this to `sys::fs::getMainExecutable` not being
i

[Support] Implement getMainExecutable on Solaris

Many `flang` tests currently `FAIL` on Solaris because the module files
aren't found. I could trace this to `sys::fs::getMainExecutable` not being
implemented.

This patch does this and fixes all affected `flang` tests.

Tested on `amd64-pc-solaris2.11`.

Differential Revision: https://reviews.llvm.org/D109374

show more ...

[Support] Fix getMainExecutable on FreeBSD when called via an absolute path

On FreeBSD, absolute paths are passed unmodified in AT_EXECPATH, but
relative paths are resolved to absolute paths, and an

[Support] Fix getMainExecutable on FreeBSD when called via an absolute path

On FreeBSD, absolute paths are passed unmodified in AT_EXECPATH, but
relative paths are resolved to absolute paths, and any symlinks will be
followed in the process. This means that the resource dir calculation
will be wrong if Clang is invoked as an absolute path to a symlink, and
this currently causes clang/test/Driver/rocm-detect.hip to fail on
FreeBSD. Thus, make sure to call realpath on the result, just like is
done on macOS.

Whilst here, clean up the old fallback auxargs loop to use the actual
type for auxargs rather than using lots of hacky casts that rely on
addresses and pointers being the same (which is not the case on CHERI,
and thus Arm's prototype Morello, although for little-endian systems it
happens to work still as the word-sized integer will be padded to a full
pointer, and it's someone academic given dereferencing past the end of
environ will give a bounds fault, but CheriBSD is new enough that the
elf_aux_info path will be used). This also makes the code easier to
follow, and removes the confusing double-increment of p.

Reviewed By: dim, arichardson

Differential Revision: https://reviews.llvm.org/D103346

show more ...

[SystemZ][z/OS] Add the functions needed for handling EBCDIC I/O

This patch adds the basic functions needed for controlling auto conversion on z/OS.
Auto conversion is enabled on untagged input file

[SystemZ][z/OS] Add the functions needed for handling EBCDIC I/O

This patch adds the basic functions needed for controlling auto conversion on z/OS.
Auto conversion is enabled on untagged input file to ASCII by making the assumption that all untagged files are EBCDIC encoded. Output files are auto converted to EBCDIC IBM-1047.
This change also enables conversion for stdin/stdout/stderr.

For more information on how fcntl controls codepage https://www.ibm.com/docs/en/zos/2.4.0?topic=descriptions-fcntl-bpx1fct-bpx4fct-control-open-file-descriptors

Reviewed By: anirudhp

Differential Revision: https://reviews.llvm.org/D100483

show more ...

Support: Add move semantics to mapped_file_region

Update llvm::sys::fs::mapped_file_region to have a move constructor and
a move assignment operator, allowing it to be used as an Optional. Also,
upd

Support: Add move semantics to mapped_file_region

Update llvm::sys::fs::mapped_file_region to have a move constructor and
a move assignment operator, allowing it to be used as an Optional. Also,
update FileOutputBuffer's OnDiskBuffer to take advantage of this,
avoiding an extra allocation from the unique_ptr.

A nice follow-up would be to make the mapped_file_region constructor
private and replace its use with a factory function, such as
mapped_file_region::create(), that returns an Expected (or ErrorOr). I
don't plan on doing that immediately, but I might swing back later.

No functionality change, besides the saved allocation in OnDiskBuffer.

Differential Revision: https://reviews.llvm.org/D100159

show more ...

Support: Remove code duplication for mapped_file_region accessors, NFC

Support: mapped_file_region: Pass MAP_NORESERVE to mmap

This allows mapping larger files, delaying OOM failures until too many
pages of them are accessed. This is makes the behavior of the
mapped_

Support: mapped_file_region: Pass MAP_NORESERVE to mmap

This allows mapping larger files, delaying OOM failures until too many
pages of them are accessed. This is makes the behavior of the
mapped_file_region in this regard consistent between its "Unix" and
"Windows" implementations.

Guard the code witih #if defined(MAP_NORESERVE), consistent with other
uses of MAP_NORESERVE in llvm-project, because some FreeBSD versions do
not provide this flag.

Reviewed By: clayborg

Differential Revision: https://reviews.llvm.org/D96626

show more ...

[llvm-objcopy] preserve file ownership when overwritten by root

As of binutils 2.36, GNU strip calls chown(2) for "sudo strip foo" and
"sudo strip foo -o foo", but no "sudo strip foo -o bar" or "sud

[llvm-objcopy] preserve file ownership when overwritten by root

As of binutils 2.36, GNU strip calls chown(2) for "sudo strip foo" and
"sudo strip foo -o foo", but no "sudo strip foo -o bar" or "sudo strip
foo -o ./foo". In other words, while "sudo strip foo -o bar" creates a
new file bar with root access, "sudo strip foo" will keep the owner and
group of foo unchanged. Currently llvm-objcopy and llvm-strip behave
differently, always changing the owner and gropu to root. The
discrepancy prevents Chrome OS from migrating to llvm-objcopy and
llvm-strip as they change file ownership and cause intended users/groups
to lose access when invoked by sudo with the following sequence
(recommended in man page of GNU strip).

1.<Link the executable as normal.>
1.<Copy "foo" to "foo.full">
1.<Run "strip --strip-debug foo">
1.<Run "objcopy --add-gnu-debuglink=foo.full foo">

This patch makes llvm-objcopy and llvm-strip follow GNU's behavior.

Link: crbug.com/1108880

show more ...

[llvm] Remove redundant return and continue statements (NFC)

Identified with readability-redundant-control-flow.

[llvm] [Support] Fix segv if argv0 is null in getMainExecutable()

When LLDB Python bindings are used and stack backtraces are enabled
for logging, getMainExecutable() is called with argv0 being null

[llvm] [Support] Fix segv if argv0 is null in getMainExecutable()

When LLDB Python bindings are used and stack backtraces are enabled
for logging, getMainExecutable() is called with argv0 being null.
This caused the fallback function getprogpath() (used on FreeBSD, NetBSD
and Linux) to segfault. Make it handle null executable name gracefully.

Differential Revision: https://reviews.llvm.org/D91012

show more ...

Fix calls to (p)read on macOS when size > INT32_MAX

On macOS, the read and pread syscalls return EINVAL when the number of
bytes to read exceeds INT32_MAX:

https://github.com/apple/darwin-xnu/blob/

Fix calls to (p)read on macOS when size > INT32_MAX

On macOS, the read and pread syscalls return EINVAL when the number of
bytes to read exceeds INT32_MAX:

https://github.com/apple/darwin-xnu/blob/a449c6a3b8014d9406c2ddbdc81795da24aa7443/bsd/kern/sys_generic.c#L355

rdar://68751407

Differential revision: https://reviews.llvm.org/D90201

show more ...

Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""

This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is fail

Revert "Reland "[Modules] Add stats to measure performance of building and loading modules.""

This reverts commit 4000c9ee18ecebe3ff0f197af8c1fb434ad986e5.

Test "LLVM :: Other/statistic.ll" is failing on Windows.

show more ...

Reland "[Modules] Add stats to measure performance of building and loading modules."

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header

Reland "[Modules] Add stats to measure performance of building and loading modules."

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

On relanding drop stats in MemoryBuffer.cpp as their value is pretty low
but affects a lot of clients and many of those aren't interested in
modules and header search.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895

show more ...

Revert "[Modules] Add stats to measure performance of building and loading modules."

This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is

Revert "[Modules] Add stats to measure performance of building and loading modules."

This reverts commit c4bacc3c9b333bb7032fb96f41d6f5b851623132.

Test "LLVM :: ThinLTO/X86/funcimport-stats.ll" is failing. Reverting now
and will recommit after making the test not fail with the added stats.

show more ...

[Modules] Add stats to measure performance of building and loading modules.

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search.

[Modules] Add stats to measure performance of building and loading modules.

Measure amount of high-level or fixed-cost operations performed during
building/loading modules and during header search. High-level operations
like building a module or processing a .pcm file are motivated by
previous issues where clang was re-building modules or re-reading .pcm
files unnecessarily. Fixed-cost operations like `stat` calls are tracked
because clang cannot change how long each operation takes but it can
perform fewer of such operations to improve the compile time.

Also tracking such stats over time can help us detect compile-time
regressions. Added stats are more stable than the actual measured
compilation time, so expect the detected regressions to be less noisy.

rdar://problem/55715134

Reviewed By: aprantl, bruno

Differential Revision: https://reviews.llvm.org/D86895

show more ...

[SystemZ/ZOS] Implement setLastAccessAndModificationTime()

The function setLastAccessAndModificationTime() uses function
futimens() or futimes() by default. Both functions are not
available in z/OS,

[SystemZ/ZOS] Implement setLastAccessAndModificationTime()

The function setLastAccessAndModificationTime() uses function
futimens() or futimes() by default. Both functions are not
available in z/OS, therefore functionality is implemented using
__fchattr() on z/OS.

Reviews by: abhina.sreeskantharajan

Differential Revision: https://reviews.llvm.org/D83945

show more ...