[llvm] Use StringRef::contains (NFC)

[SystemZ][z/OS][Windows] Add new functions that set Text/Binary mode for Stdin and Stdout based on OpenFlags

On Windows, we want to open a file in Binary mode if OF_CRLF bit is not set. On z/OS, we

[SystemZ][z/OS][Windows] Add new functions that set Text/Binary mode for Stdin and Stdout based on OpenFlags

On Windows, we want to open a file in Binary mode if OF_CRLF bit is not set. On z/OS, we want to open a file in Binary mode if the OF_Text bit is not set.

This patch creates two new functions called ChangeStdinMode and ChangeStdoutMode which will take OpenFlags as an arg to determine which mode to set stdin and stdout to. This will enable patches like https://reviews.llvm.org/D100056 to not affect Windows when setting the OF_Text flag for raw_fd_streams.

Reviewed By: rnk

Differential Revision: https://reviews.llvm.org/D100130

show more ...

[SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text

Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a C

[SystemZ][z/OS][Windows] Add new OF_TextWithCRLF flag and use this flag instead of OF_Text

Problem:
On SystemZ we need to open text files in text mode. On Windows, files opened in text mode adds a CRLF '\r\n' which may not be desirable.

Solution:
This patch adds two new flags

- OF_CRLF which indicates that CRLF translation is used.
- OF_TextWithCRLF = OF_Text | OF_CRLF indicates that the file is text and uses CRLF translation.

Developers should now use either the OF_Text or OF_TextWithCRLF for text files and OF_None for binary files. If the developer doesn't want carriage returns on Windows, they should use OF_Text, if they do want carriage returns on Windows, they should use OF_TextWithCRLF.

So this is the behaviour per platform with my patch:

z/OS:
OF_None: open in binary mode
OF_Text : open in text mode
OF_TextWithCRLF: open in text mode

Windows:
OF_None: open file with no carriage return
OF_Text: open file with no carriage return
OF_TextWithCRLF: open file with carriage return

The Major change is in llvm/lib/Support/Windows/Path.inc to only set text mode if the OF_CRLF is set.
```
if (Flags & OF_CRLF)
CrtOpenFlags |= _O_TEXT;
```

These following files are the ones that still use OF_Text which I left unchanged. I modified all these except raw_ostream.cpp in recent patches so I know these were previously in Binary mode on Windows.
./llvm/lib/Support/raw_ostream.cpp
./llvm/lib/TableGen/Main.cpp
./llvm/tools/dsymutil/DwarfLinkerForBinary.cpp
./llvm/unittests/Support/Path.cpp
./clang/lib/StaticAnalyzer/Core/HTMLDiagnostics.cpp
./clang/lib/Frontend/CompilerInstance.cpp
./clang/lib/Driver/Driver.cpp
./clang/lib/Driver/ToolChains/Clang.cpp

Reviewed By: MaskRay

Differential Revision: https://reviews.llvm.org/D99426

show more ...

Support: Simplify __HAIKU__ #ifdef in llvm::sys::Wait, NFC

This just reduces the amount of code in the `#ifndef` block as a
follow-up to 5c1cea6f406366b85f3c200a1c48f713da4450ba.

[Support] Fix build for Haiku

This change fixes two issues with building LLVM on Haiku. The first issue is
that LLVM requires wait4(), which on Haiku is hidden behind the _BSD_SOURCE
feature flag wh

[Support] Fix build for Haiku

This change fixes two issues with building LLVM on Haiku. The first issue is
that LLVM requires wait4(), which on Haiku is hidden behind the _BSD_SOURCE
feature flag when using the --std=c++14 flag. Additionally, the wait4()
function is only available in libbsd.so, so this is now a dependency.

The other fix is that Haiku does not have the (non-standard) rusage.maxrss
member, so by default the used memory info will be set to 0 on this platform.

Reviewed By: sepavloff

Differential Revision: https://reviews.llvm.org/D87920

Patch by Niels Sascha Reedijk.

show more ...

Re-land [Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /A

Re-land [Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /AFFINITY 0xF lld-link.exe ...

Would let LLD only use 4 hyper-threads.

Previously, there was an outstanding issue on Windows Server 2019 on dual-CPU machines, which was preventing from using both CPU sockets. In normal conditions, when no affinity mask was set, ProcessorGroup::AllThreads was different from ProcessorGroup::UsableThreads. The previous code in llvm/lib/Support/Windows/Threading.inc L201 was improperly assuming those two values to be equal, and consequently was limiting the execution to only one CPU socket.

Differential Revision: https://reviews.llvm.org/D92419

show more ...

Revert "[Support] On Windows, take the affinity mask into account"

This reverts commit 336ab2d51dfdd5ca09c2a9c506453db4fe653584.

[Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /AFFINITY 0xF

[Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /AFFINITY 0xF lld-link.exe ...

Would let LLD only use 4 hyper-threads.

Previously, there was an outstanding issue on Windows Server 2019 on dual-CPU machines, which was preventing from using both CPU sockets. In normal conditions, when no affinity mask was set, ProcessorGroup::AllThreads was different from ProcessorGroup::UsableThreads. The previous code in llvm/lib/Support/Windows/Threading.inc L201 was improperly assuming those two values to be equal, and consequently was limiting the execution to only one CPU socket.

Differential Revision: https://reviews.llvm.org/D92419

show more ...

[NFC][Support] Make Unix/Program.inc separately compilable

To improve CI checks, make `Unix/Program.inc` separately compilable.

[Support][AIX] Add declaration of wait4 to fix build

While `wait4` is not documented for AIX, it is available; however, even
on systems where it is available, the system headers do not always
provid

[Support][AIX] Add declaration of wait4 to fix build

While `wait4` is not documented for AIX, it is available; however, even
on systems where it is available, the system headers do not always
provide a declaration of the function. This patch provides a declaration
of `wait4` for AIX.

Reviewed By: daltenty

Differential Revision: https://reviews.llvm.org/D82282

show more ...

[Support] Get process statistics in ExecuteAndWait and Wait

The functions sys::ExcecuteAndWait and sys::Wait now have additional
argument of type pointer to structure, which is filled with process
e

[Support] Get process statistics in ExecuteAndWait and Wait

The functions sys::ExcecuteAndWait and sys::Wait now have additional
argument of type pointer to structure, which is filled with process
execution statistics upon process termination. These are total and user
execution times and peak memory consumption. By default this argument is
nullptr so existing users of these function must not change behavior.

Differential Revision: https://reviews.llvm.org/D78901

show more ...

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly m

Make llvm::StringRef to std::string conversions explicit.

This is how it should've been and brings it more in line with
std::string_view. There should be no functional change here.

This is mostly mechanical from a custom clang-tidy check, with a lot of
manual fixups. It uncovers a lot of minor inefficiencies.

This doesn't actually modify StringRef yet, I'll do that in a follow-up.

show more ...

[Support][NFC] Fix error message for posix_spawn_file_actions_addopen failed call

Seems like a copy-paste from couple lines above.

llvm-svn: 368899

Rename F_{None,Text,Append} to OF_{None,Text,Append}. NFC

F_{None,Text,Append} are kept for compatibility since r334221.

llvm-svn: 367800

posix_spawn should retry upon EINTR

Summary:
We've seen cases of bots failing with:
clang: error: unable to execute command: posix_spawn failed: Interrupted system call

Add a small retry loop to

posix_spawn should retry upon EINTR

Summary:
We've seen cases of bots failing with:
clang: error: unable to execute command: posix_spawn failed: Interrupted system call

Add a small retry loop to posix_spawn in case this happens. Don't retry too much in case there's some systemic problem going on, but retry a few times.
<rdar://problem/50181448>

Reviewers: Bigcheese, arphaman

Subscribers: jkorous, dexonsmith, kristina, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D61096

llvm-svn: 359152

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Refactor ExecuteAndWait to take StringRefs.

This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.

In the end, I th

Refactor ExecuteAndWait to take StringRefs.

This simplifies some code which had StringRefs to begin with, and
makes other code more complicated which had const char* to begin
with.

In the end, I think this makes for a more idiomatic and platform
agnostic API. Not all platforms launch process with null terminated
c-string arrays for the environment pointer and argv, but the api
was designed that way because it allowed easy pass-through for
posix-based platforms. There's a little additional overhead now
since on posix based platforms we'll be takign StringRefs which
were constructed from null terminated strings and then copying
them to null terminate them again, but from a readability and
usability standpoint of the API user, I think this API signature
is strictly better.

llvm-svn: 334518

show more ...

Attempt 3: Resubmit "[Support] Expose flattenWindowsCommandLine."

I took some liberties and quoted fewer characters than before,
based on an article from MSDN which says that only certain characters

Attempt 3: Resubmit "[Support] Expose flattenWindowsCommandLine."

I took some liberties and quoted fewer characters than before,
based on an article from MSDN which says that only certain characters
cause an arg to require quoting. This seems to be incorrect, though,
and worse it seems to be a difference in Windows version. The bot
that fails is Windows 7, and I can't reproduce the failure on Win
10. But it's definitely related to quoting and special characters,
because both tests that fail have a * in the argument, which is one
of the special characters that would cause an argument to be quoted
before but not any longer after the new patch.

Since I don't have Win 7, all I can do is just guess that I need to
restore the old quoting rules. So this patch does that in hopes that
it fixes the problem on Windows 7.

llvm-svn: 334375

show more ...

Revert "Resubmit "[Support] Expose flattenWindowsCommandLine.""

This reverts commit 65243b6d19143cb7a03f68df0169dcb63e8b4632.

Seems like it's not a flake. It might have something to do with
the '*

Revert "Resubmit "[Support] Expose flattenWindowsCommandLine.""

This reverts commit 65243b6d19143cb7a03f68df0169dcb63e8b4632.

Seems like it's not a flake. It might have something to do with
the '*' character being in a command line.

llvm-svn: 334356

show more ...

Resubmit "[Support] Expose flattenWindowsCommandLine."

There were a few linux compilation failures, but other than that
I think this was just a flake that caused the tests to fail. I'm
going to res

Resubmit "[Support] Expose flattenWindowsCommandLine."

There were a few linux compilation failures, but other than that
I think this was just a flake that caused the tests to fail. I'm
going to resubmit and see if the failures go away, if not I'll
revert again.

llvm-svn: 334355

show more ...

commandLineFitsWithinSystemLimits Overestimates System Limits

Summary:
The function `llvm::sys::commandLineFitsWithinSystemLimits` appears to be overestimating the system limits. This issue was disc

commandLineFitsWithinSystemLimits Overestimates System Limits

Summary:
The function `llvm::sys::commandLineFitsWithinSystemLimits` appears to be overestimating the system limits. This issue was discovered while attempting to enable response files in the Swift compiler. When the compiler submits its frontend jobs, those jobs are subjected to the system limits on command line length. `commandLineFitsWithinSystemLimits` is used to determine if the job's arguments need to be wrapped in a response file. There are some cases where the argument size for the job passes `commandLineFitsWithinSystemLimits`, but actually exceeds the real system limit, and the job fails.

`clang` also uses this function to decide whether or not to wrap it's job arguments in response files. See: https://github.com/llvm-mirror/clang/blob/master/lib/Driver/Driver.cpp#L1341. Clang will also fail for response files who's size falls within a certain range. I wrote a script that should find a failure point for `clang++`. All that is needed to run it is Python 2.7, and a simple "hello world" program for `test.cc`. It should run on Linux and on macOS. The script is available here: https://gist.github.com/dabelknap/71bd083cd06b91c5b3cef6a7f4d3d427. When it hits a failure point, you should see a `clang: error: unable to execute command: posix_spawn failed: Argument list too long`.

The proposed solution is to mirror the behavior of `xargs` in `commandLinefitsWithinSystemLimits`. `xargs` defaults to 128k for the command line length size (See: https://fossies.org/dox/findutils-4.6.0/buildcmd_8c_source.html#l00551). It adjusts this depending on the value of `ARG_MAX`.

Reviewers: alexfh

Reviewed By: alexfh

Subscribers: llvm-commits

Tags: #clang

Patch by Austin Belknap!

Differential Revision: https://reviews.llvm.org/D47795

llvm-svn: 334295

show more ...

Clean up some code in Program.

NFC here, this just raises some platform specific ifdef hackery
out of a class and creates proper platform-independent typedefs
for the relevant things. This allows t

Clean up some code in Program.

NFC here, this just raises some platform specific ifdef hackery
out of a class and creates proper platform-independent typedefs
for the relevant things. This allows these typedefs to be
reused in other places without having to reinvent this preprocessor
logic.

llvm-svn: 334294

show more ...

[Unix] Indent ChangeStd{in,out}ToBinary.

llvm-svn: 332432

Convenience/safety fix for llvm::sys::Execute(And|No)Wait

Summary:
Change the type of the Redirects parameter of llvm::sys::ExecuteAndWait,
ExecuteNoWait and other APIs that wrap them from `const St

Convenience/safety fix for llvm::sys::Execute(And|No)Wait

Summary:
Change the type of the Redirects parameter of llvm::sys::ExecuteAndWait,
ExecuteNoWait and other APIs that wrap them from `const StringRef **` to
`ArrayRef<Optional<StringRef>>`, which is safer and simplifies the use of these
APIs (no more local StringRef variables just to get a pointer to).

Corresponding clang changes will be posted as a separate patch.

Reviewers: bkramer

Reviewed By: bkramer

Subscribers: vsk, llvm-commits

Differential Revision: https://reviews.llvm.org/D37563

llvm-svn: 313155

show more ...

Minor style fixes in lib/Support/**/Program.(inc|cpp).

No functional changes intended.

llvm-svn: 312646