Use value_or instead of getValueOr. NFC

Add ThreadPriority::Low, and use QoS class Utility on Mac

On Apple Silicon Macs, using a Darwin thread priority of PRIO_DARWIN_BG seems to
map directly to the QoS class Background. With this priorit

Add ThreadPriority::Low, and use QoS class Utility on Mac

On Apple Silicon Macs, using a Darwin thread priority of PRIO_DARWIN_BG seems to
map directly to the QoS class Background. With this priority, the thread is
confined to efficiency cores only, which makes background indexing take forever.

Introduce a new ThreadPriority "Low" that sits in the middle between Background
and Default, and maps to QoS class "Utility" on Mac. Make this new priority the
default for indexing. This makes the thread run on all cores, but still lowers
priority enough to keep the machine responsive, and not interfere with
user-initiated actions.

I didn't change the implementations for Windows and Linux; on these systems,
both ThreadPriority::Background and ThreadPriority::Low map to the same thread
priority. This could be changed as a followup (e.g. by using SCHED_BATCH for Low
on Linux).

See also https://github.com/clangd/clangd/issues/1119.

Reviewed By: sammccall, dgoldman

Differential Revision: https://reviews.llvm.org/D124715

show more ...

[Support] reorder Threading includes to avoid conflict with FreeBSD headers

FreeBSD's condvar.h (included by user.h in Threading.inc) uses a "struct
thread" that conflicts with llvm::thread if both

[Support] reorder Threading includes to avoid conflict with FreeBSD headers

FreeBSD's condvar.h (included by user.h in Threading.inc) uses a "struct
thread" that conflicts with llvm::thread if both are visible when it's
included.

So this moves our #include after the FreeBSD code.

show more ...

Recommit: Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that all

Recommit: Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to set the new thread's
stack size. On Darwin even the default size is boosted to 8MB to match the main
thread.

It also switches all users of the older C-style `llvm_execute_on_thread` API
family over to `llvm::thread` followed by either a `detach` or `join` call and
removes the old API.

Moved definition of DefaultStackSize into the .cpp file to hopefully
fix the build on some (GCC-6?) machines.

show more ...

Revert "Support: add llvm::thread class that supports specifying stack size."

It's causing build failures because DefaultStackSize isn't defined everywhere
it should be and I need time to investigat

Revert "Support: add llvm::thread class that supports specifying stack size."

It's causing build failures because DefaultStackSize isn't defined everywhere
it should be and I need time to investigate.

show more ...

Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to

Support: add llvm::thread class that supports specifying stack size.

This adds a new llvm::thread class with the same interface as std::thread
except there is an extra constructor that allows us to set the new thread's
stack size. On Darwin even the default size is boosted to 8MB to match the main
thread.

It also switches all users of the older C-style `llvm_execute_on_thread` API
family over to `llvm::thread` followed by either a `detach` or `join` call and
removes the old API.

show more ...

Re-land [Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /A

Re-land [Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /AFFINITY 0xF lld-link.exe ...

Would let LLD only use 4 hyper-threads.

Previously, there was an outstanding issue on Windows Server 2019 on dual-CPU machines, which was preventing from using both CPU sockets. In normal conditions, when no affinity mask was set, ProcessorGroup::AllThreads was different from ProcessorGroup::UsableThreads. The previous code in llvm/lib/Support/Windows/Threading.inc L201 was improperly assuming those two values to be equal, and consequently was limiting the execution to only one CPU socket.

Differential Revision: https://reviews.llvm.org/D92419

show more ...

Revert "[Support] On Windows, take the affinity mask into account"

This reverts commit 336ab2d51dfdd5ca09c2a9c506453db4fe653584.

[Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /AFFINITY 0xF

[Support] On Windows, take the affinity mask into account

The number of hardware threads available to a ThreadPool can be limited if setting an affinity mask.
For example:

> start /B /AFFINITY 0xF lld-link.exe ...

Would let LLD only use 4 hyper-threads.

Previously, there was an outstanding issue on Windows Server 2019 on dual-CPU machines, which was preventing from using both CPU sockets. In normal conditions, when no affinity mask was set, ProcessorGroup::AllThreads was different from ProcessorGroup::UsableThreads. The previous code in llvm/lib/Support/Windows/Threading.inc L201 was improperly assuming those two values to be equal, and consequently was limiting the execution to only one CPU socket.

Differential Revision: https://reviews.llvm.org/D92419

show more ...

[ThinLTO] Allow usage of all hardware threads in the system

Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was all

[ThinLTO] Allow usage of all hardware threads in the system

Before this patch, it wasn't possible to extend the ThinLTO threads to all SMT/CMT threads in the system. Only one thread per core was allowed, instructed by usage of llvm::heavyweight_hardware_concurrency() in the ThinLTO code. Any number passed to the LLD flag /opt:lldltojobs=..., or any other ThinLTO-specific flag, was previously interpreted in the context of llvm::heavyweight_hardware_concurrency(), which means SMT disabled.

One can now say in LLD:
/opt:lldltojobs=0 -- Use one std::thread / hardware core in the system (no SMT). Default value if flag not specified.
/opt:lldltojobs=N -- Limit usage to N threads, regardless of usage of heavyweight_hardware_concurrency().
/opt:lldltojobs=all -- Use all hardware threads in the system. Equivalent to /opt:lldltojobs=$(nproc) on Linux and /opt:lldltojobs=%NUMBER_OF_PROCESSORS% on Windows. When an affinity mask is set for the process, threads will be created only for the cores selected by the mask.

When N > number-of-hardware-threads-in-the-system, the threads in the thread pool will be dispatched equally on all CPU sockets (tested only on Windows).
When N <= number-of-hardware-threads-on-a-CPU-socket, the threads will remain on the CPU socket where the process started (only on Windows).

Differential Revision: https://reviews.llvm.org/D75153

show more ...

llvm-ar: Fix MinGW compilation

llvm-ar is using CompareStringOrdinal which is available
only starting with Windows Vista (WINVER 0x600).

Fix this by hoising WindowsSupport.h, which sets _WIN32_WINN

llvm-ar: Fix MinGW compilation

llvm-ar is using CompareStringOrdinal which is available
only starting with Windows Vista (WINVER 0x600).

Fix this by hoising WindowsSupport.h, which sets _WIN32_WINNT
to 0x0601, up to llvm/include/llvm/Support and use it in llvm-ar.

Patch by Cristian Adam!

Differential revision: https://reviews.llvm.org/D74599

show more ...

[Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups

The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, s

[Support] On Windows, ensure hardware_concurrency() extends to all CPU sockets and all NUMA groups

The goal of this patch is to maximize CPU utilization on multi-socket or high core count systems, so that parallel computations such as LLD/ThinLTO can use all hardware threads in the system. Before this patch, on Windows, a maximum of 64 hardware threads could be used at most, in some cases dispatched only on one CPU socket.

== Background ==
Windows doesn't have a flat cpu_set_t like Linux. Instead, it projects hardware CPUs (or NUMA nodes) to applications through a concept of "processor groups". A "processor" is the smallest unit of execution on a CPU, that is, an hyper-thread if SMT is active; a core otherwise. There's a limit of 32-bit processors on older 32-bit versions of Windows, which later was raised to 64-processors with 64-bit versions of Windows. This limit comes from the affinity mask, which historically is represented by the sizeof(void*). Consequently, the concept of "processor groups" was introduced for dealing with systems with more than 64 hyper-threads.

By default, the Windows OS assigns only one "processor group" to each starting application, in a round-robin manner. If the application wants to use more processors, it needs to programmatically enable it, by assigning threads to other "processor groups". This also means that affinity cannot cross "processor group" boundaries; one can only specify a "preferred" group on start-up, but the application is free to allocate more groups if it wants to.

This creates a peculiar situation, where newer CPUs like the AMD EPYC 7702P (64-cores, 128-hyperthreads) are projected by the OS as two (2) "processor groups". This means that by default, an application can only use half of the cores. This situation could only get worse in the years to come, as dies with more cores will appear on the market.

== The problem ==
The heavyweight_hardware_concurrency() API was introduced so that only *one hardware thread per core* was used. Once that API returns, that original intention is lost, only the number of threads is retained. Consider a situation, on Windows, where the system has 2 CPU sockets, 18 cores each, each core having 2 hyper-threads, for a total of 72 hyper-threads. Both heavyweight_hardware_concurrency() and hardware_concurrency() currently return 36, because on Windows they are simply wrappers over std::thread::hardware_concurrency() -- which can only return processors from the current "processor group".

== The changes in this patch ==
To solve this situation, we capture (and retain) the initial intention until the point of usage, through a new ThreadPoolStrategy class. The number of threads to use is deferred as late as possible, until the moment where the std::threads are created (ThreadPool in the case of ThinLTO).

When using hardware_concurrency(), setting ThreadCount to 0 now means to use all the possible hardware CPU (SMT) threads. Providing a ThreadCount above to the maximum number of threads will have no effect, the maximum will be used instead.
The heavyweight_hardware_concurrency() is similar to hardware_concurrency(), except that only one thread per hardware *core* will be used.

When LLVM_ENABLE_THREADS is OFF, the threading APIs will always return 1, to ensure any caller loops will be exercised at least once.

Differential Revision: https://reviews.llvm.org/D71775

show more ...

[Support] fix mingw-w64 build

Older versions of Mingw-w64 do not define _beginthreadex_proc_type,
so we replace it with `unsigned (__stdcall *ThreadFunc)(void *)`.

Fixes https://github.com/clangd/c

[Support] fix mingw-w64 build

Older versions of Mingw-w64 do not define _beginthreadex_proc_type,
so we replace it with `unsigned (__stdcall *ThreadFunc)(void *)`.

Fixes https://github.com/clangd/clangd/issues/188

Patch by lh123!

Differential Revision: https://reviews.llvm.org/D69879

show more ...

Reland "[Support] Add a way to run a function on a detached thread""

This reverts commit 7bc7fe6b789d25d48d6dc71d533a411e9e981237.
The immediate callers have been fixed to pass nullopt where appropr

Reland "[Support] Add a way to run a function on a detached thread""

This reverts commit 7bc7fe6b789d25d48d6dc71d533a411e9e981237.
The immediate callers have been fixed to pass nullopt where appropriate.

show more ...

Revert "[Support] Add a way to run a function on a detached thread"

This reverts commit 40668abca4d307e02b33345cfdb7271549ff48d0.
This causes clang tests to fail, as stacksize=0 is being explicitly

Revert "[Support] Add a way to run a function on a detached thread"

This reverts commit 40668abca4d307e02b33345cfdb7271549ff48d0.
This causes clang tests to fail, as stacksize=0 is being explicitly passed and
is no longer a no-op.

show more ...

[Support] Add a way to run a function on a detached thread

This roughly mimics `std::thread(...).detach()` except it allows to
customize the stack size. Required for https://reviews.llvm.org/D50993.

[Support] Add a way to run a function on a detached thread

This roughly mimics `std::thread(...).detach()` except it allows to
customize the stack size. Required for https://reviews.llvm.org/D50993.

I've decided against reusing the existing `llvm_execute_on_thread` because
it's not obvious what to do with the ownership of the passed
function/arguments:

1. If we pass possibly owning functions data to `llvm_execute_on_thread`,
we'll lose the ability to pass small non-owning non-allocating functions
for the joining case (as it's used now). Is it important enough?
2. If we use the non-owning interface in the new use case, we'll force
clients to transfer ownership to the spawned thread manually, but
similar code would still have to exist inside
`llvm_execute_on_thread(_async)` anyway (as we can't just pass the same
non-owning pointer to pthreads and Windows implementations, and would be
forced to wrap it in some structure, and deal with its ownership.

Patch by Dmitry Kozhevnikov!

Differential Revision: https://reviews.llvm.org/D51103

show more ...

[llvm][Support] Provide interface to set thread priorities

Summary:
We have a multi-platform thread priority setting function(last piece
landed with D58683), I wanted to make this available to all l

[llvm][Support] Provide interface to set thread priorities

Summary:
We have a multi-platform thread priority setting function(last piece
landed with D58683), I wanted to make this available to all llvm community,
there seem to be other users of such functionality with portability fixmes:
lib/Support/CrashRecoveryContext.cpp
tools/clang/tools/libclang/CIndex.cpp

Reviewers: gribozavr, ioeric

Subscribers: krytarowski, jfb, kristina, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D59130

llvm-svn: 358494

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Don't write #include "Windows/WindowsSupport.h" from the Windows dir.

This generates -Wnonportable-include-dir warnings, and doesn't need
to be there. It seems this was just checked in on accident.

Don't write #include "Windows/WindowsSupport.h" from the Windows dir.

This generates -Wnonportable-include-dir warnings, and doesn't need
to be there. It seems this was just checked in on accident.

llvm-svn: 350655

show more ...

Fix C2712 build error on Windows

Move the __try/__except block outside of the set_thread_name function to avoid a conflict with object unwinding due to the use of the llvm::Storage.

Differential Re

Fix C2712 build error on Windows

Move the __try/__except block outside of the set_thread_name function to avoid a conflict with object unwinding due to the use of the llvm::Storage.

Differential Revision: https://reviews.llvm.org/D30707

llvm-svn: 297192

show more ...

Try to fix thread name truncation on non-Windows.

llvm-svn: 296976

Truncate thread names if they're too long.

llvm-svn: 296972

[Support] Provide access to current thread name/thread id.

Applications often need the current thread id when making
system calls, and some operating systems provide the notion
of a thread name, whi

[Support] Provide access to current thread name/thread id.

Applications often need the current thread id when making
system calls, and some operating systems provide the notion
of a thread name, which can be useful in enabling better
diagnostics when debugging or logging.

This patch adds an accessor for the thread id, and "best effort"
getters and setters for the thread name. Since this is
non critical functionality, no error is returned to indicate
that a platform doesn't support thread names.

Differential Revision: https://reviews.llvm.org/D30526

llvm-svn: 296887

show more ...

Revert "Revert 220932.": "Removing the static initializer in ManagedStatic.cpp by using llvm_call_once to initialize the ManagedStatic mutex"

This reverts commit r221331 and reinstate r220932 as dis

Revert "Revert 220932.": "Removing the static initializer in ManagedStatic.cpp by using llvm_call_once to initialize the ManagedStatic mutex"

This reverts commit r221331 and reinstate r220932 as discussed in D19271.
Original commit message was:

This patch adds an llvm_call_once which is a wrapper around
std::call_once on platforms where it is available and devoid
of bugs. The patch also migrates the ManagedStatic mutex to
be allocated using llvm_call_once.

These changes are philosophically equivalent to the changes
added in r219638, which were reverted due to a hang on Win32
which was the result of a bug in the Windows implementation
of std::call_once.

Differential Revision: http://reviews.llvm.org/D5922

From: Mehdi Amini <[email protected]>
llvm-svn: 269577

show more ...

#include <winbase.h> is not enough for Visual C++ 2013, it errors:

1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\minwinbase.h(46):
error C2146: syntax error : missing ';' before identifier 'n

#include <winbase.h> is not enough for Visual C++ 2013, it errors:

1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\minwinbase.h(46):
error C2146: syntax error : missing ';' before identifier 'nLength'
1>C:\Program Files (x86)\Windows Kits\8.1\Include\um\minwinbase.h(46):
error C4430: missing type specifier - int assumed. Note: C++ does not support default-int
...

including <windows.h> is actually required.

llvm-svn: 221244

show more ...