[OpenMP] libomp: honor passive wait policy requested with tasking

Currently the library ignores requested wait policy in the presence
of tasking. Threads always actively spin. The patch fixes this p

[OpenMP] libomp: honor passive wait policy requested with tasking

Currently the library ignores requested wait policy in the presence
of tasking. Threads always actively spin. The patch fixes this problem
making the wait policy passive if this explicitly requested by user.

Differential Revision: https://reviews.llvm.org/D123044

show more ...

[OpenMP][libomp] Fix some Doxygen issues

Fix spelling of variable names and remove accidental references (#)
in Doxygen comments.

[OpenMP] Fix warnings

Silenced compiler warnings after pushing the following change.
https://reviews.llvm.org/D122107

Differential Revision: https://reviews.llvm.org/D123233

[OpenMP] Add support for ompt_callback_dispatch

This change adds support for ompt_callback_dispatch with the new
dispatch chunk type introduced in 5.2. Definitions of the new
ompt_work_loop types we

[OpenMP] Add support for ompt_callback_dispatch

This change adds support for ompt_callback_dispatch with the new
dispatch chunk type introduced in 5.2. Definitions of the new
ompt_work_loop types were also added in the header file.

Differential Revision: https://reviews.llvm.org/D122107

show more ...

[OpenMP] libomp: implemented task priorities.

Before this patch task priorities were ignored, that was a valid implementation
as the task priority is a hint according to OpenMP specification.

Imple

[OpenMP] libomp: implemented task priorities.

Before this patch task priorities were ignored, that was a valid implementation
as the task priority is a hint according to OpenMP specification.

Implemented shared list of sorted (high -> low) task deques one per task
priority value. Tasks execution changed to first check if priority tasks ready
for execution exist, and these tasks executed before others;
otherwise usual tasks execution mechanics work.

Differential Revision: https://reviews.llvm.org/D119676

show more ...

[OpenMP] Add use of TPAUSE

Add use of TPAUSE (from WAITPKG) to the runtime for Intel hardware,
with an envirable to turn it on in a particular C-state. Always uses
TPAUSE if it is selected and enab

[OpenMP] Add use of TPAUSE

Add use of TPAUSE (from WAITPKG) to the runtime for Intel hardware,
with an envirable to turn it on in a particular C-state. Always uses
TPAUSE if it is selected and enabled by Intel hardware and presence of
WAITPKG, and if not, falls back to old way of checking
__kmp_use_yield, etc.

Differential Revision: https://reviews.llvm.org/D115758

show more ...

[OpenMP] Add missing `tt_hidden_helper_task_encountered` along with `tt_found_proxy_tasks`

In most cases, hidden helper task behave similar as detached tasks. That means,
for example, if we have to

[OpenMP] Add missing `tt_hidden_helper_task_encountered` along with `tt_found_proxy_tasks`

In most cases, hidden helper task behave similar as detached tasks. That means,
for example, if we have to wait for detached tasks, we have to do the same thing
for hidden helper tasks as well. This patch adds the missing condition for hidden
helper task accordingly along with detached task.

Reviewed By: AndreyChurbanov

Differential Revision: https://reviews.llvm.org/D107316

show more ...

[OpenMP] libomp: add check of task function pointer for NULL.

This patch allows to simplify compiler implementation on "taskwait nowait"
construct. The "taskwait nowait" is semantically equivalent t

[OpenMP] libomp: add check of task function pointer for NULL.

This patch allows to simplify compiler implementation on "taskwait nowait"
construct. The "taskwait nowait" is semantically equivalent to the empty task.
Instead of creating an empty routine as a task entry, compiler can just send
NULL pointer to the runtime. Then the runtime will make all the work with
dependences and return because of the absent task routine.

Differential Revision: https://reviews.llvm.org/D112015

show more ...

[OpenMP] Fix task wait doesn't work as expected in serialized team

As discussed in D107121, task wait doesn't work when a regular task T depends on
a detached task or a hidden helper task T' in a se

[OpenMP] Fix task wait doesn't work as expected in serialized team

As discussed in D107121, task wait doesn't work when a regular task T depends on
a detached task or a hidden helper task T' in a serialized team. The root cause is,
since the team is serialized, the last task will not be tracked by
`td_incomplete_child_tasks`. When T' is finished, it first releases its
dependences, and then decrements its parent counter. So far so good. For the thread
that is running task wait, if at the moment it is still spinning and trying to
execute tasks, it is fine because it can detect the new task and execute it.
However, if it happends to finish the function `flag.execute_tasks(...)`, it will
be broken because `td_incomplete_child_tasks` is 0 now.

In this patch, we update the rule to track children tasks a little bit. If the
task team encounters a proxy task or a hidden helper task, all following tasks
will be tracked.

Reviewed By: AndreyChurbanov

Differential Revision: https://reviews.llvm.org/D107496

show more ...

[OpenMP] Use `__kmpc_give_task` in `__kmp_push_task` when encountering a hidden helper task

This patch replaces the current implementation, overwrites `gtid` and `thread`,
with `__kmpc_give_task`.

[OpenMP] Use `__kmpc_give_task` in `__kmp_push_task` when encountering a hidden helper task

This patch replaces the current implementation, overwrites `gtid` and `thread`,
with `__kmpc_give_task`.

Reviewed By: AndreyChurbanov

Differential Revision: https://reviews.llvm.org/D106977

show more ...

[OpenMP] mark target task untied

OpenMP specification Tasking Terminology
target task :A mergeable and untied task that ...

Reviewed By: tianshilei1992

Differential Revision: https://reviews.llvm.

[OpenMP] mark target task untied

OpenMP specification Tasking Terminology
target task :A mergeable and untied task that ...

Reviewed By: tianshilei1992

Differential Revision: https://reviews.llvm.org/D107686

show more ...

[OpenMP] Clean up for hidden helper task

This patch makes some clean up for code of hidden helper task.

Reviewed By: protze.joachim

Differential Revision: https://reviews.llvm.org/D107008

[OpenMP] Fix performance regression reported in bug #51235

This patch fixes the "performance regression" reported in https://bugs.llvm.org/show_bug.cgi?id=51235. In fact it has nothing to do with pe

[OpenMP] Fix performance regression reported in bug #51235

This patch fixes the "performance regression" reported in https://bugs.llvm.org/show_bug.cgi?id=51235. In fact it has nothing to do with performance. The root cause is, the stolen task is not allowed to execute by another thread because by default it is tied task. Since hidden helper task will always be executed by hidden helper threads, it should be untied.

Reviewed By: protze.joachim

Differential Revision: https://reviews.llvm.org/D107121

show more ...

[OpenMP][NFC] libomp: silence warnings on unused variables.

Put declarations/definitions of unused variables under corresponding macros
to silence clang build warnings.

Differential Revision: https

[OpenMP][NFC] libomp: silence warnings on unused variables.

Put declarations/definitions of unused variables under corresponding macros
to silence clang build warnings.

Differential Revision: https://reviews.llvm.org/D106608

show more ...

[OpenMP] libomp: Add new experimental barrier: two-level distributed barrier

Two-level distributed barrier is a new experimental barrier designed
for Intel hardware that has better performance in so

[OpenMP] libomp: Add new experimental barrier: two-level distributed barrier

Two-level distributed barrier is a new experimental barrier designed
for Intel hardware that has better performance in some cases than the
default hyper barrier.

This barrier is designed to handle fine granularity parallelism where
barriers are used frequently with little compute and memory access
between barriers. There is no need to use it for codes with few
barriers and large granularity compute, or memory intensive
applications, as little difference will be seen between this barrier
and the default hyper barrier. This barrier is designed to work
optimally with a fixed number of threads, and has a significant setup
time, so should NOT be used in situations where the number of threads
in a team is varied frequently.

The two-level distributed barrier is off by default -- hyper barrier
is used by default. To use this barrier, you must set all barrier
patterns to use this type, because it will not work with other barrier
patterns. Thus, to turn it on, the following settings are required:

KMP_FORKJOIN_BARRIER_PATTERN=dist,dist
KMP_PLAIN_BARRIER_PATTERN=dist,dist
KMP_REDUCTION_BARRIER_PATTERN=dist,dist

Branching factors (set with KMP_FORKJOIN_BARRIER, KMP_PLAIN_BARRIER,
and KMP_REDUCTION_BARRIER) are ignored by the two-level distributed
barrier.

Patch fixed for ITTNotify disabled builds and non-x86 builds

Co-authored-by: Jonathan Peyton <[email protected]>
Co-authored-by: Vladislav Vinogradov <[email protected]>

Differential Revision: https://reviews.llvm.org/D103121

show more ...

[OpenMP] Fix deadlock for detachable task with child tasks

This patch fixes https://bugs.llvm.org/show_bug.cgi?id=49066.

For detachable tasks, the assumption breaks that the proxy task cannot have

[OpenMP] Fix deadlock for detachable task with child tasks

This patch fixes https://bugs.llvm.org/show_bug.cgi?id=49066.

For detachable tasks, the assumption breaks that the proxy task cannot have
remaining child tasks when the proxy completes.
In stead of increment/decrement the incomplete task count, a high-order bit
is flipped to mark and wait for the incomplete proxy task.

Differential Revision: https://reviews.llvm.org/D101082

show more ...

[OpenMP] Fix bug 50022

Bug 50022 [0] reports target nowait fails in certain case, which is added in this
patch. The root cause of the failure is, when the second task is created, its
parent's `td_in

[OpenMP] Fix bug 50022

Bug 50022 [0] reports target nowait fails in certain case, which is added in this
patch. The root cause of the failure is, when the second task is created, its
parent's `td_incomplete_child_tasks` will not be incremented because there is no
parallel region here thus its team is serialized. Therefore, when the initial
thread is waiting for its unfinished children tasks, it thought there is only
one, the first task, because it is hidden helper task, so it is tracked. The
second task will only be pushed to the queue when the first task is finished.
However, when the first task finishes, it first decrements the counter of its
parent, and then release dependences. Once the counter is decremented, the thread
will move on because its counter is reset, but actually, the second task has not
been executed at all. As a result, since in this case, the main function finishes,
then `libomp` starts to destroy. When the second task is pushed somewhere, all
some of the structures might already have already been destroyed, then anything
could happen.

This patch simply moves `__kmp_release_deps` ahead of decrement of the counter.
In this way, we can make sure that the initial thread is aware of the existence
of another task(s) so it will not move on. In addition, in order to tackle
dependence chain starting with hidden helper thread, when hidden helper task is
encountered, we force the task to release dependences.

Reference:
[0] https://bugs.llvm.org/show_bug.cgi?id=50022

Reviewed By: AndreyChurbanov

Differential Revision: https://reviews.llvm.org/D106519

show more ...

[OpenMP] Refined the logic to give a regular task from a hidden helper task

In current implementation, if a regular task depends on a hidden helper task,
and when the hidden helper task is releasing

[OpenMP] Refined the logic to give a regular task from a hidden helper task

In current implementation, if a regular task depends on a hidden helper task,
and when the hidden helper task is releasing its dependences, it directly calls
`__kmp_omp_task`. This could cause a problem that if `__kmp_push_task` returns
`TASK_NOT_PUSHED`, the task will be executed immediately. However, the hidden
helper threads are assumed to only execute hidden helper tasks. This could cause
problems because when calling `__kmp_omp_task`, the encountering gtid, which is
not the real one of the thread, is passed.

This patch uses `__kmp_give_task`, but because it is a static function, a new
wrapper `__kmpc_give_task` is added.

Reviewed By: AndreyChurbanov

Differential Revision: https://reviews.llvm.org/D106572

show more ...

[OpenMP] Fixed a segmentation fault when using taskloop and target nowait

The synchronization of task loop misses hidden helper tasks, causing segmentation
fault reported in https://bugs.llvm.org/sh

[OpenMP] Fixed a segmentation fault when using taskloop and target nowait

The synchronization of task loop misses hidden helper tasks, causing segmentation
fault reported in https://bugs.llvm.org/show_bug.cgi?id=50002.

Reviewed By: ye-luo

Differential Revision: https://reviews.llvm.org/D106220

show more ...

[OpenMP] Minor improvement in task allocation

This patch includes a few changes to improve task allocation
performance slightly. These changes are enough to restore performance
drop observed after i

[OpenMP] Minor improvement in task allocation

This patch includes a few changes to improve task allocation
performance slightly. These changes are enough to restore performance
drop observed after introducing hidden helper.

Differential Revision: https://reviews.llvm.org/D105715

show more ...

[OpenMP] Remove TSAN annotations from libomp

The annotations in libomp were never built by default. The annotations are
also superseded by the annotations which the OMPT tool libarcher.so provides.

[OpenMP] Remove TSAN annotations from libomp

The annotations in libomp were never built by default. The annotations are
also superseded by the annotations which the OMPT tool libarcher.so provides.
With respect to libarcher, libomp behaves as if libarcher would be the last
element of OMP_TOOL_LIBARARIES. I.e., if no other OMPT tool gets active,
libarcher will check if an OpenMP application is built with TSan.

Since libarcher gets loaded by default, enabling LIBOMP_TSAN_SUPPORT would
result in redundant annotations for TSan, which slightly differ in details
and coverage (e.g. task dependencies are not handled well by the annotations
in libomp).

This patch removes all TSan annotations from the OpenMP runtime code.

Differential Revision: https://reviews.llvm.org/D103767

show more ...

[OpenMP] Fix a few issues with hidden helper task

This patch includes the following changes to address a few issues when
using hidden helper task.

- Assertion is triggered when there are inadverten

[OpenMP] Fix a few issues with hidden helper task

This patch includes the following changes to address a few issues when
using hidden helper task.

- Assertion is triggered when there are inadvertent calls to hidden
helper functions on non-Linux OS
- Added deinit code in __kmp_internal_end_library function to fix random
shutdown crashes
- Moved task data access into the lock-guarded region in __kmp_push_task

Differential Revision: https://reviews.llvm.org/D105308

show more ...

Revert "[OpenMP] Add Two-level Distributed Barrier"

This reverts commit 25073a4ecfc9b2e3cb76776185e63bfdb094cd98.

This breaks non-x86 OpenMP builds for a while now. Until a solution is
ready to be

Revert "[OpenMP] Add Two-level Distributed Barrier"

This reverts commit 25073a4ecfc9b2e3cb76776185e63bfdb094cd98.

This breaks non-x86 OpenMP builds for a while now. Until a solution is
ready to be upstreamed we revert the feature and unblock those builds.
See:
https://reviews.llvm.org/rG25073a4ecfc9b2e3cb76776185e63bfdb094cd98#1005821
and
https://reviews.llvm.org/rG25073a4ecfc9b2e3cb76776185e63bfdb094cd98#1005821

The currently proposed fix (D104788) seems not to be ready yet:
https://reviews.llvm.org/D104788#2841928

show more ...

[OpenMP] Add Two-level Distributed Barrier

Two-level distributed barrier is a new experimental barrier designed
for Intel hardware that has better performance in some cases than the
default hyper ba

[OpenMP] Add Two-level Distributed Barrier

Two-level distributed barrier is a new experimental barrier designed
for Intel hardware that has better performance in some cases than the
default hyper barrier.

This barrier is designed to handle fine granularity parallelism where
barriers are used frequently with little compute and memory access
between barriers. There is no need to use it for codes with few
barriers and large granularity compute, or memory intensive
applications, as little difference will be seen between this barrier
and the default hyper barrier. This barrier is designed to work
optimally with a fixed number of threads, and has a significant setup
time, so should NOT be used in situations where the number of threads
in a team is varied frequently.

The two-level distributed barrier is off by default -- hyper barrier
is used by default. To use this barrier, you must set all barrier
patterns to use this type, because it will not work with other barrier
patterns. Thus, to turn it on, the following settings are required:

KMP_FORKJOIN_BARRIER_PATTERN=dist,dist
KMP_PLAIN_BARRIER_PATTERN=dist,dist
KMP_REDUCTION_BARRIER_PATTERN=dist,dist

Branching factors (set with KMP_FORKJOIN_BARRIER, KMP_PLAIN_BARRIER,
and KMP_REDUCTION_BARRIER) are ignored by the two-level distributed
barrier.

Differential Revision: https://reviews.llvm.org/D103121

show more ...

[OpenMP][OMPD] Implementation of OMPD debugging library - libompd.

This is the first of seven patches that implements OMPD, a debugging interface to support debugging of OpenMP programs.
It contains

[OpenMP][OMPD] Implementation of OMPD debugging library - libompd.

This is the first of seven patches that implements OMPD, a debugging interface to support debugging of OpenMP programs.
It contains support code required in "openmp/runtime" for OMPD implementation.

Reviewed By: @hbae
Differential Revision: https://reviews.llvm.org/D100181

show more ...