[Clang] Ensure vector predication loop metadata is always emitted when pragma is specified.

This patch ensures that vector predication and vectorization width
pragmas work together correctly/as expe

[Clang] Ensure vector predication loop metadata is always emitted when pragma is specified.

This patch ensures that vector predication and vectorization width
pragmas work together correctly/as expected. Specifically, this patch
fixes the issue that when vectorization_width > 1, the vector
predication behaviour (this would matter if it has NOT been disabled
explicitly by a pragma) was getting ignored, which was incorrect.

The fix here removes the dependence of vector predication on the
vectorization width. The loop metadata corresponding to clang loop
pragma vectorize_predicate is always emitted, if the pragma is
specified, even if vectorization is disabled by vectorize_width(1)
or vectorize(disable) since the option is also used for interleaving
by the LoopVectorize pass.

Reviewed By: dmgreen, Meinersbur

Differential Revision: https://reviews.llvm.org/D94779

show more ...

[SVE] Add support to vectorize_width loop pragma for scalable vectors

This patch adds support for two new variants of the vectorize_width
pragma:

1. vectorize_width(X[, fixed|scalable]) where an op

[SVE] Add support to vectorize_width loop pragma for scalable vectors

This patch adds support for two new variants of the vectorize_width
pragma:

1. vectorize_width(X[, fixed|scalable]) where an optional second
parameter is passed to the vectorize_width pragma, which indicates if
the user wishes to use fixed width or scalable vectorization. For
example the user can now write something like:

#pragma clang loop vectorize_width(4, fixed)
or
#pragma clang loop vectorize_width(4, scalable)

In the absence of a second parameter it is assumed the user wants
fixed width vectorization, in order to maintain compatibility with
existing code.
2. vectorize_width(fixed|scalable) where the width is left unspecified,
but the user hints what type of vectorization they prefer, either
fixed width or scalable.

I have implemented this by making use of the LLVM loop hint attribute:

llvm.loop.vectorize.scalable.enable

Tests were added to

clang/test/CodeGenCXX/pragma-loop.cpp

for both the 'fixed' and 'scalable' optional parameter.

See this thread for context: http://lists.llvm.org/pipermail/cfe-dev/2020-November/067262.html

Differential Revision: https://reviews.llvm.org/D89031

show more ...

[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction

Since C++11, the C++ standard has a forward progress guarantee
[intro.progress], so all such functions must have the `mustprog

[clang] Add mustprogress and llvm.loop.mustprogress attribute deduction

Since C++11, the C++ standard has a forward progress guarantee
[intro.progress], so all such functions must have the `mustprogress`
requirement. In addition, from C11 and onwards, loops without a non-zero
constant conditional or no conditional are also required to make
progress (C11 6.8.5p6). This patch implements these attribute deductions
so they can be used by the optimization passes.

Differential Revision: https://reviews.llvm.org/D86841

show more ...

Avoid unnecessary uses of `MDNode::getTemporary`, NFC

This is a long-delayed follow-up to
5e5b85098dbeaea2cfa5d01695b5d2982634d7dd.

`TempMDNode` includes a bunch of machinery for RAUW, and should o

Avoid unnecessary uses of `MDNode::getTemporary`, NFC

This is a long-delayed follow-up to
5e5b85098dbeaea2cfa5d01695b5d2982634d7dd.

`TempMDNode` includes a bunch of machinery for RAUW, and should only be
used when necessary. RAUW wasn't being used in any of these cases... it
was just a placeholder for a self-reference.

Where the real node was using `MDNode::getDistinct`, just replace the
temporary argument with `nullptr`.

Where the real node was using `MDNode::get`, the `replaceOperandWith`
call was "promoting" the node to a distinct one implicitly due to
self-reference detection in `MDNode::handleChangedOperand`. The
`TempMDNode` was serving a purpose by delaying uniquing, but it's way
simpler to just call `MDNode::getDistinct` in the first place.

Note that using a self-reference at all in these places is a hold-over
from before `distinct` metadata existed. It was an old trick to create
distinct nodes. It would be intrusive to change, including bitcode
upgrades, etc., and it's harmless so I'm not sure there's much value in
removing it from existing schemas. After this commit it still has a tiny
memory cost (in the extra metadata operand) but no more overhead in
construction.

Differential Revision: https://reviews.llvm.org/D90079

show more ...

[Clang] Add llvm.loop.unroll.disable to loops with -fno-unroll-loops.

Currently Clang does not respect -fno-unroll-loops during LTO. During
D76916 it was suggested to respect -fno-unroll-loops on a

[Clang] Add llvm.loop.unroll.disable to loops with -fno-unroll-loops.

Currently Clang does not respect -fno-unroll-loops during LTO. During
D76916 it was suggested to respect -fno-unroll-loops on a TU basis.

This patch uses the existing llvm.loop.unroll.disable metadata to
disable loop unrolling explicitly for each loop in the TU if
unrolling is disabled. This should ensure that loops from TUs compiled
with -fno-unroll-loops are skipped by the unroller during LTO.

This also means that if a loop from a TU with -fno-unroll-loops
gets inlined into a TU without this option, the loop won't be
unrolled.

Due to the fact that some transforms might drop loop metadata, there
potentially are cases in which we still unroll loops from TUs with
-fno-unroll-loops. I think we should fix those issues rather than
introducing a function attribute to disable loop unrolling during LTO.
Improving the metadata handling will benefit other use cases, like
various loop pragmas, too. And it is an improvement to clang completely
ignoring -fno-unroll-loops during LTO.

If that direction looks good, we can use a similar approach to also
respect -fno-vectorize during LTO, at least for LoopVectorize.

In the future, this might also allow us to remove the UnrollLoops option
LLVM's PassManagerBuilder.

Reviewers: Meinersbur, hfinkel, dexonsmith, tejohnson

Reviewed By: Meinersbur, tejohnson

Differential Revision: https://reviews.llvm.org/D77058

show more ...

Sink more Attr.h inline methods, NFC

This has very little impact on build time, but is a mechanical pre-req
to removing the OpenMPClause.h include, which matters. Most of these
pretty print methods

Sink more Attr.h inline methods, NFC

This has very little impact on build time, but is a mechanical pre-req
to removing the OpenMPClause.h include, which matters. Most of these
pretty print methods require Expr to be complete.

show more ...

[Clang] Pragma vectorize_width() implies vectorize(enable)

Let's try this again; this has been reverted/recommited a few times. Last time
this got reverted because for this loop:

void a() {
#

[Clang] Pragma vectorize_width() implies vectorize(enable)

Let's try this again; this has been reverted/recommited a few times. Last time
this got reverted because for this loop:

void a() {
#pragma clang loop vectorize(disable)
for (;;)
;
}

vectorisation was incorrectly enabled and the vectorize.enable metadata was set
due to a logic error. But with this fixed, we now imply vectorisation when:

1) vectorisation is enabled, which means: VectorizeWidth > 1,
2) and don't want to add it when it is disabled or enabled, otherwise we would
be incorrectly setting it or duplicating the metadata, respectively.

This should fix PR27643.

Differential Revision: https://reviews.llvm.org/D69628

show more ...

Revert "Recommit "[Clang] Pragma vectorize_width() implies vectorize(enable)""

This reverts commit 80371c74ae63d2f260bcc75408be9c6f81e38465.

Given the following source:
```
void a() {
for (;;)

Revert "Recommit "[Clang] Pragma vectorize_width() implies vectorize(enable)""

This reverts commit 80371c74ae63d2f260bcc75408be9c6f81e38465.

Given the following source:
```
void a() {
for (;;)
;
}
```

It incorrectly enables vectorization (with vector width 1), as well as generating a warning that vectorization could not be performed.

show more ...

Recommit "[Clang] Pragma vectorize_width() implies vectorize(enable)"

This was further discussed at the llvm dev list:

http://lists.llvm.org/pipermail/llvm-dev/2019-October/135602.html

I think the

Recommit "[Clang] Pragma vectorize_width() implies vectorize(enable)"

This was further discussed at the llvm dev list:

http://lists.llvm.org/pipermail/llvm-dev/2019-October/135602.html

I think the brief summary of that is that this change is an improvement,
this is the behaviour that we expect and promise in ours docs, and also
as a result there are cases where we now emit diagnostics whereas before
pragmas were silently ignored. Two areas where we can improve: 1) the
diagnostic message itself, and 2) and in some cases (e.g. -Os and -Oz)
the vectoriser is (quite understandably) not triggering.

Original commit message:

Specifying the vectorization width was supposed to implicitly enable
vectorization, except that it wasn't really doing this. It was only
setting the vectorize.width metadata, but not vectorize.enable.

This should fix PR27643.

llvm-svn: 374288

show more ...

Revert r372082 "[Clang] Pragma vectorize_width() implies vectorize(enable)"

This broke the Chromium build. Consider the following code:

float ScaleSumSamples_C(const float* src, float* dst, float

Revert r372082 "[Clang] Pragma vectorize_width() implies vectorize(enable)"

This broke the Chromium build. Consider the following code:

float ScaleSumSamples_C(const float* src, float* dst, float scale, int width) {
float fsum = 0.f;
int i;
#if defined(__clang__)
#pragma clang loop vectorize_width(4)
#endif
for (i = 0; i < width; ++i) {
float v = *src++;
fsum += v * v;
*dst++ = v * scale;
}
return fsum;
}

Compiling at -Oz, Clang now warns:

$ clang++ -target x86_64 -Oz -c /tmp/a.cc
/tmp/a.cc:1:7: warning: loop not vectorized: the optimizer was unable to
perform the requested transformation; the transformation might be disabled or
specified as part of an unsupported transformation ordering
[-Wpass-failed=transform-warning]

this suggests it's not actually enabling vectorization hard enough.

At -Os it asserts instead:

$ build.release/bin/clang++ -target x86_64 -Os -c /tmp/a.cc
clang-10: /work/llvm.monorepo/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp:2734: void
llvm::InnerLoopVectorizer::emitMemRuntimeChecks(llvm::Loop*, llvm::BasicBlock*): Assertion `
!BB->getParent()->hasOptSize() && "Cannot emit memory checks when optimizing for size"' failed.

Of course neither of these are what the developer expected from the pragma.

> Specifying the vectorization width was supposed to implicitly enable
> vectorization, except that it wasn't really doing this. It was only
> setting the vectorize.width metadata, but not vectorize.enable.
>
> This should fix PR27643.
>
> Differential Revision: https://reviews.llvm.org/D66290

llvm-svn: 372225

show more ...

[Clang] Pragma vectorize_width() implies vectorize(enable)

Specifying the vectorization width was supposed to implicitly enable
vectorization, except that it wasn't really doing this. It was only
se

[Clang] Pragma vectorize_width() implies vectorize(enable)

Specifying the vectorization width was supposed to implicitly enable
vectorization, except that it wasn't really doing this. It was only
setting the vectorize.width metadata, but not vectorize.enable.

This should fix PR27643.

Differential Revision: https://reviews.llvm.org/D66290

llvm-svn: 372082

show more ...

Don't keep stale pointers to LoopInfos.

CGLoopInfo was keeping pointers to parent loop LoopInfos, but when the loop info vector grew, it reallocated the storage and invalidated all of the parent poi

Don't keep stale pointers to LoopInfos.

CGLoopInfo was keeping pointers to parent loop LoopInfos, but when the loop info vector grew, it reallocated the storage and invalidated all of the parent pointers, causing use-after-free. Manage the lifetimes of the LoopInfos separately so that the pointers aren't stale.

Patch by Bevin Hansson.

llvm-svn: 369259

show more ...

[Clang] Pragma vectorize_predicate implies vectorize

New pragma "vectorize_predicate(enable)" now implies "vectorize(enable)",
and it is ignored when vectorization is disabled with e.g.
"vectorize(d

[Clang] Pragma vectorize_predicate implies vectorize

New pragma "vectorize_predicate(enable)" now implies "vectorize(enable)",
and it is ignored when vectorization is disabled with e.g.
"vectorize(disable) vectorize_predicate(enable)".

Differential Revision: https://reviews.llvm.org/D65776

llvm-svn: 368970

show more ...

[Clang] New loop pragma vectorize_predicate

This adds a new vectorize predication loop hint:

#pragma clang loop vectorize_predicate(enable)

that can be used to indicate to the vectoriser that al

[Clang] New loop pragma vectorize_predicate

This adds a new vectorize predication loop hint:

#pragma clang loop vectorize_predicate(enable)

that can be used to indicate to the vectoriser that all (load/store)
instructions should be predicated (masked). This allows, for example, folding
of the remainder loop into the main loop.

This patch will be followed up with D64916 and D65197. The former is a
refactoring in the loopvectorizer and the groundwork to make tail loop folding
a more general concept, and in the latter the actual tail loop folding
transformation will be implemented.

Differential Revision: https://reviews.llvm.org/D64744

llvm-svn: 366989

show more ...

[CodeGen] Generate follow-up metadata for loops with more than one transformation.

Before this patch, CGLoop would dump all transformations for a loop into
a single LoopID without encoding any order

[CodeGen] Generate follow-up metadata for loops with more than one transformation.

Before this patch, CGLoop would dump all transformations for a loop into
a single LoopID without encoding any order in which to apply them.
rL348944 added the possibility to encode a transformation order using
followup-attributes.

When a loop has more than one transformation, use the follow-up
attribute define the order in which they are applied. The emitted order
is the defacto order as defined by the current LLVM pass pipeline,
which is:

LoopFullUnrollPass
LoopDistributePass
LoopVectorizePass
LoopUnrollAndJamPass
LoopUnrollPass
MachinePipeliner

This patch should therefore not change the assembly output, assuming
that all explicit transformations can be applied, and no implicit
transformations in-between. In the former case,
WarnMissedTransformationsPass should emit a warning (except for
MachinePipeliner which is not implemented yet). The latter could be
avoided by adding 'llvm.loop.disable_nonforced' attributes.

Because LoopUnrollAndJamPass processes a loop nest, generation of the
MDNode is delayed to after the inner loop metadata have been processed.
A temporary LoopID is therefore used to annotate instructions and
RAUW'ed by the actual LoopID later.

Differential Revision: https://reviews.llvm.org/D57978

llvm-svn: 357415

show more ...

[OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint))

Summary:
[OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint))

For both !{!"llv

[OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint))

Summary:
[OpenCL] Generate 'unroll.enable' metadata for __attribute__((opencl_unroll_hint))

For both !{!"llvm.loop.unroll.enable"} and !{!"llvm.loop.unroll.full"} the unroller
will try to fully unroll a loop unless the trip count is not known at compile time.
In that case for '.full' metadata no unrolling will be processed, while for '.enable'
the loop will be partially unrolled with a heuristically chosen unroll factor.

See: docs/LanguageExtensions.rst

From https://www.khronos.org/registry/OpenCL/sdk/2.0/docs/man/xhtml/attributes-loopUnroll.html

__attribute__((opencl_unroll_hint))
for (int i=0; i<2; i++)
{
...
}

In the example above, the compiler will determine how much to unroll the loop.


Before the patch for __attribute__((opencl_unroll_hint)) was generated metadata
!{!"llvm.loop.unroll.full"}, which limits ability of loop unroller to decide, how
much to unroll the loop.

Reviewers: Anastasia, yaxunl

Reviewed By: Anastasia

Subscribers: zzheng, dmgreen, jdoerfert, cfe-commits, asavonic, AlexeySotkin

Tags: #clang

Differential Revision: https://reviews.llvm.org/D59493

llvm-svn: 356571

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

Add two new pragmas for controlling software pipelining optimizations.

This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compi

Add two new pragmas for controlling software pipelining optimizations.

This patch adds #pragma clang loop pipeline and #pragma clang loop pipeline_initiation_interval for debugging or reducing compile time purposes. It is possible to disable SWP for concrete loops to save compilation time or to find bugs by not doing SWP to certain loops. It is possible to set value of initiation interval to concrete number to save compilation time by not doing extra pipeliner passes or to check created schedule for specific initiation interval.

Patch by Alexey Lapshin.

llvm-svn: 350414

show more ...

[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata.

Instead of generating llvm.mem.parallel_loop_access metadata, generate
llvm.access.group on instruct

[CodeGen] Generate llvm.loop.parallel_accesses instead of llvm.mem.parallel_loop_access metadata.

Instead of generating llvm.mem.parallel_loop_access metadata, generate
llvm.access.group on instructions and llvm.loop.parallel_accesses on
loops. There is one access group per generated loop.

This is clang part of D52116/r349725.

Differential Revision: https://reviews.llvm.org/D52117

llvm-svn: 349823

show more ...

Move LoopHint.h from Sema to Parse

struct LoopHint was only used within Parse and not in any of the Sema or
Codegen files. In the non-Parse files where it was included, it either wasn't
used or Loo

Move LoopHint.h from Sema to Parse

struct LoopHint was only used within Parse and not in any of the Sema or
Codegen files. In the non-Parse files where it was included, it either wasn't
used or LoopHintAttr was used, so its inclusion did nothing.

llvm-svn: 347728

show more ...

[TI removal] Test predicate rather than casting to detect a terminator
and use the range based successor API.

llvm-svn: 344730

[CodeGen] Emit parallel_loop_access for each loop in the loop stack.

Summary:
Emit !llvm.mem.parallel_loop_access metadata for memory accesses even if the parallel loop is not the top on the loop st

[CodeGen] Emit parallel_loop_access for each loop in the loop stack.

Summary:
Emit !llvm.mem.parallel_loop_access metadata for memory accesses even if the parallel loop is not the top on the loop stack.

Fixes llvm.org/PR37558.

Reviewers: ABataev, hfinkel, amusman, tyler.nowicki

Reviewed By: hfinkel

Subscribers: Meinersbur, hfinkel, cfe-commits

Differential Revision: https://reviews.llvm.org/D48808

llvm-svn: 338810

show more ...

[UnrollAndJam] Add unroll_and_jam pragma handling

This adds support for the unroll_and_jam pragma, to go with the recently
added unroll and jam pass. The name of the pragma is the same as is used
in

[UnrollAndJam] Add unroll_and_jam pragma handling

This adds support for the unroll_and_jam pragma, to go with the recently
added unroll and jam pass. The name of the pragma is the same as is used
in the Intel compiler, and most of the code works the same as for unroll.

#pragma clang loop unroll_and_jam has been separated into a different
patch. This part adds #pragma unroll_and_jam with an optional count, and
#pragma no_unroll_and_jam to disable the transform.

Differential Revision: https://reviews.llvm.org/D47267

llvm-svn: 338566

show more ...

Remove trailing space

sed -Ei 's/[[:space:]]+$//' include/**/*.{def,h,td} lib/**/*.{cpp,h}

llvm-svn: 338291

[CodeGen] Pass objects that are expensive to copy by const ref.

No functionality change. Found by clang-tidy's
performance-unnecessary-value-param.

llvm-svn: 287894