[IRBuilder] Add assert for AtomicRMW ordering

Add assert for AtomicRMW: Ordering != AtomicOrdering::Unordered
(https://github.com/llvm/llvm-project/blob/main/llvm/lib/IR/Verifier.cpp#L3944)
and adju

[IRBuilder] Add assert for AtomicRMW ordering

Add assert for AtomicRMW: Ordering != AtomicOrdering::Unordered
(https://github.com/llvm/llvm-project/blob/main/llvm/lib/IR/Verifier.cpp#L3944)
and adjust expandAtomicStore accordingly.

Test plan:
1/ ninja check-llvm check-clang check-lld
2/ Bootstrapped LLVM/Clang pass tests

Differential revision: https://reviews.llvm.org/D130457

show more ...

[CodeGen] Qualify auto variables in for loops (NFC)

[LLVM] Add the support for fmax and fmin in atomicrmw instruction

This patch adds the support for `fmax` and `fmin` operations in `atomicrmw`
instruction. For now (at least in this patch), the instr

[LLVM] Add the support for fmax and fmin in atomicrmw instruction

This patch adds the support for `fmax` and `fmin` operations in `atomicrmw`
instruction. For now (at least in this patch), the instruction will be expanded
to CAS loop. There are already a couple of targets supporting the feature. I'll
create another patch(es) to enable them accordingly.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D127041

show more ...

Allow pointer types for atomicrmw xchg

This adds support for pointer types for `atomic xchg` and let us write
instructions such as `atomicrmw xchg i64** %0, i64* %1 seq_cst`. This
is similar to the

Allow pointer types for atomicrmw xchg

This adds support for pointer types for `atomic xchg` and let us write
instructions such as `atomicrmw xchg i64** %0, i64* %1 seq_cst`. This
is similar to the patch for allowing atomicrmw xchg on floating point
types: https://reviews.llvm.org/D52416.

Differential Revision: https://reviews.llvm.org/D124728

show more ...

[LLVM] Add a check if should cast atomic operations to integer type

Currently for atomic load, store, and rmw instructions, as long as the
operand is floating-point value, they are casted to integer

[LLVM] Add a check if should cast atomic operations to integer type

Currently for atomic load, store, and rmw instructions, as long as the
operand is floating-point value, they are casted to integer. Nowadays many
targets can actually support part of atomic operations with floating-point
operands. For example, NVPTX supports atomic load and store of floating-point
values. This patch adds a series interface functions `shouldCastAtomicXXXInIR`,
and the default implementations are same as what we currently do. Later for
targets can have their specialization.

Reviewed By: jdoerfert

Differential Revision: https://reviews.llvm.org/D125652

show more ...

Transforms: Fix code duplication between LowerAtomic and AtomicExpand

AtomicExpand: Add NotAtomic lowering strategy

Currently LowerAtomics exists as a separate pass which blindly
replaces all atomics. Add a new lowering strategy option to eliminate
the atomics which t

AtomicExpand: Add NotAtomic lowering strategy

Currently LowerAtomics exists as a separate pass which blindly
replaces all atomics. Add a new lowering strategy option to eliminate
the atomics which the target can control on a per-instruction level.

show more ...

AtomicExpand: Change return type for shouldExpandAtomicStoreInIR

Use the same enum as the other atomic instructions for consistency, in
preparation for addition of another strategy.

Introduce a new

AtomicExpand: Change return type for shouldExpandAtomicStoreInIR

Use the same enum as the other atomic instructions for consistency, in
preparation for addition of another strategy.

Introduce a new "Expand" option, since the store expansion does not
use cmpxchg. Alternatively, the existing CmpXChg strategy could be
renamed to Expand.

show more ...

Fix computation of MadeChange bit in AtomicExpandPass.

Fixes llvm-clang-x86_64-expensive-checks-debian failure with 2f497ec3.

expandAtomicStore always modifies the function, so make sure we set
Mad

Fix computation of MadeChange bit in AtomicExpandPass.

Fixes llvm-clang-x86_64-expensive-checks-debian failure with 2f497ec3.

expandAtomicStore always modifies the function, so make sure we set
MadeChange unconditionally. Not sure how nobody else has stumbled over
this before.

show more ...

[AtomicExpand][PowerPC] Fix all-one mask value

When generating a all-one mask value whose bitwidth is larger than 64, signed extension should be used rather then zero extension.

Reviewed By: jsji

[AtomicExpand][PowerPC] Fix all-one mask value

When generating a all-one mask value whose bitwidth is larger than 64, signed extension should be used rather then zero extension.

Reviewed By: jsji

Differential Revision: https://reviews.llvm.org/D120865

show more ...

[AtomicExpandPass][NFC] Reformat with clang-format

NFCI.

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-in

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681

show more ...

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https:/

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169

show more ...

Cleanup codegen includes

after: 1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169

[X86] Use bit test instructions to optimize some logic atomic operations

This is to match GCC's optimizations: https://gcc.godbolt.org/z/3odh9e7WE

Reviewed By: craig.topper

Differential Revision:

[X86] Use bit test instructions to optimize some logic atomic operations

This is to match GCC's optimizations: https://gcc.godbolt.org/z/3odh9e7WE

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D120199

show more ...

[llvm] Use range-based for loops with instructions (NFC)

[Remarks] [AMDGPU] Emit optimization remarks for atomics generating hardware instructions

Produce remarks when atomic instructions are expanded into hardware instructions
in SIISelLowering.cpp. Curr

[Remarks] [AMDGPU] Emit optimization remarks for atomics generating hardware instructions

Produce remarks when atomic instructions are expanded into hardware instructions
in SIISelLowering.cpp. Currently, these remarks are only emitted for atomic fadd
instructions.

Differential Revision: https://reviews.llvm.org/D108150

show more ...

[NFC] Cleanup more AttributeList::addAttribute()

[Remarks] Emit optimization remarks for atomics generating CAS loop

Implements ORE in AtomicExpand pass to report atomics generating a
compare and swap loop.

Differential Revision: https://reviews.

[Remarks] Emit optimization remarks for atomics generating CAS loop

Implements ORE in AtomicExpand pass to report atomics generating a
compare and swap loop.

Differential Revision: https://reviews.llvm.org/D106891

show more ...

Revert "[Remarks] Emit optimization remarks for atomics generating CAS loop"

This reverts commit 435785214f73ff0c92e97f2ade6356e3ba3bf661. Still same compile time issues for -O0 -g, eg. +1.3% for sq

Revert "[Remarks] Emit optimization remarks for atomics generating CAS loop"

This reverts commit 435785214f73ff0c92e97f2ade6356e3ba3bf661. Still same compile time issues for -O0 -g, eg. +1.3% for sqlite3.

show more ...

[Remarks] Emit optimization remarks for atomics generating CAS loop

Implements ORE in AtomicExpand pass to report atomics generating
a compare and swap loop.

Differential Revision: https://reviews.

[Remarks] Emit optimization remarks for atomics generating CAS loop

Implements ORE in AtomicExpand pass to report atomics generating
a compare and swap loop.

Differential Revision: https://reviews.llvm.org/D106891

show more ...

Revert "[Remarks] Emit optimization remarks for atomics generating CAS loop"

This reverts commit c4e5425aa579d21530ef1766d7144b38a347f247.

[Remarks] Emit optimization remarks for atomics generating CAS loop

Implements ORE in AtomicExpandPass to report atomics generating a compare
and swap loop.

Differential Revision: https://reviews.l

[Remarks] Emit optimization remarks for atomics generating CAS loop

Implements ORE in AtomicExpandPass to report atomics generating a compare
and swap loop.

Differential Revision: https://reviews.llvm.org/D106891

show more ...

[PowerPC] Generate inlined quadword lock free atomic operations via AtomicExpand

This patch uses AtomicExpandPass to implement quadword lock free atomic operations. It adopts the method introduced i

[PowerPC] Generate inlined quadword lock free atomic operations via AtomicExpand

This patch uses AtomicExpandPass to implement quadword lock free atomic operations. It adopts the method introduced in https://reviews.llvm.org/D47882, which expand atomic operations post RA to avoid spilling that might prevent LL/SC progress.

Reviewed By: jsji

Differential Revision: https://reviews.llvm.org/D103614

show more ...

[OpaquePtr] Add type parameter to emitLoadLinked

Differential Revision: https://reviews.llvm.org/D105353