[LSV] Change the default value of InstertElement to poison

This patch is changing the InsertElement's placeholder to poison without changing the LSV's behavior.

Regardless of whether `StoreTy` is F

[LSV] Change the default value of InstertElement to poison

This patch is changing the InsertElement's placeholder to poison without changing the LSV's behavior.

Regardless of whether `StoreTy` is FixedVectorType or not, the poison value will be overwritten with a different value.
Therefore, whether the InsertElement's placeholder is poison or undef will not affect the result of the program.

Reviewed By: nikic

Differential Revision: https://reviews.llvm.org/D111005

show more ...

Revert "Revert "[AMDGPU] Reorganize GCN subtarget features for unaligned access""

This reverts commit 8b08fa0103c8d8e624b19fad5a5006e7a783ecb7.

The underlying problems were fixed by D90607.

Revert "[AMDGPU] Reorganize GCN subtarget features for unaligned access"

This reverts commit f5cd7ec9f3fc969ff5e1feed961996844333de3b.

Certain rocPRIM/rocThrust/hipCUB tests were failing because of

Revert "[AMDGPU] Reorganize GCN subtarget features for unaligned access"

This reverts commit f5cd7ec9f3fc969ff5e1feed961996844333de3b.

Certain rocPRIM/rocThrust/hipCUB tests were failing because of this change.

show more ...

[AMDGPU] Reorganize GCN subtarget features for unaligned access

Features UnalignedBufferAccess and UnalignedDSAccess are now used to determine
whether hardware supports such access.
UnalignedAccessM

[AMDGPU] Reorganize GCN subtarget features for unaligned access

Features UnalignedBufferAccess and UnalignedDSAccess are now used to determine
whether hardware supports such access.
UnalignedAccessMode should be used to enable them.
hasUnalignedBufferAccessEnabled() and hasUnalignedDSAccessEnabled() can be
now used to quickly check both.

Differential Revision: https://reviews.llvm.org/D84522

show more ...

[AMDGPU] Fix alignment requirements for 96bit and 128bit local loads and stores

Adjust alignment requirements for ds_read/write_b96/b128.
GFX9 and onwards allow misaligned access for reads and write

[AMDGPU] Fix alignment requirements for 96bit and 128bit local loads and stores

Adjust alignment requirements for ds_read/write_b96/b128.
GFX9 and onwards allow misaligned access for reads and writes but only if
SH_MEM_CONFIG.alignment_mode allows it.
UnalignedDSAccess is set on GCN subtargets from GFX9 onward to let us know if we
can relax alignment requirements.
UnalignedAccessMode acts similary to UnalignedBufferAccess for DS instructions
but only from GFX9 onward and is supposed to match alignment_mode. By default
alignment of 4 is required.

Differential Revision: https://reviews.llvm.org/D82788

show more ...

AMDGPU: Fix broken check lines

AMDGPU: Don't report 2-byte alignment as fast

This is apparently worse than 1-byte alignment. This does not attempt
to decompose 2-byte aligned wide stores, but will stop trying to
produce them.

Al

AMDGPU: Don't report 2-byte alignment as fast

This is apparently worse than 1-byte alignment. This does not attempt
to decompose 2-byte aligned wide stores, but will stop trying to
produce them.

Also fix bug in LoadStoreVectorizer which was decreasing the alignment
and vectorizing stack accesses. It was assuming a stack object was an
alloca that could have its base alignment changed, which is not true
if the pointer is derived from a function argument.

show more ...

Revert "Temporarily Revert "Add basic loop fusion pass.""

The reversion apparently deleted the test/Transforms directory.

Will be re-reverting again.

llvm-svn: 358552

AMDGPU: Fix some outdated datalayouts in tests

llvm-svn: 342131

[AMDGPU] Re-enabled 128bit wide-vector generation for local addr space by default.

Summary: Bug reported here https://bugs.freedesktop.org/show_bug.cgi?id=105464 found
to be resolved by som

[AMDGPU] Re-enabled 128bit wide-vector generation for local addr space by default.

Summary: Bug reported here https://bugs.freedesktop.org/show_bug.cgi?id=105464 found
to be resolved by some other fixes.

Author: FarhanaAleen
llvm-svn: 333380

show more ...

AMDGPU: enable 128-bit for local addr space under an option

Author: Samuel Pitoiset

ds_read_b128 and ds_write_b128 have been recently enabled
under the amdgpu-ds128 option because the performance b

AMDGPU: enable 128-bit for local addr space under an option

Author: Samuel Pitoiset

ds_read_b128 and ds_write_b128 have been recently enabled
under the amdgpu-ds128 option because the performance benefit
is unclear.

Though, using 128-bit loads/stores for the local address space
appears to introduce regressions in tessellation shaders. Not
sure what is broken, but as ds_read_b128/ds_write_b128 are not
enabled by default, just introduce a global option and enable
128-bit only if requested (until it's fixed/used correctly).

v2: - fix regressions in merge-stores.ll and multiple_tails.ll

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=105464
llvm-svn: 329764

show more ...

[AMDGPU] Supported ds_read_b128 generation; Widened vector length for local address-space.

Summary: Starting from GCN 2nd generation, ISA supports ds_read_b128 on top of ds_read_b64.
This p

[AMDGPU] Supported ds_read_b128 generation; Widened vector length for local address-space.

Summary: Starting from GCN 2nd generation, ISA supports ds_read_b128 on top of ds_read_b64.
This patch supports ds_read_b128 instruction pattern and generation of this instruction.
In the vectorizer, this patch also widen the vector length so that vectorizer generates
128 bit loads for local address-space which gets translated to ds_read_b128.
Since the performance benefit is not clear; compiler generates ds_read_b128 under -amdgpu-ds128.

Author: FarhanaAleen

Reviewed By: rampitec, arsenm

Subscribers: llvm-commits, AMDGPU

Differential Revision: https://reviews.llvm.org/D44210

llvm-svn: 327153

show more ...

[LSV] Avoid adding vectors of pointers as candidates

Summary:
We no longer add vectors of pointers as candidates for
load/store vectorization. It does not seem to work anyway,
but without this patch

[LSV] Avoid adding vectors of pointers as candidates

Summary:
We no longer add vectors of pointers as candidates for
load/store vectorization. It does not seem to work anyway,
but without this patch we can end up in asserts when trying
to create casts between an integer type and the pointer of
vectors type.

The test case I've added used to assert like this when trying to
cast between i64 and <2 x i16*>:
opt: ../lib/IR/Instructions.cpp:2565: Assertion `castIsValid(op, S, Ty) && "Invalid cast!"' failed.
#0 PrintStackTraceSignalHandler(void*)
#1 SignalHandler(int)
#2 __restore_rt
#3 __GI_raise
#4 __GI_abort
#5 __GI___assert_fail
#6 llvm::CastInst::Create(llvm::Instruction::CastOps, llvm::Value*, llvm::Type*, llvm::Twine const&, llvm::Instruction*)
#7 llvm::IRBuilder<llvm::ConstantFolder, llvm::IRBuilderDefaultInserter>::CreateBitOrPointerCast(llvm::Value*, llvm::Type*, llvm::Twine const&)
#8 Vectorizer::vectorizeStoreChain(llvm::ArrayRef<llvm::Instruction*>, llvm::SmallPtrSet<llvm::Instruction*, 16u>*)

Reviewers: arsenm

Reviewed By: arsenm

Subscribers: nhaehnle, llvm-commits

Differential Revision: https://reviews.llvm.org/D39296

llvm-svn: 316665

show more ...

AMDGPU: Mark all unspecified CC functions in tests as amdgpu_kernel

Currently the default C calling convention functions are treated
the same as compute kernels. Make this explicit so the default
ca

AMDGPU: Mark all unspecified CC functions in tests as amdgpu_kernel

Currently the default C calling convention functions are treated
the same as compute kernels. Make this explicit so the default
calling convention can be changed to a non-kernel.

Converted with perl -pi -e 's/define void/define amdgpu_kernel void/'
on the relevant test directories (and undoing in one place that actually
wanted a non-kernel).

llvm-svn: 298444

show more ...

[LSV] Use the original loads' names for the extractelement instructions.

Summary:
LSV replaces multiple adjacent loads with one vectorized load and a
bunch of extractelement instructions. This patc

[LSV] Use the original loads' names for the extractelement instructions.

Summary:
LSV replaces multiple adjacent loads with one vectorized load and a
bunch of extractelement instructions. This patch makes the
extractelement instructions' names match those of the original loads,
for (hopefully) improved readability.

Reviewers: asbirlea, tstellarAMD

Subscribers: arsenm, mzolotukhin

Differential Revision: https://reviews.llvm.org/D23748

llvm-svn: 280818

show more ...

Add TLI.allowsMisalignedMemoryAccesses to LoadStoreVectorizer

Summary: Extend TTI to access TLI.allowsMisalignedMemoryAccesses(). Check condition when vectorizing load and store chains.
Add addition

Add TLI.allowsMisalignedMemoryAccesses to LoadStoreVectorizer

Summary: Extend TTI to access TLI.allowsMisalignedMemoryAccesses(). Check condition when vectorizing load and store chains.
Add additional parameters: AddressSpace, Alignment, Fast.

Reviewers: llvm-commits, jlebar

Subscribers: arsenm, mzolotukhin

Differential Revision: http://reviews.llvm.org/D21935

llvm-svn: 275100

show more ...

LoadStoreVectorizer: Don't increase alignment with no align set

If no alignment was set on the load/stores, it would vectorize
to the new type even though this increases the default alignment.

llvm

LoadStoreVectorizer: Don't increase alignment with no align set

If no alignment was set on the load/stores, it would vectorize
to the new type even though this increases the default alignment.

llvm-svn: 274323

show more ...

LoadStoreVectorizer: Check TTI for vec reg bit width

llvm-svn: 274322

LoadStoreVectorizer: if one element of a vector is integer, default to
integer.

Fixes issues on some architectures where we use arithmetic ops to build
vectors, which can cause bad things to happen

LoadStoreVectorizer: if one element of a vector is integer, default to
integer.

Fixes issues on some architectures where we use arithmetic ops to build
vectors, which can cause bad things to happen for loads/stores of mixed
types.

Patch by Fiona Glaser

llvm-svn: 274307

show more ...

Add LoadStoreVectorizer pass

This was contributed by Apple, and I've been working on
minimal cleanups and generalizing it.

llvm-svn: 274293