Recommit "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 6a9bbd9f20dcd700e28738788bb63a160c6c088c.

Code review: https://reviews.llvm.or

Recommit "[SLP][TTI] Refactoring of `getShuffleCost` `Args` to work like `getArithmeticInstrCost`"

This reverts commit 6a9bbd9f20dcd700e28738788bb63a160c6c088c.

Code review: https://reviews.llvm.org/D124202

show more ...

[AMDGPU] Fix regression with vectorization limiting

D67148 has removed TTI::getNumberOfRegisters(bool Vector) and
started to call TTI::getNumberOfRegisters(unsigned ClassID) from
the LoopVectorize.

[AMDGPU] Fix regression with vectorization limiting

D67148 has removed TTI::getNumberOfRegisters(bool Vector) and
started to call TTI::getNumberOfRegisters(unsigned ClassID) from
the LoopVectorize. This has resulted in an unrestricted vectorization
on AMDGPU blowing up register pressure.

Differential Revision: https://reviews.llvm.org/D122850

show more ...

Add support for atomic memory copy lowering

Currently, the utility supports lowering of non atomic memory transfer routines only. This patch adds support for atomic version of memcopy. This may be u

Add support for atomic memory copy lowering

Currently, the utility supports lowering of non atomic memory transfer routines only. This patch adds support for atomic version of memcopy. This may be useful for targets not supporting atomic memcopy.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D118443

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 3, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit e6ead19b774718113007ecb1a4

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 3, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit e6ead19b774718113007ecb1a4449d7af0cbcfeb.

show more ...

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash."

This reverts commit 27bd8f94928201f87f6b659fc2228efd539e8245.

Causes crashes, see comme

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash."

This reverts commit 27bd8f94928201f87f6b659fc2228efd539e8245.

Causes crashes, see comments in D121973

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit f7d7d2a08d16356c57f6d2d36b

Recommit "[SLP] Fix lookahead operand reordering for splat loads." attempt 2, fixed assertion crash.

Original review: https://reviews.llvm.org/D121354

This reverts commit f7d7d2a08d16356c57f6d2d36bc2fc0589a55df9.

show more ...

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads.""

This reverts commit 79613185d305013de743cdbd6690e4d77c8af27e.

Causes crashes, see comments in https://reviews.llvm.org/D1

Revert "Recommit "[SLP] Fix lookahead operand reordering for splat loads.""

This reverts commit 79613185d305013de743cdbd6690e4d77c8af27e.

Causes crashes, see comments in https://reviews.llvm.org/D121973.

show more ...

Recommit "[SLP] Fix lookahead operand reordering for splat loads."

Original review: https://reviews.llvm.org/D121354

The original commit 9136145eb019e1d18c966d4d06a3df349b88cc14 broke the build on

Recommit "[SLP] Fix lookahead operand reordering for splat loads."

Original review: https://reviews.llvm.org/D121354

The original commit 9136145eb019e1d18c966d4d06a3df349b88cc14 broke the build on several targets.

Differential Revision: https://reviews.llvm.org/D121973

show more ...

[AMDGPU] Fixed physreg asm constraint parsing

We are always failing parsing of the physreg constraint because
we do not drop trailing brace, thus getAsInteger() returns a
non-empty string and we del

[AMDGPU] Fixed physreg asm constraint parsing

We are always failing parsing of the physreg constraint because
we do not drop trailing brace, thus getAsInteger() returns a
non-empty string and we delegate reparsing to the TargetLowering.

In addition it did not parse register tuples.

Fixed which has allowed to remove w/a in two places we call it.

Differential Revision: https://reviews.llvm.org/D117055

show more ...

[CostModel][AMDGPU] Fix intrinsics costs estimations.

1. Fixed costs inconsistency for llvm.fma.vXf16 instinsiscs.
2. Added tests for llvm.sadd.sat, llvm.ssub.sat, llvm.uadd.sat, llvm.usub.sat
in

[CostModel][AMDGPU] Fix intrinsics costs estimations.

1. Fixed costs inconsistency for llvm.fma.vXf16 instinsiscs.
2. Added tests for llvm.sadd.sat, llvm.ssub.sat, llvm.uadd.sat, llvm.usub.sat
intrisics since they have special processing in cost model.
3. Minor intrisics' costs tests updat and refinement.

Reviewed By: rampitec

Differential Revision: https://reviews.llvm.org/D115385

show more ...

[CostModel][AMDGPU] Fix instructions costs estimation for vector types.

1. Fixed vector instructions costs estimations incosistency - removed different
logic for "not simple types" since it biase

[CostModel][AMDGPU] Fix instructions costs estimation for vector types.

1. Fixed vector instructions costs estimations incosistency - removed different
logic for "not simple types" since it biases costs for these types.
2. Fixed legalization penalty for vectors too big for the target: changed from
overwrite default legalization cost value estimation to added penalty.
3. Fixed few typos in tests.

Reviewed By: rampitec

Differential Revision: https://reviews.llvm.org/D114893

show more ...

[AMDGPU] Minor cleanup after D109483. NFC.

[NFC][AMDGPU] Reduce includes dependencies.

1. Splitted out some parts of R600 target to separate modules/headers.
2. Reduced some include lists in headers.
3. Found and fixed issue with override `G

[NFC][AMDGPU] Reduce includes dependencies.

1. Splitted out some parts of R600 target to separate modules/headers.
2. Reduced some include lists in headers.
3. Found and fixed issue with override `GCNTargetMachine::getSubtargetImpl()`
and `R600TargetMachine::getSubtargetImpl()` had different return value type
than base class.
4. Minor forward declarations cleanup.

Reviewed By: foad

Differential Revision: https://reviews.llvm.org/D108596

show more ...

[BasicTTIImpl][LoopUnroll] getUnrollingPreferences(): emit ORE remark when advising against unrolling due to a call in a loop

I'm not sure this is the best way to approach this,
but the situation is

[BasicTTIImpl][LoopUnroll] getUnrollingPreferences(): emit ORE remark when advising against unrolling due to a call in a loop

I'm not sure this is the best way to approach this,
but the situation is rather not very detectable unless we explicitly call it out when refusing to advise to unroll.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D107271

show more ...

[Analysis] Add simple cost model for strict (in-order) reductions

I have added a new FastMathFlags parameter to getArithmeticReductionCost
to indicate what type of reduction we are performing:

1.

[Analysis] Add simple cost model for strict (in-order) reductions

I have added a new FastMathFlags parameter to getArithmeticReductionCost
to indicate what type of reduction we are performing:

1. Tree-wise. This is the typical fast-math reduction that involves
continually splitting a vector up into halves and adding each
half together until we get a scalar result. This is the default
behaviour for integers, whereas for floating point we only do this
if reassociation is allowed.
2. Ordered. This now allows us to estimate the cost of performing
a strict vector reduction by treating it as a series of scalar
operations in lane order. This is the case when FP reassociation
is not permitted. For scalable vectors this is more difficult
because at compile time we do not know how many lanes there are,
and so we use the worst case maximum vscale value.

I have also fixed getTypeBasedIntrinsicInstrCost to pass in the
FastMathFlags, which meant fixing up some X86 tests where we always
assumed the vector.reduce.fadd/mul intrinsics were 'fast'.

New tests have been added here:

Analysis/CostModel/AArch64/reduce-fadd.ll
Analysis/CostModel/AArch64/sve-intrinsics.ll
Transforms/LoopVectorize/AArch64/strict-fadd-cost.ll
Transforms/LoopVectorize/AArch64/sve-strict-fadd-cost.ll

Differential Revision: https://reviews.llvm.org/D105432

show more ...

[TTI] Remove IsPairwiseForm from getArithmeticReductionCost

This patch removes the IsPairwiseForm flag from the Reduction Cost TTI
hooks, along with some accompanying code for pattern matching reduc

[TTI] Remove IsPairwiseForm from getArithmeticReductionCost

This patch removes the IsPairwiseForm flag from the Reduction Cost TTI
hooks, along with some accompanying code for pattern matching reductions
from trees starting at extract elements. IsPairWise is now assumed to be
false, which was the predominant way that the value was used from both
the Loop and SLP vectorizers. Since the adjustments such as D93860, the
SLP vectorizer has not relied upon this distinction between paiwise and
non-pairwise reductions.

This also removes some code that was detecting reductions trees starting
from extract elements inside the costmodel. This case was
double-counting costs though, adding the individual costs on the
individual instruction _and_ the total cost of the reduction. Removing
it changes the costs in llvm/test/Analysis/CostModel/X86/reduction.ll to
not double count. The cost of reduction intrinsics is still tested
through the various tests in
llvm/test/Analysis/CostModel/X86/reduce-xyz.ll.

Differential Revision: https://reviews.llvm.org/D105484

show more ...

[AMDGPU] PHI node cost should not be counted for the size and latency.

Details: https://reviews.llvm.org/D96805 changed the GCNTTIImpl::getCFInstrCost to return 1 for the PHI nodes
for the TTI::

[AMDGPU] PHI node cost should not be counted for the size and latency.

Details: https://reviews.llvm.org/D96805 changed the GCNTTIImpl::getCFInstrCost to return 1 for the PHI nodes
for the TTI::TCK_CodeSize and TTI::TCK_SizeAndLatency. This is incorrect because the value moves that are the
result of the PHI lowering are inserted into the basic block predecessors - not into the block itself.
As a result of this change LoopRotate and LoopUnroll were broken because of the incorrect Loop header and loop
body size/cost estimation.

Reviewed By: rampitec

Differential Revision: https://reviews.llvm.org/D105104

show more ...

[CodeGen] Add missing includes (NFC)

These currently rely on the IRBuilder.h include in TargetLowering.h.
Make them explicit.

[TTI] NFC: Change getTypeLegalizationCost to return InstructionCost.

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https

[TTI] NFC: Change getTypeLegalizationCost to return InstructionCost.

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https://reviews.llvm.org/D91174
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2020-November/146408.html

Reviewed By: sdesmalen, kparzysz

Differential Revision: https://reviews.llvm.org/D101533

show more ...

[COST] Improve shuffle kind detection if shuffle mask is provided.

Added an extra analysis for better choosing of shuffle kind in
getShuffleCost functions for better cost estimation if mask was
prov

[COST] Improve shuffle kind detection if shuffle mask is provided.

Added an extra analysis for better choosing of shuffle kind in
getShuffleCost functions for better cost estimation if mask was
provided.

Differential Revision: https://reviews.llvm.org/D100865

show more ...

Revert "[COST] Improve shuffle kind detection if shuffle mask is provided."

This reverts commit 92399322217917e67c0d72a55ec51ddc82251cf6 to fix
a compiler crash on mask checks.

[COST] Improve shuffle kind detection if shuffle mask is provided.

Added an extra analysis for better choosing of shuffle kind in
getShuffleCost functions for better cost estimation if mask was
prov

[COST] Improve shuffle kind detection if shuffle mask is provided.

Added an extra analysis for better choosing of shuffle kind in
getShuffleCost functions for better cost estimation if mask was
provided.

Differential Revision: https://reviews.llvm.org/D100865

show more ...

[TTI] NFC: Use InstructionCost to store ScalarizationCost in IntrinsicCostAttributes.

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction o

[TTI] NFC: Use InstructionCost to store ScalarizationCost in IntrinsicCostAttributes.

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https://reviews.llvm.org/D91174
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2020-November/146408.html

Reviewed By: samparker

Differential Revision: https://reviews.llvm.org/D101151

show more ...

[TTI] NFC: Change getArithmeticInstrCost to return InstructionCost

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https:/

[TTI] NFC: Change getArithmeticInstrCost to return InstructionCost

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https://reviews.llvm.org/D91174
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2020-November/146408.html

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D100317

show more ...

[TTI] NFC: Change getVectorInstrCost to return InstructionCost

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https://rev

[TTI] NFC: Change getVectorInstrCost to return InstructionCost

This patch migrates the TTI cost interfaces to return an InstructionCost.

See this patch for the introduction of the type: https://reviews.llvm.org/D91174
See this thread for context: http://lists.llvm.org/pipermail/llvm-dev/2020-November/146408.html

Reviewed By: dmgreen

Differential Revision: https://reviews.llvm.org/D100315

show more ...