GlobalISel: Allow forming atomic/volatile G_ZEXTLOAD

SelectionDAG has a target hook, getExtendForAtomicOps, which it uses
in the computeKnownBits implementation for ATOMIC_LOAD. This is pretty
ugly

GlobalISel: Allow forming atomic/volatile G_ZEXTLOAD

SelectionDAG has a target hook, getExtendForAtomicOps, which it uses
in the computeKnownBits implementation for ATOMIC_LOAD. This is pretty
ugly (as is having a separate load opcode for atomics), so instead
allow making use of atomic zextload. Enable this for AArch64 since the
DAG path defaults in to the zext behavior.

The tablegen changes are pretty ugly, but partially helps migrate
SelectionDAG from using ISD::ATOMIC_LOAD to regular ISD::LOAD with
atomic memory operands. For now the DAG emitter will emit matchers for
patterns which the DAG will not produce.

I'm still a bit confused by the intent of the isLoad/isStore/isAtomic
bits. The DAG implementation rejects trying to use any of these in
combination. For now I've opted to make the isLoad checks also check
isAtomic, although I think having isLoad and isAtomic set on these
makes most sense.

show more ...

[GlobalISel][SelectionDAG] Implement the HasNoUse builtin predicate

This change introduces the HasNoUse builtin predicate in PatFrags that
checks for the absence of use of the first result operand.

[GlobalISel][SelectionDAG] Implement the HasNoUse builtin predicate

This change introduces the HasNoUse builtin predicate in PatFrags that
checks for the absence of use of the first result operand.
GlobalISelEmitter will allow source PatFrags with this predicate to be
matched with destination instructions with empty outs. This predicate is
required for selecting the no-return variant of atomic instructions in
AMDGPU.

Differential Revision: https://reviews.llvm.org/D125212

show more ...

[TableGen] Rewrite type set intersection in type inference

The previous code had a bug when dealing with matching iPTR against a
set of integer types. It was trying to handle it all in a compact way

[TableGen] Rewrite type set intersection in type inference

The previous code had a bug when dealing with matching iPTR against a
set of integer types. It was trying to handle it all in a compact way,
but that implementation couldn't be modified to correct the problem in
a simple way. The code wasn't long, and it was easier to rewrite it.

The actual issue was that non-scalar-integer types were considered when
matching against iPTR. For example {iPTR} intersected with {i32 f32}
was {iPTR} (due to multiple types in the other set), but should be just
{i32}, because i32 is the only integer scalar in the other set.

show more ...

[TableGen] Move printing to stream directly to MachineValueTypeSet

[TableGen] Remove the use of global Record state

This commits removes TableGens reliance on managed static global record state
by moving the RecordContext into the RecordKeeper. The RecordKeeper is

[TableGen] Remove the use of global Record state

This commits removes TableGens reliance on managed static global record state
by moving the RecordContext into the RecordKeeper. The RecordKeeper is now
treated similarly to a (LLVM|MLIR|etc)Context object and is passed to static
construction functions. This is an important step forward in removing TableGens
reliance on global state, and in a followup will allow for users that parse tablegen
to parse multiple tablegen files without worrying about Record lifetime.

Differential Revision: https://reviews.llvm.org/D125276

show more ...

Cleanup include: TableGen

This also includes a few cleanup from Support.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.l

Cleanup include: TableGen

This also includes a few cleanup from Support.

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121331

show more ...

[TableGen][RISCV] Relax a restriction in generating patterns for commutable SDNodes.

Previously, all children would be checked to see if any were an
explicit Register. If anywhere no commutable patt

[TableGen][RISCV] Relax a restriction in generating patterns for commutable SDNodes.

Previously, all children would be checked to see if any were an
explicit Register. If anywhere no commutable patterns would be
generated. This patch loosens the restriction to only check the
children that are being commuted.

Digging back through history, this code predates the existence of
commutable intrinsics and commutable SDNodes with more than 2
operands. At that time the loop would count the number of children that
weren't registers and if that was equal to 2 it would allow commuting.
I don't think this loop was re-considered when commutable
intrinsics were added or when we allowed SDNodes with more than 2
operands.

This important for RISCV were our isel patterns have a V0 mask
operand after the commutable operands on some RISCVISD opcodes.

Reviewed By: arsenm

Differential Revision: https://reviews.llvm.org/D117955

show more ...

utils: Remove some no-op raw_string_ostream flush calls, NFC

Since 65b13610a5226b84889b923bae884ba395ad084d, raw_string_ostream has
been unbuffered by default. Based on an audit of llvm/utils/, this

utils: Remove some no-op raw_string_ostream flush calls, NFC

Since 65b13610a5226b84889b923bae884ba395ad084d, raw_string_ostream has
been unbuffered by default. Based on an audit of llvm/utils/, this
commit removes every call to `raw_string_ostream::flush()` and any call
to `raw_string_ostream::str()` whose result is ignored or that doesn't
help with clarity.

I left behind a few calls to `str()`. In these cases, the underlying
std::string was declared pretty far away and never used again, whereas
stream recently had its last write. The code is easier to read as-is;
the no-op call to `flush()` inside `str()` isn't harmful, and when
https://reviews.llvm.org/D115421 lands it'll be gone anyway.

show more ...

[TableGen][SelectionDAG] Use ComplexPattern type for non-leaf nodes

When used as a non-leaf node, TableGen does not currently use the type
of a ComplexPattern for type inference, which also means it

[TableGen][SelectionDAG] Use ComplexPattern type for non-leaf nodes

When used as a non-leaf node, TableGen does not currently use the type
of a ComplexPattern for type inference, which also means it does not
check it doesn't conflict with the use. This differs from when used as a
leaf value, where the type is used for inference. This addresses that
discrepancy. The test case is not representative of most real-world uses
but is sufficient to demonstrate inference is working.

Some of these uses also make use of ValueTypeByHwMode rather than
SimpleValueType and so the existing type inference is extended to
support that alongside the new type inference.

There are also currently various cases of using ComplexPatterns with an
untyped type, but only for non-leaf nodes. For compatibility this is
permitted, and uses the old behaviour of not inferring for non-leaf
nodes, but the existing logic is still used for leaf values. This
remaining discrepancy should eventually be eliminated, either by
removing all such uses of untyped so the special case goes away (I
imagine Any, or a more specific type in certain cases, would be
perfectly sufficient), or by copying it to the leaf value case so
they're consistent with one another if this is something that does need
to keep being supported.

All non-experimental targets have been verified to produce bit-for-bit
identical TableGen output with this change applied.

Reviewed By: kparzysz

Differential Revision: https://reviews.llvm.org/D109035

show more ...

[TableGen] Replace static_cast with llvm's cast. NFC

These all appear next to an isa<> and cast<> is much more
common in these cases.

[TableGen] Fix both sides of '&&' are same

The operand of the second any_of in EnforceSmallerThan should be
B not S like the FP code in the if below.

Unfortunately, fixing that causes an infinite l

[TableGen] Fix both sides of '&&' are same

The operand of the second any_of in EnforceSmallerThan should be
B not S like the FP code in the if below.

Unfortunately, fixing that causes an infinite loop in the build
of RISCV. So I've added a workaround for that as well.

Fixes PR44768.

Reviewed By: RKSimon

Differential Revision: https://reviews.llvm.org/D111502

show more ...

Fixed warnings in LLVM produced by -Wbitwise-instead-of-logical

Convert TableGen assert to error

This gives a nice message about the location of errors in a large
tablegen file, which is much more useful for users

Differential Revision: https://reviews.llvm.org

Convert TableGen assert to error

This gives a nice message about the location of errors in a large
tablegen file, which is much more useful for users

Differential Revision: https://reviews.llvm.org/D102740

show more ...

[AArch64][RISCV] Make sure isel correctly honors failure orderings.

If a cmpxchg specifies acquire or seq_cst on failure, make sure we
generate code consistent with that ordering even if the success

[AArch64][RISCV] Make sure isel correctly honors failure orderings.

If a cmpxchg specifies acquire or seq_cst on failure, make sure we
generate code consistent with that ordering even if the success ordering
is not acquire/seq_cst.

At one point, it was ambiguous whether this sort of construct was valid,
but the C++ standad and LLVM now accept arbitrary combinations of
success/failure orderings.

This doesn't address the corresponding issue in AtomicExpand. (This was
reported as https://bugs.llvm.org/show_bug.cgi?id=33332 .)

Fixes https://bugs.llvm.org/show_bug.cgi?id=50512.

Differential Revision: https://reviews.llvm.org/D103284

show more ...

Revert rG632668c1c0e7dcf97154d2e377491cdc8cb6963c : "[TableGen] TreePatternNode::isIsomorphicTo - early out for matching leafs. NFCI."

Revert premature (and very broken....) experimental commit.

[TableGen] TreePatternNode::isIsomorphicTo - early out for matching leafs. NFCI.

If the leafs are the same then no need to perform DefInit matching.

[ValueTypes] Rename MVT::getVectorNumElements() to MVT::getVectorMinNumElements(). Fix some misuses of getVectorNumElements()

getVectorNumElements() returns a value for scalable vectors
without any

[ValueTypes] Rename MVT::getVectorNumElements() to MVT::getVectorMinNumElements(). Fix some misuses of getVectorNumElements()

getVectorNumElements() returns a value for scalable vectors
without any warning so it is effectively getVectorMinNumElements().
By renaming it and making getVectorNumElements() forward to
it, we can insert a check for scalable vectors into getVectorNumElements()
similar to EVT. I didn't do that in this patch because there are still more
fixes needed, but I was able to temporarily do it and passed the RISCV
lit tests with these changes.

The changes to isPow2VectorType and getPow2VectorType are copied from EVT.

The change to TypeInfer::EnforceSameNumElts reduces the size of AArch64's isel table.
We're now considering SameNumElts to require the scalable property to match which
removes some unneeded type checks.

This was motivated by the bug I fixed yesterday in 80b9510806cf11c57f2dd87191d3989fc45defa8

Reviewed By: frasercrmck, sdesmalen

Differential Revision: https://reviews.llvm.org/D102262

show more ...

[TableGen] Remove predicate filtering from GenerateVariants.

After D100691, predicates should be cheap to compare again so
we don't need to filter anymore.

This is mostly just a revert of several p

[TableGen] Remove predicate filtering from GenerateVariants.

After D100691, predicates should be cheap to compare again so
we don't need to filter anymore.

This is mostly just a revert of several patches going back to 2018.

Reviewed By: kparzysz

Differential Revision: https://reviews.llvm.org/D100695

show more ...

[TableGen] Store predicates in PatternToMatch as ListInit *. Add string for HwModeFeatures

This uses to be how predicates were handled prior to HwMode being
added. When the Predicates were converted

[TableGen] Store predicates in PatternToMatch as ListInit *. Add string for HwModeFeatures

This uses to be how predicates were handled prior to HwMode being
added. When the Predicates were converted to a std::vector it
significantly increased the cost of a compare in GenerateVariants.
Since ListInit's are uniquified by tablegen, we can use a simple
pointer comparison to check for identical lists.

In order to store the HwMode, we now add a separate string to
PatternToMatch. This will be appended separately to the predicate
string in getPredicateCheck. A new getPredicateRecords is added
to allow GlobalISel and getPredicateCheck to both get the sorted
list of Records. GlobalISel was ignoring any HwMode predicates
before and still is.

There is one slight change here, ListInits with different predicate
orders aren't sorted so the filtering in GenerateVariants might
fail to detect two isomorphic patterns with different predicate
orders. This doesn't seem to be happening in tree today.

My hope is this will allow us to remove all the BitVector tracking
in GenerateVariants that was making up for predicates beeing
expensive to compare. There's a decent amount of heap allocations
there on large targets like X86, AMDGPU, and RISCV.

Differential Revision: https://reviews.llvm.org/D100691

show more ...

[TableGen] Add predicate checks to isel patterns for default HwMode.

As discussed in D100691 and based on D100889.

I removed the ModeChecks cache which provides little value. Reduced
from three loo

[TableGen] Add predicate checks to isel patterns for default HwMode.

As discussed in D100691 and based on D100889.

I removed the ModeChecks cache which provides little value. Reduced
from three loops to two. Used ArrayRef to pass the Predicate to
AppendPattern to avoid needing to construct a vector for single
mode. Used SmallVector to avoid heap allocation constructing
DefaultCheck for the in tree targets the use it.

Reviewed By: kparzysz

Differential Revision: https://reviews.llvm.org/D101240

show more ...

[TableGen] Pass SmallVector to union_modes instead of returning a std::vector.

The number of modes is small so this should avoid a heap allocation.

Also replace std::set with SmallSet.

[TableGen] Use MachineValueTypeSet in place of SmallSet.

MachineValueTypeSet is effectively a std::bitset<256>. This allows
us quickly insert into the set and check if a type is in the set.

[TableGen] Remove local SmallSet from TypeSetByHwMode::insert.

This keeps track of which modes are in VVT so we can find out
if a mode is missing later. But we can just ask VVT whether it
has a part

[TableGen] Remove local SmallSet from TypeSetByHwMode::insert.

This keeps track of which modes are in VVT so we can find out
if a mode is missing later. But we can just ask VVT whether it
has a particular mode.

show more ...

[TableGen] Replace two SmallDenseSets with SmallSets.

The key here is HwMode indices. They're going to be small numbers,
contiguous, and only a few different values. I don't think we need
to go thro

[TableGen] Replace two SmallDenseSets with SmallSets.

The key here is HwMode indices. They're going to be small numbers,
contiguous, and only a few different values. I don't think we need
to go through the SmallDenseSet hashing.

A BitVector would be even better, but we don't have the upper
bound here.

show more ...

[TableGen] Run GenerateVariants before ExpandHwModeBasedTypes.

A large portion of the patterns are duplicated for HwMode on RISCV.
If we expand HwMode first, we need to check nearly twice as many
pa

[TableGen] Run GenerateVariants before ExpandHwModeBasedTypes.

A large portion of the patterns are duplicated for HwMode on RISCV.
If we expand HwMode first, we need to check nearly twice as many
patterns for variants. HwModes shouldn't affect whether a variant
is valid so we should be able to expand after.

This also reduces the RISCV isel table by 539 bytes due to factoring
working better on this pattern order. Unfortunately it increases
Hexagon table size by ~50 bytes. But I think this is a reasonable
trade.

show more ...