[RegAllocFast] Handle new debug values for spills

These new debug values get inserted after the place where the spill
happens, which means they won't be reached by the reverse traversal of
basic blo

[RegAllocFast] Handle new debug values for spills

These new debug values get inserted after the place where the spill
happens, which means they won't be reached by the reverse traversal of
basic block instructions. This would crash or fail assertions if they
contained any virtual registers to be replaced. We can manually handle
the new debug values right away to resolve this.

Fixes https://github.com/llvm/llvm-project/issues/59172

Reviewed By: StephenTozer

Differential Revision: https://reviews.llvm.org/D139590

(cherry picked from commit 87f57f459e7acbb00a6ca4ee6dec6014c5a97e07)

show more ...

[CodeGen] Qualify auto variables in for loops (NFC)

[CodeGen] Use RegClassFilterFunc where appropriate (NFC)

Revert "[fastalloc] Support allocating specific register class in fastalloc"

This reverts commit 719658d078c4093d1ee716fb65ae94673df7b22b.
Breaks a few things, see comments on https://reviews.llvm.o

Revert "[fastalloc] Support allocating specific register class in fastalloc"

This reverts commit 719658d078c4093d1ee716fb65ae94673df7b22b.
Breaks a few things, see comments on https://reviews.llvm.org/D128437
There's disagreement about the best fix.
So let's keep HEAD green while discussions are happening.

show more ...

[fastalloc] Support allocating specific register class in fastalloc

The base RA support infrastructure that only allow a specific register
class be allocated in RA pss. Since greedy RA, basic RA der

[fastalloc] Support allocating specific register class in fastalloc

The base RA support infrastructure that only allow a specific register
class be allocated in RA pss. Since greedy RA, basic RA derived from
base RA, they all allow allocating specific register class. Fast RA
doesn't support allocating register for specific register class. This
patch is to enable ShouldAllocateClass in fast RA, so that it can
support allocating register for specific register class.

Differential Revision: https://reviews.llvm.org/D126771

show more ...

[fastregalloc] Enhance the heuristics for liveout in self loop.

For below case, virtual register is defined twice in the self loop. We
don't need to spill %0 after the third instruction `%0 = def (t

[fastregalloc] Enhance the heuristics for liveout in self loop.

For below case, virtual register is defined twice in the self loop. We
don't need to spill %0 after the third instruction `%0 = def (tied %0)`,
because it is defined in the second instruction `%0 = def`.

1 bb.1
2 %0 = def
3 %0 = def (tied %0)
4 ...
5 jmp bb.1

Reviewed By: MatzeB

Differential Revision: https://reviews.llvm.org/D125079

show more ...

[fastregalloc] Fix bug when undef value is tied to def.

If the tied use is undef value, fastregalloc should free the def
register. There is no reload needed for the undef value.

Reviewed By: MatzeB

[fastregalloc] Fix bug when undef value is tied to def.

If the tied use is undef value, fastregalloc should free the def
register. There is no reload needed for the undef value.

Reviewed By: MatzeB

Differential Revision: https://reviews.llvm.org/D124834

show more ...

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-in

Cleanup codegen includes

This is a (fixed) recommit of https://reviews.llvm.org/D121169

after: 1061034926
before: 1063332844

Discourse thread: https://discourse.llvm.org/t/include-what-you-use-include-cleanup
Differential Revision: https://reviews.llvm.org/D121681

show more ...

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https:/

Revert "Cleanup codegen includes"

This reverts commit 7f230feeeac8a67b335f52bd2e900a05c6098f20.
Breaks CodeGenCUDA/link-device-bitcode.cu in check-clang,
and many LLVM tests, see comments on https://reviews.llvm.org/D121169

show more ...

Cleanup codegen includes

after: 1061034926
before: 1063332844

Differential Revision: https://reviews.llvm.org/D121169

[llvm] Use range-based for loops (NFC)

[RegAllocFast] Fix nondeterminism in debuginfo generation

Changes from commit 1db137b1859692ae33228c530d4df9f2431b2151
added iteration over hash map that can result in non-deterministic
order. Fix t

[RegAllocFast] Fix nondeterminism in debuginfo generation

Changes from commit 1db137b1859692ae33228c530d4df9f2431b2151
added iteration over hash map that can result in non-deterministic
order. Fix that by using a SmallMapVector to preserve the order.

Differential Revision: https://reviews.llvm.org/D113468

show more ...

RegAlloc: Allow targets to split register allocation

AMDGPU normally spills SGPRs to VGPRs. Previously, since all register
classes are handled at the same time, this was problematic. We don't
know a

RegAlloc: Allow targets to split register allocation

AMDGPU normally spills SGPRs to VGPRs. Previously, since all register
classes are handled at the same time, this was problematic. We don't
know ahead of time how many registers will be needed to be reserved to
handle the spilling. If no VGPRs were left for spilling, we would have
to try to spill to memory. If the spilled SGPRs were required for exec
mask manipulation, it is highly problematic because the lanes active
at the point of spill are not necessarily the same as at the restore
point.

Avoid this problem by fully allocating SGPRs in a separate regalloc
run from VGPRs. This way we know the exact number of VGPRs needed, and
can reserve them for a second run. This fixes the most serious
issues, but it is still possible using inline asm to make all VGPRs
unavailable. Start erroring in the case where we ever would require
memory for an SGPR spill.

This is implemented by giving each regalloc pass a callback which
reports if a register class should be handled or not. A few passes
need some small changes to deal with leftover virtual registers.

In the AMDGPU implementation, a new pass is introduced to take the
place of PrologEpilogInserter for SGPR spills emitted during the first
run.

One disadvantage of this is currently StackSlotColoring is no longer
used for SGPR spills. It would need to be run again, which will
require more work.

Error if the standard -regalloc option is used. Introduce new separate
-sgpr-regalloc and -vgpr-regalloc flags, so the two runs can be
controlled individually. PBQB is not currently supported, so this also
prevents using the unhandled allocator.

show more ...

MachineBasicBlock: add liveout iterator aware of which liveins are defined by the runtime.

Using this in RegAlloc fast reduces register pressure, and in some cases allows
x86 code to compile that wo

MachineBasicBlock: add liveout iterator aware of which liveins are defined by the runtime.

Using this in RegAlloc fast reduces register pressure, and in some cases allows
x86 code to compile that wouldn't before.

show more ...

[RegAllocFast] properly handle STATEPOINT instruction.

STATEPOINT is a fancy and complex pseudo instruction which
has both tied defs and regmask operand.

Basic FastRA algorithm is as follows:

1. M

[RegAllocFast] properly handle STATEPOINT instruction.

STATEPOINT is a fancy and complex pseudo instruction which
has both tied defs and regmask operand.

Basic FastRA algorithm is as follows:

1. Mark registers used by defs as free
2. If instruction has regmask operand displace clobbered registers
according to regmask.
3. Assign registers for use operands.

In case of tied defs step 1 is replaced with allocation of registers
for them. But regmask is still processed, which may displace already
allocated registers. As a result, tied use and def will get assigned
to different registers.

This patch makes FastRA to process instruction's RegMask (if any) when
checking for physical registers interference.
That way tied operands won't get registers clobbered by regmask.

Reviewed By: arsenm, skatkov
Differential Revision: https://reviews.llvm.org/D99284

show more ...

Revert "RegAlloc: do not consider liveins to EH-pad successors as liveout."

Some liveins *can* come from this block (e.g. any SSA value except the call),
it's only the ones that produce `landingpad`

Revert "RegAlloc: do not consider liveins to EH-pad successors as liveout."

Some liveins *can* come from this block (e.g. any SSA value except the call),
it's only the ones that produce `landingpad` values that can't and I didn't
think it through properly.

show more ...

RegAlloc: do not consider liveins to EH-pad successors as liveout.

These registers get defined by the runtime, not the block being allocated, and
treating them as preassigned in RegAllocFast adds ex

RegAlloc: do not consider liveins to EH-pad successors as liveout.

These registers get defined by the runtime, not the block being allocated, and
treating them as preassigned in RegAllocFast adds extra pressure, sometimes
enough to make the function unallocatable.

show more ...

[DebugInfo] Handle DBG_VALUES with multiple variable location operands in MIR

This patch adds handling for DBG_VALUE_LIST in the MIR-passes (after
finalize-isel), excluding the debug liveness passes

[DebugInfo] Handle DBG_VALUES with multiple variable location operands in MIR

This patch adds handling for DBG_VALUE_LIST in the MIR-passes (after
finalize-isel), excluding the debug liveness passes and DWARF emission. This
most significantly affects MachineSink, which now needs to consider all used
registers of a debug value when sinking, but for most passes this change is
simply replacing getDebugOperand(0) with an iteration over all debug operands.

Differential Revision: https://reviews.llvm.org/D92578

show more ...

Reapply "[DebugInfo] Add new instruction and DIExpression operator for variadic debug values"

Rewrites test to use correct architecture triple; fixes incorrect
reference in SourceLevelDebugging doc

Reapply "[DebugInfo] Add new instruction and DIExpression operator for variadic debug values"

Rewrites test to use correct architecture triple; fixes incorrect
reference in SourceLevelDebugging doc; simplifies `spillReg` behaviour
so as to not be dependent on changes elsewhere in the patch stack.

This reverts commit d2000b45d033c06dc7973f59909a0ad12887ff51.

show more ...

Revert "[DebugInfo] Add new instruction and DIExpression operator for variadic debug values"

This reverts commit d07f106f4a48b6e941266525b6f7177834d7b74e.

[DebugInfo] Add new instruction and DIExpression operator for variadic debug values

This patch adds a new instruction that can represent variadic debug values,
DBG_VALUE_VAR. This patch alone covers

[DebugInfo] Add new instruction and DIExpression operator for variadic debug values

This patch adds a new instruction that can represent variadic debug values,
DBG_VALUE_VAR. This patch alone covers the addition of the instruction and a set
of basic code changes in MachineInstr and a few adjacent areas, but does not
correctly handle variadic debug values outside of these areas, nor does it
generate them at any point.

The new instruction is similar to the existing DBG_VALUE instruction, with the
following differences: the operands are in a different order, any number of
values may be used in the instruction following the Variable and Expression
operands (these are referred to in code as “debug operands”) and are indexed
from 0 so that getDebugOperand(X) == getOperand(X+2), and the Expression in a
DBG_VALUE_VAR must use the DW_OP_LLVM_arg operator to pass arguments into the
expression.

The new DW_OP_LLVM_arg operator is only valid in expressions appearing in a
DBG_VALUE_VAR; it takes a single argument and pushes the debug operand at the
index given by the argument onto the Expression stack. For example the
sub-expression `DW_OP_LLVM_arg, 0` has the meaning “Push the debug operand at
index 0 onto the expression stack.”

Differential Revision: https://reviews.llvm.org/D82363

show more ...

[CodeGen] Use range-based for loops (NFC)

[CodeGen, Transforms] Use llvm::sort (NFC)

[FastRA] Fix handling of bundled MIs

Fast register allocator skips bundled MIs, as the main assignment
loop uses MachineBasicBlock::iterator (= MachineInstrBundleIterator)
This was causing SIInsertW

[FastRA] Fix handling of bundled MIs

Fast register allocator skips bundled MIs, as the main assignment
loop uses MachineBasicBlock::iterator (= MachineInstrBundleIterator)
This was causing SIInsertWaitcnts to crash which expects all
instructions to have registers assigned.

This patch makes sure to set everything inside bundle to the same
assignments done on BUNDLE header.

Reviewed By: qcolombet

Differential Revision: https://reviews.llvm.org/D90369

show more ...

[NFC][MC] TargetRegisterInfo::getSubReg is a MCRegister.

Typing the API appropriately.

Differential Revision: https://reviews.llvm.org/D92341