[MachineScheduler] Order more stores by ascending address

According D125377, we order STP Q's by ascending address. While on some
targets, paired 128 bit loads and stores are slow, so the STP will s

[MachineScheduler] Order more stores by ascending address

According D125377, we order STP Q's by ascending address. While on some
targets, paired 128 bit loads and stores are slow, so the STP will split
into STRQ and STUR, so I hope these stores will also be ordered.
Also add subtarget feature ascend-store-address to control the aggressive order.

Reviewed By: dmgreen, fhahn

Differential Revision: https://reviews.llvm.org/D126700

show more ...

[Target] use getSubtarget<> instead of static_cast<>(getSubtarget())

Differential Revision: https://reviews.llvm.org/D125391

[AArch64] Async unwind - Adjust unwind info in AArch64LoadStoreOptimizer

[Re-commit after fixing a dereference of "end" iterator]

The AArch64LoadStoreOptimnizer pass may merge a register
increment/

[AArch64] Async unwind - Adjust unwind info in AArch64LoadStoreOptimizer

[Re-commit after fixing a dereference of "end" iterator]

The AArch64LoadStoreOptimnizer pass may merge a register
increment/decrement with a following memory operation. In doing so, it
may break CFI by moving a stack pointer adjustment past the CFI
instruction that described *that* adjustment.

This patch fixes this issue by moving said CFI instruction after the
merged instruction, where the SP increment/decrement actually takes
place.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D114547

show more ...

Revert "[AArch64] Async unwind - Adjust unwind info in AArch64LoadStoreOptimizer"

This reverts commit ecbf32dd88fc91b4fe709dc14bb3493dda6e8854.

It's possible this patch is the reason for an asertio

Revert "[AArch64] Async unwind - Adjust unwind info in AArch64LoadStoreOptimizer"

This reverts commit ecbf32dd88fc91b4fe709dc14bb3493dda6e8854.

It's possible this patch is the reason for an asertion failure
`!NodePtr->isKnownSentinel()` in `AArch64LoadStoreOpt::mergeUpdateInsn`
(https://lab.llvm.org/buildbot/#/builders/185/builds/1555) reverting while I
investigate.

show more ...

[AArch64] Async unwind - Adjust unwind info in AArch64LoadStoreOptimizer

The AArch64LoadStoreOptimnizer pass may merge a register
increment/decrement with a following memory operation. In doing so,

[AArch64] Async unwind - Adjust unwind info in AArch64LoadStoreOptimizer

The AArch64LoadStoreOptimnizer pass may merge a register
increment/decrement with a following memory operation. In doing so, it
may break CFI by moving a stack pointer adjustment past the CFI
instruction that described *that* adjustment.

This patch fixes this issue by moving said CFI instruction after the
merged instruction, where the SP increment/decrement actually takes
place.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D114547

show more ...

[AArch64] Pass Reg instead of MI to tryToFindRenameRegister (NFC).

FirstMI is only used to get the load/store operand and the machine
function. Pass the MF and register explicitly, so the helper can

[AArch64] Pass Reg instead of MI to tryToFindRenameRegister (NFC).

FirstMI is only used to get the load/store operand and the machine
function. Pass the MF and register explicitly, so the helper can be used
to find rename registers for other instructions in the future.

show more ...

[AArch64] Remove unused argument from tryToFindRegisterToRename (NFC).

The MI argument is not used by the function. Remove it.

[AArch64][LoadStoreOptimizer] Ignore undef registers when checking rename register used between paired instructions.

The content of undef registers are not used in meaningful ways, when checking
if

[AArch64][LoadStoreOptimizer] Ignore undef registers when checking rename register used between paired instructions.

The content of undef registers are not used in meaningful ways, when checking
if a rename register is used between paired instructions we should ignore
undef registers.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D119305

show more ...

[NFC] Use Register instead of unsigned

AArch64: don't form indexed paired ops if base reg overlaps operands.

The registers involved might not be identical, but can still overlap (e.g.
"str w0, [x0, #4]!").

Revert "[AArch64LoadStoreOptimizer] Recommit: Generate more STPs by renaming registers earlier"

This reverts commit ea011ec5ed53599305de62ca5fcfd31f4b3448c3.

This still causes some miscompiles, I'l

Revert "[AArch64LoadStoreOptimizer] Recommit: Generate more STPs by renaming registers earlier"

This reverts commit ea011ec5ed53599305de62ca5fcfd31f4b3448c3.

This still causes some miscompiles, I'll follow up in the phabricator
review with a sample of that issue (which is part of the sample of
the previous issue).

show more ...

[AArch64LoadStoreOptimizer] Recommit: Generate more STPs by renaming registers earlier

This is a recommit that fixes unwanted STP generation by checking that
the base register has not been modified

[AArch64LoadStoreOptimizer] Recommit: Generate more STPs by renaming registers earlier

This is a recommit that fixes unwanted STP generation by checking that
the base register has not been modified or used elsewhere.

Our initial motivating case was memcpy's with alignments > 16. The
loads/stores, to which small memcpy's expand, are kept together in
several places so that we get a sequence like this for a 64 bit copy:
LD w0
LD w1
ST w0
ST w1
The load/store optimiser can generate a LDP/STP w0, w1 from this because
the registers read/written are consecutive. In our case however, the
sequence is optimised during ISel, resulting in:
LD w0
ST w0
LD w0
ST w0
This instruction reordering allows reuse of registers. Since the registers
are no longer consecutive (i.e. they are the same), it inhibits LDP/STP
creation. The approach here is to perform renaming:
LD w0
ST w0
LD w1
ST w1
to enable the folding of the stores into a STP. We do not yet generate
the LDP due to a limitation in the renaming implementation, but plan to
look at that in a follow-up so that we fully support this case. While
this was initially motivated by certain memcpy's, this is a general
approach and thus is beneficial for other cases too, as can be seen
in some test changes.

Differential Revision: https://reviews.llvm.org/D103597

show more ...

Revert "[AArch64LoadStoreOptimizer] Generate more STPs by renaming registers earlier"

This reverts commit d96ea46629803641038ebe46d8cd512f8cf7e20f, as it
caused various misoptimizations, see https:/

Revert "[AArch64LoadStoreOptimizer] Generate more STPs by renaming registers earlier"

This reverts commit d96ea46629803641038ebe46d8cd512f8cf7e20f, as it
caused various misoptimizations, see https://reviews.llvm.org/D103597
for discussion on the issues.

show more ...

[AArch64LoadStoreOptimizer] Generate more STPs by renaming registers earlier

Our initial motivating case was memcpy's with alignments > 16. The
loads/stores, to which small memcpy's expand, are kept

[AArch64LoadStoreOptimizer] Generate more STPs by renaming registers earlier

Our initial motivating case was memcpy's with alignments > 16. The
loads/stores, to which small memcpy's expand, are kept together in
several places so that we get a sequence like this for a 64 bit copy:
LD w0
LD w1
ST w0
ST w1
The load/store optimiser can generate a LDP/STP w0, w1 from this because
the registers read/written are consecutive. In our case however, the
sequence is optimised during ISel, resulting in:
LD w0
ST w0
LD w0
ST w0
This instruction reordering allows reuse of registers. Since the registers
are no longer consecutive (i.e. they are the same), it inhibits LDP/STP
creation. The approach here is to perform renaming:
LD w0
ST w0
LD w1
ST w1
to enable the folding of the stores into a STP. We do not yet generate
the LDP due to a limitation in the renaming implementation, but plan to
look at that in a follow-up so that we fully support this case. While
this was initially motivated by certain memcpy's, this is a general
approach and thus is beneficial for other cases too, as can be seen
in some test changes.

Differential Revision: https://reviews.llvm.org/D103597

show more ...

[AArch64] Fix for the pre-indexed paired load/store optimization.

This patch fixes an issue where a pre-indexed store e.g.,
STR x1, [x0, #24]! with a store like STR x0, [x0, #8] are
merged into a si

[AArch64] Fix for the pre-indexed paired load/store optimization.

This patch fixes an issue where a pre-indexed store e.g.,
STR x1, [x0, #24]! with a store like STR x0, [x0, #8] are
merged into a single store: STP x1, x0, [x0, #24]!
. They shouldn’t be merged because the second store uses
x0 as both the stored value and the address and so it needs to be using the updated x0.
Therefore, it should not be folded into a STP <>pre.

Additionally a new test case is added to verify this fix.

Differential Revision: https://reviews.llvm.org/D101888

Change-Id: I26f1985ac84e970961e2cdca23c590fa6773851a

show more ...

[AArch64] Adds a pre-indexed paired Load/Store optimization for LDR-STR.

This patch merges STR<S,D,Q,W,X>pre-STR<S,D,Q,W,X>ui and
LDR<S,D,Q,W,X>pre-LDR<S,D,Q,W,X>ui instruction pairs into a single
S

[AArch64] Adds a pre-indexed paired Load/Store optimization for LDR-STR.

This patch merges STR<S,D,Q,W,X>pre-STR<S,D,Q,W,X>ui and
LDR<S,D,Q,W,X>pre-LDR<S,D,Q,W,X>ui instruction pairs into a single
STP<S,D,Q,W,X>pre and LDP<S,D,Q,W,X>pre instruction, respectively.
For each pair, there is a MIR test that verifies this optimization.

Differential Revision: https://reviews.llvm.org/D99272

Change-Id: Ie97a20c8c716c08492fe229c22e14e3c98ef08b7

show more ...

[AArch64] Do not fold SP adjustments into pre-increment addr modes if it overflows the redzone.

Instead of outright disabling this completely with the noredzone attribute,
we only avoid doing the op

[AArch64] Do not fold SP adjustments into pre-increment addr modes if it overflows the redzone.

Instead of outright disabling this completely with the noredzone attribute,
we only avoid doing the optimization if there are memory operations between
the adjustment and the load/store that the adjustment would be folded into.
This avoids the case of something like a stack cookie being corrupted if an
exception happens before the pre-increment to the SP occurs.

This also prevents the folding happening if we have a redzone, but the offset
being folded is above the redzone amount (128 bytes in this case).

rdar://73269336

Differential Revision: https://reviews.llvm.org/D95179

show more ...

[AArch64] Don't merge sp decrement into later stores when using WinCFI

This matches the corresponding existing case in
AArch64LoadStoreOpt::findMatchingUpdateInsnForward.

Both cases could also be m

[AArch64] Don't merge sp decrement into later stores when using WinCFI

This matches the corresponding existing case in
AArch64LoadStoreOpt::findMatchingUpdateInsnForward.

Both cases could also be modified to check
MBBI->getFlag(FrameSetup/FrameDestroy) instead of forbidding any
optimization involving SP, but the effect is probably pretty much
the same.

Differential Revision: https://reviews.llvm.org/D88541

show more ...

[AArch64] Avoid pairing loads when the base reg is modified

When pairing loads, we should check if in between the two loads the
base register has been modified. If that is the case then avoid pairin

[AArch64] Avoid pairing loads when the base reg is modified

When pairing loads, we should check if in between the two loads the
base register has been modified. If that is the case then avoid pairing
them because the second load actually loads from a different address.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D86956

show more ...

[AArch64] Fix ldst optimization of non-immediate store offset

When matching store instruction for ldst opt, we should make sure store instr is in 'reg+imm' form as load instr,
otherwise, it will hav

[AArch64] Fix ldst optimization of non-immediate store offset

When matching store instruction for ldst opt, we should make sure store instr is in 'reg+imm' form as load instr,
otherwise, it will have assertion in isLdOffsetInRangeOfSt since it will use getImm() directly.

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D87905

show more ...

[AArch64] Avoid pairing loads with same result reg

When pairing ldr instructions to an ldp instruction, we cannot pair two ldr
destination registers where one is a sub or super register of the other

[AArch64] Avoid pairing loads with same result reg

When pairing ldr instructions to an ldp instruction, we cannot pair two ldr
destination registers where one is a sub or super register of the other.

Reviewed By: fhahn

Differential Revision: https://reviews.llvm.org/D86906

show more ...

[AArch64] Fix ldst-opt of multiple disjunct subregs.

Currently aarch64-ldst-opt will incorrectly rename registers with
multiple disjunct subregisters (e.g. result of LD3). This patch updates
the can

[AArch64] Fix ldst-opt of multiple disjunct subregs.

Currently aarch64-ldst-opt will incorrectly rename registers with
multiple disjunct subregisters (e.g. result of LD3). This patch updates
the canRenameUpToDef to bail out if it encounters such a register class
that contains the register to rename.

Fixes PR46105.

Reviewers: efriedma, dmgreen, paquette, t.p.northover

Reviewed By: efriedma

Differential Revision: https://reviews.llvm.org/D81108

show more ...

[llvm][CodeGen] Check for memory instructions when querying for alias status

Summary:
Add a check to make sure that MachineInstr::mayAlias returns prematurely if at least one of its instruction para

[llvm][CodeGen] Check for memory instructions when querying for alias status

Summary:
Add a check to make sure that MachineInstr::mayAlias returns prematurely if at least one of its instruction parameters does not access memory. This prevents calls to TargetInstrInfo::areMemAccessesTriviallyDisjoint with incompatible instructions.

A side effect of this change is to render the mayAlias helper in the AArch64 load/store optimizer obsolete. We can now directly call the MachineInstr::mayAlias member function.

Reviewers: hfinkel, t.p.northover, mcrosier, eli.friedman, efriedma

Reviewed By: efriedma

Subscribers: efriedma, kristof.beyls, hiraditya, llvm-commits

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D78823

show more ...

[AArch64LoadStoreOptimizer] Skip debug insts during pattern matching [12/14]

Do not count the presence of debug insts against the limit set by
LdStLimit, and allow the optimizer to find matching ins

[AArch64LoadStoreOptimizer] Skip debug insts during pattern matching [12/14]

Do not count the presence of debug insts against the limit set by
LdStLimit, and allow the optimizer to find matching insts by skipping
over debug insts.

Differential Revision: https://reviews.llvm.org/D78411

show more ...

[AArch64][Fix] LdSt optimization generate premature stack-popping

Summary:
When moving add and sub to memory operand instructions,
aarch64-ldst-opt would prematurally pop the stack pointer,
before m

[AArch64][Fix] LdSt optimization generate premature stack-popping

Summary:
When moving add and sub to memory operand instructions,
aarch64-ldst-opt would prematurally pop the stack pointer,
before memory instructions that do access the stack using
indirect loads.
e.g.
```
int foo(int offset){
int local[4] = {0};
return local[offset];
}
```
would generate:
```
sub sp, sp, #16 ; Push the stack
mov x8, sp ; Save stack in register
stp xzr, xzr, [sp], #16 ; Zero initialize stack, and post-increment, making it invalid
------ If an exception goes here, the stack value might be corrupted
ldr w0, [x8, w0, sxtw #2] ; Access correct position, but it is not guarded by SP
```

Reviewers: fhahn, foad, thegameg, eli.friedman, efriedma

Reviewed By: efriedma

Subscribers: efriedma, kristof.beyls, hiraditya, danielkiss, llvm-commits, simon_tatham

Tags: #llvm

Differential Revision: https://reviews.llvm.org/D75755

show more ...