[X86] Add RDPRU instruction

Add support for the RDPRU instruction on Zen2 processors.

User-facing features:

- Clang option -m[no-]rdpru to enable/disable the feature
- Support is implicit for znve

[X86] Add RDPRU instruction

Add support for the RDPRU instruction on Zen2 processors.

User-facing features:

- Clang option -m[no-]rdpru to enable/disable the feature
- Support is implicit for znver2/znver3 processors
- Preprocessor symbol __RDPRU__ to indicate support
- Header rdpruintrin.h to define intrinsics
- "rdpru" mnemonic supported for assembler code

Internal features:

- Clang builtin __builtin_ia32_rdpru
- IR intrinsic @llvm.x86.rdpru

Differential Revision: https://reviews.llvm.org/D128934

show more ...

[PS5] Use same debug trap instruction as PS4

[X86] Fix cosmetic issues in instruction mnemonics

- Remove spurious } in invlpgb mnemonic
- Add \t between mnemonic and operands for ud1 instructions

Reviewed By: skan, craig.topper

Differential

[X86] Fix cosmetic issues in instruction mnemonics

- Remove spurious } in invlpgb mnemonic
- Add \t between mnemonic and operands for ud1 instructions

Reviewed By: skan, craig.topper

Differential Revision: https://reviews.llvm.org/D121570

show more ...

[X86] Don't add implicit REP prefix to VIA PadLock xstore

Commit 8fa3e8fa1492 added an implicit REP prefix to all VIA PadLock
instructions, but GNU as doesn't add one to xstore, only all the others.

[X86] Don't add implicit REP prefix to VIA PadLock xstore

Commit 8fa3e8fa1492 added an implicit REP prefix to all VIA PadLock
instructions, but GNU as doesn't add one to xstore, only all the others.
This resulted in a kernel panic regression in FreeBSD upon updating to
LLVM 11 (https://bugs.freebsd.org/259218) which includes the commit in
question. This partially reverts that commit.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D112355

show more ...

[X86] Remove isel predicates for xgetbv/xsetbv instructions so they can work on Windows.

https://reviews.llvm.org/D56686 was supposed to allow these to
work on Windows without needing to enable the

[X86] Remove isel predicates for xgetbv/xsetbv instructions so they can work on Windows.

https://reviews.llvm.org/D56686 was supposed to allow these to
work on Windows without needing to enable the xsave feature to
match MSVC. It seems this didn't work because the backend isel
patterns would still block it.

This patch removes the predicates from the isel patterns.

Fixes PR51706.

Reviewed By: pengfei

Differential Revision: https://reviews.llvm.org/D109097

show more ...

[X86] Modify LOOP*, HLT control flow attributes

Add missing control flow attributes:
- LOOP*: isBranch, isTerminator
- HLT: isTerminator

This helps downstream disassemblers (such as BOLT) reconstru

[X86] Modify LOOP*, HLT control flow attributes

Add missing control flow attributes:
- LOOP*: isBranch, isTerminator
- HLT: isTerminator

This helps downstream disassemblers (such as BOLT) reconstruct the control
flow graph more accurately.

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D102297

show more ...

[NFCI][X86] Mark a few lately-added system instructions as such for Scheduling purposes

[X86] Add segment and address-size override prefixes

X86 allows for the "addr32" and "addr16" address size override prefixes.
Also, these and the segment override prefixes should be recognized as
va

[X86] Add segment and address-size override prefixes

X86 allows for the "addr32" and "addr16" address size override prefixes.
Also, these and the segment override prefixes should be recognized as
valid prefixes.

Differential Revision: https://reviews.llvm.org/D94726

show more ...

UBSAN: emit distinctive traps

Sometimes people get minimal crash reports after a UBSAN incident. This change
tags each trap with an integer representing the kind of failure encountered,
which can ai

UBSAN: emit distinctive traps

Sometimes people get minimal crash reports after a UBSAN incident. This change
tags each trap with an integer representing the kind of failure encountered,
which can aid in tracking down the root cause of the problem.

show more ...

[X86] Don't disassemble wbinvd with 0xf2 or 0x66 prefix.

The 0xf3 prefix has been defined as wbnoinvd on Icelake Server. So
the prefix isn't ignored by the CPU. AMD documentation suggests that
wbnoi

[X86] Don't disassemble wbinvd with 0xf2 or 0x66 prefix.

The 0xf3 prefix has been defined as wbnoinvd on Icelake Server. So
the prefix isn't ignored by the CPU. AMD documentation suggests that
wbnoinvd is treated as wbinvd on older processors. Intel documentation
is not clear. Perhaps 0xf2 and 0x66 are treated the same, but its
not documented.

This patch changes TB to PS in the td file so 0xf2 and 0x66 will
be treated as errors. This matches versions of objdump after
wbnoinvd was added.

show more ...

[X86] Allow lsl/lar to be parsed with a GR16, GR32, or GR64 as source register.

This matches GNU assembler behavior. Operand size is determined
only from the destination register.

[X86] Teach assembler parser to accept lsl and lar with a 64 or 32 source register when the destination is a 64 register.

Previously we only accepted a 32-bit source with a 64-bit dest.

Accepting 6

[X86] Teach assembler parser to accept lsl and lar with a 64 or 32 source register when the destination is a 64 register.

Previously we only accepted a 32-bit source with a 64-bit dest.

Accepting 64-bit as well is more consistent with gas behavior. I
think maybe we should accept 16 bit register as well, but I'm not
sure.

show more ...

[X86] Correct the implementation of ud1(a.k.a. ud2b) instruction.

We were missing the modrm byte this instruction has according
to current Intel SDM. Experiments with gcc indicate that different
mod

[X86] Correct the implementation of ud1(a.k.a. ud2b) instruction.

We were missing the modrm byte this instruction has according
to current Intel SDM. Experiments with gcc indicate that different
modrm values are chosen based on 2 operands so I've added those
as well.

I think our previous implementation was based on an older behavior of
binutils that has since been changed.

show more ...

[X86] Force VIA PadLock crypto instructions to emit a 0xF3 prefix when they encode to match what GNU as does.

The spec for these says they need 0xf3 but also mentions REP
before the mnemonic. But I

[X86] Force VIA PadLock crypto instructions to emit a 0xF3 prefix when they encode to match what GNU as does.

The spec for these says they need 0xf3 but also mentions REP
before the mnemonic. But I don't think its fair to users to make
them write REP first. And gas doesn't make them. objdump seems to
disassemble with or without the prefix and just prints any 0xf3
as REP.

show more ...

[X86] Replace TB with PS on instructions that are documented in the SDM with 'NP'

'NP' means that the instruction is not recognized with a 66, F2 or F3
prefix. It will either #UD or decode to a diff

[X86] Replace TB with PS on instructions that are documented in the SDM with 'NP'

'NP' means that the instruction is not recognized with a 66, F2 or F3
prefix. It will either #UD or decode to a different instruction.

All of the cases are here should fall into the #UD variety since
we should be detecting the collision with other instructions when
we build the disassembler tables.

show more ...

[X86] Add a new format type for instructions that represent named prefix bytes like data16 and rep. Use it to make a simpler version of isPrefix.

isPrefix was added to support the patches to align b

[X86] Add a new format type for instructions that represent named prefix bytes like data16 and rep. Use it to make a simpler version of isPrefix.

isPrefix was added to support the patches to align branches.
it relies on a switch over instruction names.

This moves those opcodes to a new format so the information is
tablegen and we can just check for a specific value in some bits
in TSFlags instead.

I've left the other function in place for now so that the
existing patches in phabricator will still work. I'll work with
the owner to get them migrated.

show more ...

Reapply r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"

This reverts r372314, reapplying r372285 and the commits which depend
on it (r372286-r372293, and r372296-r372297)

Thi

Reapply r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"

This reverts r372314, reapplying r372285 and the commits which depend
on it (r372286-r372293, and r372296-r372297)

This was missing one switch to getTargetConstant in an untested case.

llvm-svn: 372338

show more ...

Revert r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"

This broke the Chromium build, causing it to fail with e.g.

fatal error: error in backend: Cannot select: t362: v4i32

Revert r372285 "GlobalISel: Don't materialize immarg arguments to intrinsics"

This broke the Chromium build, causing it to fail with e.g.

fatal error: error in backend: Cannot select: t362: v4i32 = X86ISD::VSHLI t392, Constant:i8<15>

See llvm-commits thread of r372285 for details.

This also reverts r372286, r372287, r372288, r372289, r372290, r372291,
r372292, r372293, r372296, and r372297, which seemed to depend on the
main commit.

> Encode them directly as an imm argument to G_INTRINSIC*.
>
> Since now intrinsics can now define what parameters are required to be
> immediates, avoid using registers for them. Intrinsics could
> potentially want a constant that isn't a legal register type. Also,
> since G_CONSTANT is subject to CSE and legalization, transforms could
> potentially obscure the value (and create extra work for the
> selector). The register bank of a G_CONSTANT is also meaningful, so
> this could throw off future folding and legalization logic for AMDGPU.
>
> This will be much more convenient to work with than needing to call
> getConstantVRegVal and checking if it may have failed for every
> constant intrinsic parameter. AMDGPU has quite a lot of intrinsics wth
> immarg operands, many of which need inspection during lowering. Having
> to find the value in a register is going to add a lot of boilerplate
> and waste compile time.
>
> SelectionDAG has always provided TargetConstant for constants which
> should not be legalized or materialized in a register. The distinction
> between Constant and TargetConstant was somewhat fuzzy, and there was
> no automatic way to force usage of TargetConstant for certain
> intrinsic parameters. They were both ultimately ConstantSDNode, and it
> was inconsistently used. It was quite easy to mis-select an
> instruction requiring an immediate. For SelectionDAG, start emitting
> TargetConstant for these arguments, and using timm to match them.
>
> Most of the work here is to cleanup target handling of constants. Some
> targets process intrinsics through intermediate custom nodes, which
> need to preserve TargetConstant usage to match the intrinsic
> expectation. Pattern inputs now need to distinguish whether a constant
> is merely compatible with an operand or whether it is mandatory.
>
> The GlobalISelEmitter needs to treat timm as a special case of a leaf
> node, simlar to MachineBasicBlock operands. This should also enable
> handling of patterns for some G_* instructions with immediates, like
> G_FENCE or G_EXTRACT.
>
> This does include a workaround for a crash in GlobalISelEmitter when
> ARM tries to uses "imm" in an output with a "timm" pattern source.

llvm-svn: 372314

show more ...

GlobalISel: Don't materialize immarg arguments to intrinsics

Encode them directly as an imm argument to G_INTRINSIC*.

Since now intrinsics can now define what parameters are required to be
immediat

GlobalISel: Don't materialize immarg arguments to intrinsics

Encode them directly as an imm argument to G_INTRINSIC*.

Since now intrinsics can now define what parameters are required to be
immediates, avoid using registers for them. Intrinsics could
potentially want a constant that isn't a legal register type. Also,
since G_CONSTANT is subject to CSE and legalization, transforms could
potentially obscure the value (and create extra work for the
selector). The register bank of a G_CONSTANT is also meaningful, so
this could throw off future folding and legalization logic for AMDGPU.

This will be much more convenient to work with than needing to call
getConstantVRegVal and checking if it may have failed for every
constant intrinsic parameter. AMDGPU has quite a lot of intrinsics wth
immarg operands, many of which need inspection during lowering. Having
to find the value in a register is going to add a lot of boilerplate
and waste compile time.

SelectionDAG has always provided TargetConstant for constants which
should not be legalized or materialized in a register. The distinction
between Constant and TargetConstant was somewhat fuzzy, and there was
no automatic way to force usage of TargetConstant for certain
intrinsic parameters. They were both ultimately ConstantSDNode, and it
was inconsistently used. It was quite easy to mis-select an
instruction requiring an immediate. For SelectionDAG, start emitting
TargetConstant for these arguments, and using timm to match them.

Most of the work here is to cleanup target handling of constants. Some
targets process intrinsics through intermediate custom nodes, which
need to preserve TargetConstant usage to match the intrinsic
expectation. Pattern inputs now need to distinguish whether a constant
is merely compatible with an operand or whether it is mandatory.

The GlobalISelEmitter needs to treat timm as a special case of a leaf
node, simlar to MachineBasicBlock operands. This should also enable
handling of patterns for some G_* instructions with immediates, like
G_FENCE or G_EXTRACT.

This does include a workaround for a crash in GlobalISelEmitter when
ARM tries to uses "imm" in an output with a "timm" pattern source.

llvm-svn: 372285

show more ...

[X86] Remove CustomInserters for RDPKRU/WRPKRU. Use some custom lowering and new ISD opcodes instead.

These inserters inserted some instructions to zero some registers and copied from virtual regist

[X86] Remove CustomInserters for RDPKRU/WRPKRU. Use some custom lowering and new ISD opcodes instead.

These inserters inserted some instructions to zero some registers and copied from virtual registers to physical registers.

This change instead inserts the zeros directly into the DAG at lowering time using new ISD opcodes
that take the extra zeroes as inputs. The zeros will then go through isel on their own to select
the MOV32r0 pseudo. Then we just need to mention the physical registers directly
in the isel patterns and the isel table and InstrEmitter will take care of inserting the necessary
copies to/from physical registers.

llvm-svn: 357659

show more ...

[X86] Remove X86 specific dag nodes for RDTSC/RDTSCP/RDPMC. NFCI

This patch removes the following dag node opcodes from namespace X86ISD:

RDTSC_DAG,
RDTSCP_DAG,
RDPMC_DAG

The logic that expands RD

[X86] Remove X86 specific dag nodes for RDTSC/RDTSCP/RDPMC. NFCI

This patch removes the following dag node opcodes from namespace X86ISD:

RDTSC_DAG,
RDTSCP_DAG,
RDPMC_DAG

The logic that expands RDTSC/RDPMC/XGETBV intrinsics is basically the same. The
only differences are:

RDTSC/RDTSCP don't implicitly read ECX.
RDTSCP also implicitly writes ECX.

I moved the common expansion logic into a helper function with the goal to get
rid of code repetition. That helper is now used for the expansion of
RDTSC/RDTSCP/RDPMC/XGETBV intrinsics.

No functional change intended.

Differential Revision: https://reviews.llvm.org/D59547

llvm-svn: 356546

show more ...

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the ne

Update the file headers across all of the LLVM projects in the monorepo
to reflect the new license.

We understand that people may be surprised that we're moving the header
entirely to discuss the new license. We checked this carefully with the
Foundation's lawyer and we believe this is the correct approach.

Essentially, all code in the project is now made available by the LLVM
project under our new license, so you will see that the license headers
include that license only. Some of our contributors have contributed
code under our old license, and accordingly, we have retained a copy of
our old license notice in the top-level files in each project and
repository.

llvm-svn: 351636

show more ...

[cfi-verify] Support AArch64.

This patch adds support for AArch64 to cfi-verify.

This required three changes to cfi-verify. First, it generalizes checking if an instruction is a trap by adding a n

[cfi-verify] Support AArch64.

This patch adds support for AArch64 to cfi-verify.

This required three changes to cfi-verify. First, it generalizes checking if an instruction is a trap by adding a new isTrap flag to TableGen (and defining it for x86 and AArch64). Second, the code that ensures that the operand register is not clobbered between the CFI check and the indirect call needs to allow a single dereference (in x86 this happens as part of the jump instruction). Third, we needed to ensure that return instructions are not counted as indirect branches. Technically, returns are indirect branches and can be covered by CFI, but LLVM's forward-edge CFI does not protect them, and x86 does not consider them, so we keep that behavior.

In addition, we had to improve AArch64's code to evaluate the branch target of a MCInst to handle calls where the destination is not the first operand (which it often is not).

Differential Revision: https://reviews.llvm.org/D48836

llvm-svn: 337007

show more ...

[X86] Add NotMemoryFoldable to a bunch of instructions to suppress them from the autogenerated load folding table.

Most of these are system instructions or other instructions we don't use in CodeGen

[X86] Add NotMemoryFoldable to a bunch of instructions to suppress them from the autogenerated load folding table.

Most of these are system instructions or other instructions we don't use in CodeGen. No point wasting space for them in the table. Removing them from the autogenerated table makes it easier to review the manual table.

A few are real opcode collisions where the memory and register forms are completely different instructions.

llvm-svn: 334474

show more ...

[x86] invpcid LLVM intrinsic

Re-add the feature flag for invpcid, which was removed in r294561.
Add an intrinsic, which always uses a 32 bit integer as first argument,
while the instruction actually

[x86] invpcid LLVM intrinsic

Re-add the feature flag for invpcid, which was removed in r294561.
Add an intrinsic, which always uses a 32 bit integer as first argument,
while the instruction actually uses a 64 bit register in 64 bit mode
for the INVPCID_TYPE argument.

Reviewers: craig.topper

Reviewed By: craig.topper

Differential Revision: https://reviews.llvm.org/D47141

llvm-svn: 333255

show more ...