Couple cleanups to the flags/settings handling in Cranelift (#11744)

* Remove unused shared flags

* Get rid of predicate settings

They were important in the old backend framework, but with the new

Couple cleanups to the flags/settings handling in Cranelift (#11744)

* Remove unused shared flags

* Get rid of predicate settings

They were important in the old backend framework, but with the new
backend framework if we need a combination of multiple settings, that
can just be done as a regular extractor doing &&. This simplifies the
settings implementation.

show more ...

x64: Migrate `setcc` instructions to the new assembler (#11112)

These instructions were relatively straightforward and the notable parts
here are:

* The manual doesn't have `/0` but these instructi

x64: Migrate `setcc` instructions to the new assembler (#11112)

These instructions were relatively straightforward and the notable parts
here are:

* The manual doesn't have `/0` but these instructions use `.digit(0)`,
but the manual also says the reg bits are ignored in the ModR/M byte
so it doesn't actually matter what we put in these bits, just need to
put something in there.

* The previous implementation in `emit.rs` unconditionally emitted a REX
byte but this was not necessary for some registers, so many binary
offsets have changed in tests as some unnecessary REX bytes are now
omitted.

* As with `cmov*` instructions in #11095 some mnemonics that Cranelift
uses differ from the canonical ones that Capstone disassembles, so
there are some naming discrepancies which are documented inline as
well.

show more ...

x64: Migrate `cmov*` to the new assembler (#11095)

This commit adds all `cmov*` variants from the Intel manual to the new
assembler. This then additionally removes the `Cmove` pseudo-inst in
favor o

x64: Migrate `cmov*` to the new assembler (#11095)

This commit adds all `cmov*` variants from the Intel manual to the new
assembler. This then additionally removes the `Cmove` pseudo-inst in
favor of these new instructions. One difference from before is that the
naming in the `CC` enum does not exactly match what mnemonics Capstone
uses to disassemble. For example `CC.NB` in ISLE corresponds to the
Intel instruction `CMOVNB`. This instruction, however, has the same
encoding as `CMOVAE` and Capstone disassembles as `CMOVAE`. This means
that the instruction selection in ISLE isn't a 1:1 match with mnemonics.

This additionally adds support in the assembler ISLE generation to
understand that instructions which read EFLAGS generate a
`ConsumesFlags` variant in their instruction helpers.

show more ...

x64: convert unary VEX instructions (#11088)

* asm: cover more of SSE in `Features::is_sse()`

* asm: add `VexPrefix::two_op`

This is simply a convenience method when `vvvv` is unspecified.

* asm:

x64: convert unary VEX instructions (#11088)

* asm: cover more of SSE in `Features::is_sse()`

* asm: add `VexPrefix::two_op`

This is simply a convenience method when `vvvv` is unspecified.

* asm: print failing instruction in SSE-to-AVX alternate mapping

* asm: fill out AVX functionality for unary VEX instructions

These include `vmov*`, `vpmov*`, `vsqrt`, and some lane operations.

* asm: add `Location::xmm_m8`

* asm: add av2 feature

* asm: add AVX broadcast instructions

* x64: add `is_mem` for extracting addresses from `XmmMem`

* x64: convert unary VEX instructions

* x64: bless Cranelift filetests

* x64: use alternate AVX version for `pabs*`

* x64: bless `pabs*` Cranelift filetests

* x64: remove `XmmUnaryRmRVex` and now-unnecessary `AvxOpcode`s

* winch: convert `XmmUnaryRmRVex` instructions

show more ...

x64: Migrate cmp/test to the new assembler (#11072)

* x64: Migrate cmp/test to the new assembler

This commit migrates the `cmp` and `test` family of instructions, the
`CmpRmiR` variant in ISLE, to

x64: Migrate cmp/test to the new assembler (#11072)

* x64: Migrate cmp/test to the new assembler

This commit migrates the `cmp` and `test` family of instructions, the
`CmpRmiR` variant in ISLE, to the new assembler. This required
fiddling with various helpers for immediates and such throughout a few
locations and refactoring various callsites of creating these
instructions to fit nicely into these new idioms. EFLAGS-writing
instructions with no other results are now also modeled in the assembler
as `ProducesFlags` rather than `SideEffectNoResult` as their base case
enabling using the generated constructors a bit more too.

No functional change is expected from this, it should just be internal
refactoring.

* Swap between rm and mr encodings

show more ...

x64: Migrate `cmpxchg{,16b}` to the new assembler (#11035)

* x64: Migrate `cmpxchg{,16b}` to the new assembler

A few new operands were needed to define this as well as a number of new
instruction "

x64: Migrate `cmpxchg{,16b}` to the new assembler (#11035)

* x64: Migrate `cmpxchg{,16b}` to the new assembler

A few new operands were needed to define this as well as a number of new
instruction "shapes" in a few locations, but nothing too too radical.

* Fix emit test

show more ...

x64: Migrate `LockXadd` to the new assembler (#11032)

* x64: Migrate `LockXadd` to the new assembler

This required a custom mnemonic implementation to handle the `lock`
prefix and then this also re

x64: Migrate `LockXadd` to the new assembler (#11032)

* x64: Migrate `LockXadd` to the new assembler

This required a custom mnemonic implementation to handle the `lock`
prefix and then this also required a custom visit implementation because
Intel says the memory operand comes first but regalloc requires the
register operand comes first.

* Remove unused variables

show more ...

x64: Migrate `xchg` to the new assembler (#11008)

This required a custom visit function because printing operands has to
happen in one order to match Capstone but allocating operations has to
happen

x64: Migrate `xchg` to the new assembler (#11008)

This required a custom visit function because printing operands has to
happen in one order to match Capstone but allocating operations has to
happen in another order to satisfy regalloc, so so one of the two has to
be custom.

show more ...

x64: Convert `compare` instructions to the new assembler (#10836)

* Implement compare instructions for new assembler, implement eflags logic, custom display logic for operands

* clippy lints fix

*

x64: Convert `compare` instructions to the new assembler (#10836)

* Implement compare instructions for new assembler, implement eflags logic, custom display logic for operands

* clippy lints fix

* add custom:display for compare instructions

* nit change panic message

* format fix, bless tests, prtest:full

show more ...

x64: generate rules to pick between SSE or AVX instructions (#10942)

* asm: introduce `Inst::alternate`, with auto-assignment of AVX to SSE

This adds the ability for an instruction to track an alte

x64: generate rules to pick between SSE or AVX instructions (#10942)

* asm: introduce `Inst::alternate`, with auto-assignment of AVX to SSE

This adds the ability for an instruction to track an alternate version
of itself. This is helpful for connecting SSE instructions to their
upgraded AVX version. The logic here compares the mnemonic name minus
the `v` prefix and checks that the opcodes are the same. This
auto-assignment cautiously limits us to SSE instructions like `rw(xmm),
r(xmm_m*)`, which have a most regular SSE-AVX relationship.

To see the list of instructions that have been assigned alternates, from
the `cranelift/assembler-x64/meta` directory, run:

```console
$ cargo run | grep alternate
```

* x64: use generated `x64_..._or_avx` rule to lower `x64_addp*`

This change updates `gen_asm.rs` to emit an extra rule for instructions
that have an AVX alternate: `x64_<inst>_or_avx`. The rule logic has the
same shape as what we currently hand-write ~180 times in `inst.isle`
(!).

* review: make SSE-to-AVX replacements explicit

This change uses `Inst::alt()` to explicitly mark which instructions
have an AVX alternate. This retains the strict checks preventing us from
manually making a mistake. It also passes along the `Feature` in an
`Alternate` structure to make this easier to extend later.

* review: move opcode calculation

show more ...

x64: Migrate push/pop to the new assembler (#10977)

* x64: Migrate push/pop to the new assembler

Figured these would be some interesting instruction shapes. This adds a
helper to the ISLE generati

x64: Migrate push/pop to the new assembler (#10977)

* x64: Migrate push/pop to the new assembler

Figured these would be some interesting instruction shapes. This adds a
helper to the ISLE generation to skip some instructions and the "push"
instructions are skipped as they're not needed in ISLE and would
otherwise require binding new instruction shapes which didn't seem worth
it. Additionally many items in `gen_asm.rs` were made private (removed
`pub`).

* Update test expectations

show more ...

x64: Migrate `mulx` to the new assembler (#10887)

* x64: Migrate `mulx` to the new assembler

This is an interesting instruction as it has a relatively unique shape
compared to many others. The VEX

x64: Migrate `mulx` to the new assembler (#10887)

* x64: Migrate `mulx` to the new assembler

This is an interesting instruction as it has a relatively unique shape
compared to many others. The VEX encoding is used to give it a 3-operand
form, although it still has an implicit 4th operand as well. The other
unique part about this instruction is that if the two write-only
operands are the same then that has a different semantic meaning than if
they are different.

Modeling the two-output form of the instruction was pretty easy, the
only changes needed were to add the `r32a` and `r32b` locations as
previously only `r32` was available. Modeling the one-output form of the
instruction led to a "hook" where these instructions specify that they
use a custom regalloc function. That skips the auto-generated regalloc
entirely and defers to a new `custom` submodule in the assembler crate.
These custom variants handle only having a single write operand on the
instruction.

* Fix fuzz build

show more ...

x64: Migrate xmm mov-family instructions to new assembler (#10834)

* x64: Migrate xmm mov-family instructions to new assembler

This commit migrates xmm movement-style instructions, such as
`mov{s{s

x64: Migrate xmm mov-family instructions to new assembler (#10834)

* x64: Migrate xmm mov-family instructions to new assembler

This commit migrates xmm movement-style instructions, such as
`mov{s{s,d},{a,u}p{s,d},dq{a,u}}` to the new assembler. This enables
deleting the `XmmMovRM` class of instructions in ISLE. Along the way a
number of notable changes were made:

* ISLE constructors for assembler instructions now take `SyntheticAmode`
instead of `Amode` since it's already supported anyway and it's a more
flexible argument to take.
* The conversion from `SyntheticAmode` to the assembler `Amode<Gpr>` was
fixed where one variant needed to use `rsp` instead of `rbp`.
* The `is_move` method was updated to ignore `movss` and `movsd`
instructions since they're not true movement-related instructions in
register-to-register situations.

* Use `gen_move` helper in Winch

show more ...

x64: Convert some lane-manipulation instructions to the new assembler (#10825)

* x64: Convert some lane-manipulation instructions to the new assembler

This was mostly straightforward but ended up r

x64: Convert some lane-manipulation instructions to the new assembler (#10825)

* x64: Convert some lane-manipulation instructions to the new assembler

This was mostly straightforward but ended up requiring handling for a
new style of instruction not previously before-seen: a write-only
`GprMem` operand. This needed some extra treatment/helpers in the ISLE
integration to ensure that this was all hooked up correctly from ISLE
constructors down to the assembler instructions.

* Remove `GprToXmm` as well

Already have mappings for `movd` and `movq`, so let's migrate this as well.

* Remove some old emit tests

show more ...

x64: Migrate div instructions to the new assembler (#10820)

* x64: Migrate div instructions to the new assembler

This mostly required adding a new kind of operand representing a
`TrapCode` and plum

x64: Migrate div instructions to the new assembler (#10820)

* x64: Migrate div instructions to the new assembler

This mostly required adding a new kind of operand representing a
`TrapCode` and plumbing around a few bits and bobs to ensure that this
compile-time-only-abstraction does not need to be accounted for in all
the encoding bits.

* Fix doc tests

* Shrink the size of `Amode`

Reduces the size of `KnownOffset` so `Option<KnownOffset>` doesn't have
such a high alignment or size, greatly shrinking the size of the
instruction and prevents the changes to `Div` from increasing the size
of the overall `Inst` enum. In fact it now shrinks!

* Don't model a trap code as an operand

* Render trap information in `Display`

show more ...

x64: convert all multiplication-related instructions (#10782)

* asm: add SSE4.1 feature

* asm: refactor format constructors with expanding structs

* asm: add implicit operands

Implicit operands a

x64: convert all multiplication-related instructions (#10782)

* asm: add SSE4.1 feature

* asm: refactor format constructors with expanding structs

* asm: add implicit operands

Implicit operands are used by an instruction but not present in its
disassembled output. Instructions like `mul`, e.g., will write to the
`%rax` and `%rdx` registers, but this is all invisible in disassembly.
Implicit operands are always fixed (i.e., the register is known), but
not all fixed operands are implicit (i.e., some fixed registers _are_
disassembled).

* asm: add `mul*` instructions, vector and scalar

* asm: generate ISLE for multi-return constructors--`AssemblerOutputs::RetValueRegs`

Certain `mul*` instructions write to multiple registers. For register
allocation, Cranelift needs to know about all of these registers. This
change uses the pre-existing pattern of returning a `ValueRegs` type to
indicate this. This change is limited to what is needed now: the only
multi-return needed now uses two fixed registers.

* x64: convert SSE multiplications

* x64: convert scalar multiplications

* x64: remove old `Inst::Mul*` variants

* winch: convert multiplications

This does not include any special "small immediate resizing" rules for
Winch, so the Winch disassembly tests gain a few bytes (e.g., some
immediates that _could_ fit in 8 bits are emitted as the full 32 bits).

* refactor: move `i32`-related extractors to `prelude.isle`

* review: print the implicit registers as well

* fix: the instruction size has changed (again)

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

asm: implement write-only operands (#10759)

* asm: implement write-only operands

This change adds support to the new assembler for write-only operands.
This implementation appeared first in [#10754

asm: implement write-only operands (#10759)

* asm: implement write-only operands

This change adds support to the new assembler for write-only operands.
This implementation appeared first in [#10754] but is split out here to
unblock implementation of instructions that require it: multiplication,
conversions, moves, etc. This starts roughly the same as what was
implemented for write-only XMMs in [#10754] but includes support for
write-only GPRs as well and generates the temporary registers which are
needed.

[#10754]: https://github.com/bytecodealliance/wasmtime/pull/10754

Co-authored-by: Johnnie Birch <[email protected]>

* fix: use `to_reg()` to extract the Cranelift type from `Writable`

---------

Co-authored-by: Johnnie Birch <[email protected]>

show more ...

asm: refactor using `RegClass` and `xmm_m{32|64|128}` (#10716)

This change involves no change in functionality but does pave the way
for defining more SSE instructions in two ways:
- it adds the `xm

asm: refactor using `RegClass` and `xmm_m{32|64|128}` (#10716)

This change involves no change in functionality but does pave the way
for defining more SSE instructions in two ways:
- it adds the `xmm_m32` and `xmm_m64` variants as locations and changes
`rm128` to read `xmm_m128` to match; this is necessary for SSE `*ss`
and `*sd` instructions that load 32- and 64-bit values into XMM
registers
- by returning `RegClass` from `Location::reg_class` it is much easier
to distinguish what kind of a register an operand is using:
previously this matched on the number of bits (i.e., 128) which does
no longer makes sense for `xmm_m32` and `xmm_m64`.

show more ...

asm: plumb fixed registers through (#10629)

* asm: plumb fixed registers through

This change introduces a new `Fixed` wrapper type that allows us to
plumb fixed registers through the assembler. The

asm: plumb fixed registers through (#10629)

* asm: plumb fixed registers through

This change introduces a new `Fixed` wrapper type that allows us to
plumb fixed registers through the assembler. These instructions are not
yet used by `cranelift-codegen`, but will be necessary at some point as
described in [#10238]. The end result of this is that up at the
`cranelift-codegen` level, fixed registers should look like
run-of-the-mill virtual registers using the appropriate types, but down
in `cranelift-assembler-x64` we panic if the register allocator does not
give them the correct register.

[#10238]: https://github.com/bytecodealliance/wasmtime/issues/10238

* fix: reduce `size_of::<Inst>` test

It is unclear why adding more `Fixed<R>` fields would somehow _reduce_
the size of `Inst`. What is quite clear is that I do not understand how
Rust chooses to lay out these fields.

show more ...

asm: add `LOCK`-prefixed instructions (#10477)

* asm: add initial `LOCK`-prefixed instructions

This change is the first in a series to add new instructions that emit
the `LOCK` prefix during encodi

asm: add `LOCK`-prefixed instructions (#10477)

* asm: add initial `LOCK`-prefixed instructions

This change is the first in a series to add new instructions that emit
the `LOCK` prefix during encoding.

* asm: add all `lock and*` instructions

This defines all `LOCK`-prefixed variants for `and*`.

* asm: add all `lock add|adc*` instructions

* asm: add all `lock or*` instructions

* asm: add all `lock sub|sbb*` instructions

* asm: add all `lock xor*` instructions

* asm: reformat using slightly longer lines

This allows the `LOCK`-prefixed definitions to still fit on a single
line.

* asm: integrate mem-only operands in `cranelift-codegen`

`LOCK`-prefixed instructions can only write to memory operands (see
prior commits). This change wires up the necessary codegen to handle the
new `m*` operands.

* asm: lower RMW instructions with new `LOCK`-prefixed instructions

* Bump allowed size of the `Inst` enum

It is unclear why this would happen now, though perhaps the addition of
all the `LOCK`-prefixed instructions could do it (?).

show more ...

asm: notate instruction alignment, SSE feature flags (#10446)

* asm: notate instruction alignment, SSE feature flags

This change improves the definitions of the assembler's SSE instructions
in two

asm: notate instruction alignment, SSE feature flags (#10446)

* asm: notate instruction alignment, SSE feature flags

This change improves the definitions of the assembler's SSE instructions
in two ways:
- vector instructions that require aligned memory accesses (i.e., most
everything pre-AVX) are now noted with an `align` attribute in the
AST. This is used for generating the expected `XmmMemAligned` types in
`cranelift-codegen-meta` the "right way," resolving the temporary fix
introduced in #10417.
- previously-added vector instructions did not have the correct feature
flags; this change adds the `sse` feature and also tags the applicable
instructions with the `compat` feature to allow their use in some future
32-bit target.

* asm: disallow duplicate features

show more ...

x64: fix misaligned load fault with sunk load when AVX is disabled (#10417)

In [#10408], the new assembler re-opened an old issue related to
unaligned loads with SSE instructions. SSE instructions e

x64: fix misaligned load fault with sunk load when AVX is disabled (#10417)

In [#10408], the new assembler re-opened an old issue related to
unaligned loads with SSE instructions. SSE instructions expect 128-bit
aligned loads when using the `m128` operand and fault if that is not the
case. This had been fixed previously by disallowing load-sinking for
`XmmMem` ([#4891]) but more recently we had adopted the use of
`XmmMemAligned` in `cranelift-codegen`. Since [#10316] had no knowledge
of `XmmMemAligned` (only `XmmMem`), it caused the same kind fault--an
OOB trap that was in fact an unaligned load.

Why didn't CI catch this? Since all the CI machines have AVX and we do
not explicitly test the SSE-only case, these unaligned, sunk loads would
use the AVX lowering in CI. AVX loads handle unaligned accesses without
a fault. This was only discovered during fuzzing when AVX was disabled
(i.e., `--target x86_64-unknown-linux-gnu`).

To fix this, this change adopts the `XmmMemAligned` type in the
generated assembler code. This is temporary, though: a more lasting fix
should pass along an "alignment required" bit from the assembler AST.
In the meantime, this closes #10408.

[#10408]: https://github.com/bytecodealliance/wasmtime/issues/10408
[#4891]: https://github.com/bytecodealliance/wasmtime/pull/4891
[#10316]: https://github.com/bytecodealliance/wasmtime/pull/10316

show more ...

meta: move all ISLE-related generated code to `cranelift-codegen-meta` (#10352)

* meta: move all ISLE-related generated code to `cranelift-codegen-meta`

Previously, the new `cranelift-assembler-x64

meta: move all ISLE-related generated code to `cranelift-codegen-meta` (#10352)

* meta: move all ISLE-related generated code to `cranelift-codegen-meta`

Previously, the new `cranelift-assembler-x64` crate would generate the
assembler Rust code, the ISLE integration code, and the ISLE's
accompanying Rust macro all in the same crate. Because these last two
files were generated in the assembler's `OUT_DIR`, they caused issues
for deterministic builds that expect those files in
`cranelift-codegen`'s `OUT_DIR` (see #10348). This change fixes that by
moving all of the assembler's generated integration code, the ISLE
instruction definitions and their accompanying Rust macro, to the
`cranelift-codegen-meta` crate.

This has the added advantage that `cranelift-assembler-x64` now can
focus exclusively on being an assembler and not being concerned with the
ISLE integration, which is substantial.

* Move lint `allow` to problem site

Apparently allowing unused variables for unimplemented parts of fixed
register instructions will not work at the macro scope; this moves the
`allow` to the offending line.

* Remove `rustfmt` of generated assembler code

show more ...