x64: handle ISA features more completely (#11272)

* asm: generate boolean terms of CPU features

As discussed [here], we will soon need the ability to express more
complex combinations of CPU featur

x64: handle ISA features more completely (#11272)

* asm: generate boolean terms of CPU features

As discussed [here], we will soon need the ability to express more
complex combinations of CPU features. These are best expressed as
boolean terms: e.g., `(32-bit OR 64-bit) AND ...`, `(32-bit OR 64-bit)
AND ((AVX512VL AND AVX512F) OR AVX10.1)`. This change modifies the
generated code to have a `Inst::is_available` method which contains a
Rust-ified version of the instruction's boolean term. To do this, we now
pass in a `Features` trait, which the instruction can query to see if
its desired feature set is available.

[here]: https://bytecodealliance.zulipchat.com/#narrow/channel/217117-cranelift/topic/boolean.20terms.20for.20x64.20features

* x64: wire up `Inst::is_available` in Cranelift

This change makes us of the assembler's new generated
`Inst::is_available` methods to check an instruction's feature set in a
more succinct (and likely quicker) way. Unfortunately, this does not
allow us to print the missing ISA requirements on failure--something to
address later.

* Rename `Features` to `AvailableFeatures`

* Remove unused `InstructionSet`

* asm: fix all feature definitions

This is a mechanical transformation converting all instruction
definitions. Now, instructions should have a correct boolean term
describe the features required: e.g., `(_64b | compat) & avx`.

* x64: replace `use_*` with `has_*` when checking ISA features

In Cranelift, the `has_*` flags of `isa::x64::settings::Flags` indicate
that the CPU _has_ some capability; the `use_*` flags indicate that
Cranelift _should emit_ instructions using those capabilities. Further,
the `use_*` flags may turned on by the presence of more than one `has_*`
flags; e.g., when `has_avx` and `has_avx2` are available, `use_avx2` is
enabled.

Now that Cranelift's new x64 assembler understands boolean terms, we no
longer need the `use_*` flags for checking if an instruction can be
emitted: instead, we should use the `has_*` flags and rely on the logic
encoded in `Inst::is_available`.

* asm: materialize `Features` via `Inst::features`

For better error messages (and just for general use of CPU features, see
discussion [here]), this change adds `Inst::features`--a way to
explicitly examine the boolean term for an instruction. This function
returns a `&'static Features` that contains the `AND` and `OR` branches
defining when an instruction is available. This is all generated into
something that looks like:

```rust
pub fn features(&self) -> &'static Features {
const F1: &'static Features = &Features::Feature(Feature::_64b);
const F2: &'static Features = &Features::Feature(Feature::compat);
const F0: &'static Features = &Features::Or(F1, F2);
F0
}
```

This change makes use of `for_each_feature` more: we build up the
`AvailableFeatures` trait and the `Feature` enum from it. This should be
a bit more direct than searching through the generated code (?).

[here]: https://github.com/bytecodealliance/wasmtime/pull/11272#discussion_r2214465871

show more ...

x64: Migrate EVEX instructions to the new assembler (#11303)

* x64: Migrate vpabsq to the new assembler

* x64: Migrate vpopcntb to new assembler

* x64: Migrate vpmullq to the new assembler

* x64:

x64: Migrate EVEX instructions to the new assembler (#11303)

* x64: Migrate vpabsq to the new assembler

* x64: Migrate vpopcntb to new assembler

* x64: Migrate vpmullq to the new assembler

* x64: Migrate vcvtudq2ps to the new assembler

* x64: Migrate vpermi2b to new assembler

* Fix test warnings

show more ...

x64: Add EVEX shifts to the new assembler (#11270)

* x64: Add EVEX shifts to the new assembler

This commit adds bindings for the EVEX encodings of `vps{ll,ra,rl}{d,q}`
to the new assembler. Current

x64: Add EVEX shifts to the new assembler (#11270)

* x64: Add EVEX shifts to the new assembler

This commit adds bindings for the EVEX encodings of `vps{ll,ra,rl}{d,q}`
to the new assembler. Currently the 16-bit shifts `vps{ll,ra,rl}w` are
omitted due to the `avx512bw` feature not yet being bound in Cranelift.
In implementing these instructions a few refactorings/fixes were
necessary:

* Primarily all EVEX instructions now need to be defined not only with
their vector length but also their "tuple type" found in encoding
tables. This is required to correctly handle the 8-bit displacement
scaling that happens with EVEX instructions.
* Some small helpers to the `Evex` structure were added such as
`Evex::digit` and `Evex::ib`.
* The `evex_scaling` factor is now calculated in `generate_evex_prefix`
according to the instruction format itself.
* The VEX and EVEX `generate_*_prefix` functions now delegate to a
shared function to handle the same operand formats across both of
them.
* Fuzz generation of `AmodeOffset` is now updated to bias to some
"interesting" offsets that exercise the cases where EVEX scaling is
necessary.
* The ISLE `XmmUnaryRmRImmEvex` instruction format was removed as it's
no longer necessary.

* Fix emit tests

show more ...

x64: EVEX Encoding for new assembler (#11153)

* evex prefix for new assembler, vaddpd inst, prtest:full

* review: refactor `VexLength` and `EvexLength` into a single `Length`

* review: fix order o

x64: EVEX Encoding for new assembler (#11153)

* evex prefix for new assembler, vaddpd inst, prtest:full

* review: refactor `VexLength` and `EvexLength` into a single `Length`

* review: fix order of instruction definition to match manual

Also, adds some missing `.r()` helpers for consistency.

* review: remove opmask registers and mask kind for now

This will require more work to expose these as special,
runtime-available flags on the generated instruction.

* review: remove as-yet unknown operand patterns for EVEX

It's likely they will share logic with the VEX patterns but we can
refactor that in the future.

* review: fix up the `mmm` bit field, document

* fix: use new `Length` name in doc tests

* review: rename `VexW` to `WBit`

* Ensure VEX instructions cannot be defined with `Length::L512`

* Use `WBit` in REX instructions

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

x64: Add conditional jumps to the new assembler (#11196)

While this doesn't remove any pseudo insts from ISLE (as was thought
might be the case with `WinchJmpIf`) this does clean up some emission
co

x64: Add conditional jumps to the new assembler (#11196)

While this doesn't remove any pseudo insts from ISLE (as was thought
might be the case with `WinchJmpIf`) this does clean up some emission
code to use some helpers and make it more clear what's happening.

show more ...

x64: Migrate indirect jmp to the new assembler (#11150)

* x64: Migrate indirect jmp to the new assembler

This migrates `JmpUnknown` from ISLE to the new assembler. Other jumps
are going to be more

x64: Migrate indirect jmp to the new assembler (#11150)

* x64: Migrate indirect jmp to the new assembler

This migrates `JmpUnknown` from ISLE to the new assembler. Other jumps
are going to be more complicated so they're deferred to later.

* Fix dead code warnings

show more ...

x64: finish migrating SSE and AVX instructions (#11149)

* asm: remove some already-converted AVX instructions

* asm: remove already-converted `vmov*` instructions

* x64: remove `vpblendw`

* x64:

x64: finish migrating SSE and AVX instructions (#11149)

* asm: remove some already-converted AVX instructions

* asm: remove already-converted `vmov*` instructions

* x64: remove `vpblendw`

* x64: remove `vpalignr`

* x64: remove `vptest` and its container, `XmmCmpRmRVex`

* x64: replace `insertps`

This adds `{v}insertps`, replacing all old uses. It also adds
`{v}extractps` in case it is needed later.

* x64: remove `SseOpcode` and `AvxOpcode`

This also converts some final constructors in winch.

* Remove dead code

* Fix unused variables

* x64: bless Cranelift filetests

show more ...

x64: migrate FMA instructions (#11143)

* asm: add `fma` instructions

This includes all size (`ss|sd|ps`) and order (`132|213|231`) variants
of:
- `vfma*`
- `vfnma*`
- `vfsub*`
- `vfnsub*`

* x64: r

x64: migrate FMA instructions (#11143)

* asm: add `fma` instructions

This includes all size (`ss|sd|ps`) and order (`132|213|231`) variants
of:
- `vfma*`
- `vfnma*`
- `vfsub*`
- `vfnsub*`

* x64: replace `fma` instructions with their assembler versions

In doing so, this also removes the `Inst::XmmRmRVex3` variant.

* x64: bless Cranelift filetests

* clippy: fix unused variable

show more ...

Add shuf and pshuf variants to the new assembler (#11135)

Add ptest and vptest to the new assembler (#11139)

Add reciprocal instructions to new assembler (#11138)

Add pblendw and vpblendw to the new assembler (#11137)

x64: migrate many AVX instructions (#11115)

* asm: add all AVX instructions in the `rmi -> r` family (`XmmRmiRVex`)

This involves quite a few instructions, but this change should only
really add AV

x64: migrate many AVX instructions (#11115)

* asm: add all AVX instructions in the `rmi -> r` family (`XmmRmiRVex`)

This involves quite a few instructions, but this change should only
really add AVX versions of pre-existing instructions.

* x64: convert `xmm_rmir_vex` instructions

The only remaining instructions to remove `Inst::XmmRmiRVex` are:
`vpmaddwd`, `vpmaddubsw`, and `vpshufb`.

* x64: bless Cranelift filetests

* winch: convert `xmm_rmir_vex` instructions

* winch: bless Winch disassembly tests

* cranelift: remove now-missing instructions from emit tests

* winch: use original lowering to generate all 1s

* x64: fix AVX choice for `pabs*`

* x64: fix AVX choice for `psrlq`

show more ...

Add pmadd variants to the new assembler (#11131)

x64: Add lowerings for the `bt` instruction (#11128)

* x64: Add lowerings for the `bt` instruction

This commit builds on the previous refactorings to leverage the `bt`
instruction on x64 which move

x64: Add lowerings for the `bt` instruction (#11128)

* x64: Add lowerings for the `bt` instruction

This commit builds on the previous refactorings to leverage the `bt`
instruction on x64 which moves a specific bit into the CF bit of EFLAGS
which can be useful for comparisons/branches/selects/etc.

* Update cranelift/codegen/src/isa/x64/lower/isle.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

* Update comments

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

x64: Migrate `setcc` instructions to the new assembler (#11112)

These instructions were relatively straightforward and the notable parts
here are:

* The manual doesn't have `/0` but these instructi

x64: Migrate `setcc` instructions to the new assembler (#11112)

These instructions were relatively straightforward and the notable parts
here are:

* The manual doesn't have `/0` but these instructions use `.digit(0)`,
but the manual also says the reg bits are ignored in the ModR/M byte
so it doesn't actually matter what we put in these bits, just need to
put something in there.

* The previous implementation in `emit.rs` unconditionally emitted a REX
byte but this was not necessary for some registers, so many binary
offsets have changed in tests as some unnecessary REX bytes are now
omitted.

* As with `cmov*` instructions in #11095 some mnemonics that Cranelift
uses differ from the canonical ones that Capstone disassembles, so
there are some naming discrepancies which are documented inline as
well.

show more ...

x64: Migrate callq to the new assembler (#11093)

* x64: Migrate callq to the new assembler

No ISLE rules for this one, just some updates to `emit.rs`
pseudo-instructions to use symbolic names inste

x64: Migrate callq to the new assembler (#11093)

* x64: Migrate callq to the new assembler

No ISLE rules for this one, just some updates to `emit.rs`
pseudo-instructions to use symbolic names instead of raw bytes.

* Fix emission of stack maps

show more ...

x64: Migrate `cmov*` to the new assembler (#11095)

This commit adds all `cmov*` variants from the Intel manual to the new
assembler. This then additionally removes the `Cmove` pseudo-inst in
favor o

x64: Migrate `cmov*` to the new assembler (#11095)

This commit adds all `cmov*` variants from the Intel manual to the new
assembler. This then additionally removes the `Cmove` pseudo-inst in
favor of these new instructions. One difference from before is that the
naming in the `CC` enum does not exactly match what mnemonics Capstone
uses to disassemble. For example `CC.NB` in ISLE corresponds to the
Intel instruction `CMOVNB`. This instruction, however, has the same
encoding as `CMOVAE` and Capstone disassembles as `CMOVAE`. This means
that the instruction selection in ISLE isn't a 1:1 match with mnemonics.

This additionally adds support in the assembler ISLE generation to
understand that instructions which read EFLAGS generate a
`ConsumesFlags` variant in their instruction helpers.

show more ...

x64: convert `vpextr*` instructions (#11094)

This replaces all AVX versions of XMM lane extraction to use the new
assembler. This removes the `Inst::XmmMovRMImmVex` and
`Inst::XmmToGprImmVex` varian

x64: convert `vpextr*` instructions (#11094)

This replaces all AVX versions of XMM lane extraction to use the new
assembler. This removes the `Inst::XmmMovRMImmVex` and
`Inst::XmmToGprImmVex` variants.

show more ...

Adds palignr and vpalignr to the new assembler (#11082)

x64: Migrate `lea` to the new assembler (#11084)

* x64: Add a shrink-before-emit optimization

This commit adds a helper function to attempt to shrink a function just
before emission if possible ba

x64: Migrate `lea` to the new assembler (#11084)

* x64: Add a shrink-before-emit optimization

This commit adds a helper function to attempt to shrink a function just
before emission if possible based on the results of register allocation.
While many instructions can be selected ahead of time (e.g. instructions
with smaller immediate widths) some are only possible after register
allocation has happened. For example the `AND RAX, imm32` instruction is
never generated in ISLE because it doesn't make sense to constrain a
register to only `RAX`, but if register allocation happened to put
`AND r/m64, imm32` into `RAX` then it's possible to use the more compact
encoding instead.

This function updates the emission of external instructions to, just
before emission, attempt to pattern-match these instructions and shrink
to a smaller instruction. This is done for a number of instructions
which support a smaller encoding when the operands are `RAX` and an
immediate. In the future my thinking was to use this additionally for
the optimization to change `LEA` instructions into `ADD` instructions if
the src/destination are equal and the addressing mode matches.

* x64: Migrate `lea` to the new assembler

Adding `lea` is easy enough but preserving the optimization of
using `add` instead of `lea` required a bit of finesse. This hooks into
the `emit_maybe_shrink` added previously for implementing this.

* Fix test build

show more ...

x64: convert unary VEX instructions (#11088)

* asm: cover more of SSE in `Features::is_sse()`

* asm: add `VexPrefix::two_op`

This is simply a convenience method when `vvvv` is unspecified.

* asm:

x64: convert unary VEX instructions (#11088)

* asm: cover more of SSE in `Features::is_sse()`

* asm: add `VexPrefix::two_op`

This is simply a convenience method when `vvvv` is unspecified.

* asm: print failing instruction in SSE-to-AVX alternate mapping

* asm: fill out AVX functionality for unary VEX instructions

These include `vmov*`, `vpmov*`, `vsqrt`, and some lane operations.

* asm: add `Location::xmm_m8`

* asm: add av2 feature

* asm: add AVX broadcast instructions

* x64: add `is_mem` for extracting addresses from `XmmMem`

* x64: convert unary VEX instructions

* x64: bless Cranelift filetests

* x64: use alternate AVX version for `pabs*`

* x64: bless `pabs*` Cranelift filetests

* x64: remove `XmmUnaryRmRVex` and now-unnecessary `AvxOpcode`s

* winch: convert `XmmUnaryRmRVex` instructions

show more ...

x64: Optimize codegen of shift/rotate-by-1 (#11083)

There are custom encodings of shift/rotate instructions when the amount
being shifted/rotated by is 1, and these encodings are slightly smaller
th

x64: Optimize codegen of shift/rotate-by-1 (#11083)

There are custom encodings of shift/rotate instructions when the amount
being shifted/rotated by is 1, and these encodings are slightly smaller
than the default encodings. This commit updates instruction selection to
use these encodings when shifts-by-immediate are encountered.

Note that these new instructions in the assembler all have custom
`Display` implementations. The Capstone disassembly is a bit odd where
the immediate is printed as `$1` instead of `$0x1` and additionally the
immediate isn't printed at all when the thing-to-be-shifted is a memory
location.

show more ...

Add pack convert (*packss*, *packus*) instructions to new assembler (#11080)

* Add pack convert (*packss*, *packus*) instructions to new assembler

* Use the or_avx ISLE rule for pack convert instru

Add pack convert (*packss*, *packus*) instructions to new assembler (#11080)

* Add pack convert (*packss*, *packus*) instructions to new assembler

* Use the or_avx ISLE rule for pack convert instructions

show more ...

x64: Migrate cmp/test to the new assembler (#11072)

* x64: Migrate cmp/test to the new assembler

This commit migrates the `cmp` and `test` family of instructions, the
`CmpRmiR` variant in ISLE, to

x64: Migrate cmp/test to the new assembler (#11072)

* x64: Migrate cmp/test to the new assembler

This commit migrates the `cmp` and `test` family of instructions, the
`CmpRmiR` variant in ISLE, to the new assembler. This required
fiddling with various helpers for immediates and such throughout a few
locations and refactoring various callsites of creating these
instructions to fit nicely into these new idioms. EFLAGS-writing
instructions with no other results are now also modeled in the assembler
as `ProducesFlags` rather than `SideEffectNoResult` as their base case
enabling using the generated constructors a bit more too.

No functional change is expected from this, it should just be internal
refactoring.

* Swap between rm and mr encodings

show more ...