S390x: emit new instructions added in z17 (#12319)

* s390x: Emit instructions from MIE4 & VXRS_EXT3 on z17

This emits & tests a bunch of instructions:
* from Miscellaneous-Instruction-Extensions

S390x: emit new instructions added in z17 (#12319)

* s390x: Emit instructions from MIE4 & VXRS_EXT3 on z17

This emits & tests a bunch of instructions:
* from Miscellaneous-Instruction-Extensions Facility 4:
* CLZ, 64bit
* CTZ, 64bit
* from Vector-Enhancements Facility 3:
* 32x4, 64x2 & 128x1 variants of the following:
* Divide
* Remainder
* 64x2 & 128x1 multiply variants
* 128x1 vaiants of:
* Compare
* CLZ
* CTZ
* Max
* Min
* Average
* Negation
* Evaluate

Co-authored-by: Jimmy Brisson <[email protected]>

* s390x: Emit vector blend on z17

* Rename x86_blendv to blendv

Now that s390x implements blendv as well, we should refer to
the instruction without the x86 prefix.

---------

Co-authored-by: Ulrich Weigand <[email protected]>

show more ...

cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

Cranelift: add debug tag infrastructure. (#11768)

* Cranelift: add debug tag infrastructure.

This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCo

Cranelift: add debug tag infrastructure. (#11768)

* Cranelift: add debug tag infrastructure.

This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCode instructions and as metadata on
the produced compiled code. It also adds opaque descriptor blobs
carried with stackslots. Together, these two features allow decorating
IR with first-class debug instrumentation that is properly preserved
by the compiler, including across optimizations and
inlining. (Wasmtime's use of these features will come in followup
PRs.)

The key idea of a "debug tag" is to allow the Cranelift embedder to
express whatever information it needs to, in a format that is opaque
to Cranelift itself, except for the parts that need translation during
lowering. In particular, the `DebugTag::StackSlot` variant gets
translated to a physical offset into the stackframe in the compiled
metadata output. So, for example, the embedder can emit a tag
referring to a stackslot, and another describing an offset in that
stackslot.

The debug tags exist as a *sequence* on any given instruction; the
meaning of the sequence is known only to the embedder, *except* that
during inlining, the tags for the inlining call instruction are
prepended to the tags of inlined instructions. In this way, a
canonical use-case of tags as describing original source-language
frames can preserve the source-language view even when multiple
functions are inlined into one.

The descriptor on a stackslot may look a little odd at first, but its
purpose is to allow serializing some description of
stackslot-contained runtime user-program data, in a way that is firmly
attached to the stackslot. In particular, in the face of inlining,
this descriptor is copied into the inlining (parent) function from the
inlined function when the stackslot entity is copied; no other
metadata outside Cranelift needs to track the identity of stackslots
and know about that motion. This fits nicely with the ability of tags
to refer to stackslots; together, the embedder can annotate
instructions as having certain state in stackslots, and describe the
format of that state per stackslot.

This infrastructure is tested with some compile-tests now;
testing of the interpretation of the metadata output will come with
end-to-end debug instrumentation tests in a followup PR.

* Review feedback: add back sequence points and enforce tags only on sequence points or calls.

* Use Vecs for debug metadata in MachBuffer to avoid SmallVec size penalty in not-used case.

* Review feedback: switch from inlined stackslot descriptor blobs to u64 keys.

show more ...

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conver

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conversions for hashing the TargetIsa

* Remove func_body_len

It is identical to buffer.data.len()

* Introduce IsaFlagsHashKey

show more ...

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as wel

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as well.

* Update dependency directive in `Cargo.toml`

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Switch to using Wasmtime-style builtins for ceil, floor, etc. (#10657)

* Switch to using Wasmtime-style builtins for ceil, floor, etc.

With this patch, all emitted calls are Wasmtime-style builtins

Switch to using Wasmtime-style builtins for ceil, floor, etc. (#10657)

* Switch to using Wasmtime-style builtins for ceil, floor, etc.

With this patch, all emitted calls are Wasmtime-style builtins, rather
than Cranelift-style libcalls. This ensures that all calls from
Cranelift-generated code into Wasmtime host code use the same mechanism,
and eliminates the relocation handling code for the libcall mechanism.

* Update tests.

* Avoid using x86_pshufb on non-x86 platforms.

* Revert unneeded f32/f64 changes in Pulley.

* Define i8x16 as an unconstructible type if sse isn't available.

* Delete the setters too.

* Fix f32/f64 setters.

* Test with prtest:full.

prtest:full

* Support fma.

* Return true for `has_native_fma` on pulley.

This works because pulley already has code implementing fma. This avoids
needing to marshal f32x4 values into builtin function calls on pulley.

* Update tests.

show more ...

Fix argument extension on riscv64 for Wasmtime builtins (#10069)

* Fix argument extension on riscv64 for Wasmtime builtins

This commit fixes a crash found in the CI of #10040. That PR itself
isn't

Fix argument extension on riscv64 for Wasmtime builtins (#10069)

* Fix argument extension on riscv64 for Wasmtime builtins

This commit fixes a crash found in the CI of #10040. That PR itself
isn't the fault per-se but rather uncovered a preexisting issue on
riscv64. According to riscv64's ABI docs it looks like arguments are all
expected to be sign-extended, whereas currently in Wasmtime all host
signatures are zero-extended on all platforms. This commit applies two
changes to fix this:

* A new `TargetIsa::default_argument_extension` method was added which
is now used instead of unconditionally using `uext` to apply to all
arguments/results.

* While I was here I went ahead and split apart things where the wasm
signature of a builtin doesn't use `uext` or `sext`. The host
signature still does, however, which means that any extension
necessary happens in the trampoline we generate per-libcall as opposed
to at all callsites.

I'm not certain that all platforms require `uext` but I've left the
`TargetIsa` implementation as `uext` for now with a comment explaining
why. Currently the only non-`uext` platforms are riscv64, which is
`sext` to fix the issue from #10040, and Pulley which is "none" as
things work differently there.

* Update test expectations

show more ...

pulley: Refactor register restores on tail-calls (#10022)

* pulley: Refactor register restores on tail-calls

Use helpers from `abi.rs` where possible to avoid duplicating logic.

* Shuffle where st

pulley: Refactor register restores on tail-calls (#10022)

* pulley: Refactor register restores on tail-calls

Use helpers from `abi.rs` where possible to avoid duplicating logic.

* Shuffle where stack increments from tail args happens

show more ...

[DWARF] Add logging to range building (#9969)

pulley: Add more addressing modes for loads/stores (#9994)

* pulley: Add more addressing modes for loads/stores

This commit adds a new "g32" addressing mode to Pulley that matches the
pattern emitt

pulley: Add more addressing modes for loads/stores (#9994)

* pulley: Add more addressing modes for loads/stores

This commit adds a new "g32" addressing mode to Pulley that matches the
pattern emitted by Cranelift for 32-bit wasm guests running on hosts.
The general idea here is that this addressing mode encompasses an
addition of a host-width value to a zero-extended (optionally) 32-bit
value. On 32-bit hosts there's no zero-extension but on 64-bit hosts
there's a zero-extension. The wasm address is always 32-bits though
which enables using a single instruction for both 32 and 64-bit hosts.

New "g32" loads and stores are added to Pulley with varying sizes and
options according to what seems to be common in wasm. The `disas` test
suite was updated to showcase using these instructions for wasm
loads/stores on 32 and 64-bit hosts.

An additional change in this commit is to deduplicate the 32/64-bit
bounds-check macro-ops. The trick in this commit works for those as well
meaning that only a single instruction is needed instead of
one-per-host-pointer-width. Additionally the load of the bound from the
`VMContext` is folded into the bounds check itself as it was found that
this was always present anyway before the bounds check.

Overall this shrinks the size of `spidermonkey.cwasm` from 21M to 20M
and the runtime of `pulldown-cmark`, `bz2`, and `spidermonkey` on
Sightglass have all been reduced by 10%. Not as big wins as I was hoping
for but alas.

* Fix debug assertions on 32-bit

show more ...

pulley: Implement float arithmetic operations (#9808)

* pulley: Implement float arithmetic operations

Fill out enough to get `f32.wast` and `f64.wast` spec tests working. A
minor ABI issue was disc

pulley: Implement float arithmetic operations (#9808)

* pulley: Implement float arithmetic operations

Fill out enough to get `f32.wast` and `f64.wast` spec tests working. A
minor ABI issue was discovered along the way which is also required to
get a new test working on both 32 and 64-bit platforms.

cc #9783

* Centralize handling of float operations

Add a new `wasmtime-math` crate which is tasked with providing the float
math functions needed by Wasm with with, in theory, most optimal
platform implementation available.

* Fix riscv64 tests

show more ...

pulley: Implement more of loads/stores (#9775)

* pulley: Implement more of loads/stores

This commit gets the `address.wast` spec test working by filling out
more load/store infrastructure in Pulley

pulley: Implement more of loads/stores (#9775)

* pulley: Implement more of loads/stores

This commit gets the `address.wast` spec test working by filling out
more load/store infrastructure in Pulley. In doing so I've done a
refactoring of the existing load/store methods. Changes here are:

* All load/stores are `*_offset32` now instead of optionally having no
offset, a 8-bit offset, or a 64-bit offset.
* All x-register loads/stores are prefixed with `x` now.
* All loads/stores have "le" for little-endian in their name.
* Loads/stores are refactored to have 8 and 16-bit variants.
* Sign-extending loads now either extend to 32 or 64 depending on the
opcode.
* Float loads/stores are added.
* Big-endian is handled with explicit big-endian loads/stores instead
`bswap` to handle this transparently in the backend (e.g. for stores
not ISLE-generated) and to handle floats.

* Remove pulley interpreter fuzz target

This is a bit onerous to keep updated and is probably best subsumed by
the fuzzing support we have in general for wasm.

* Update pulley tests

* Fixes from a rebase

* Review comments

* Update test expectations

show more ...

pulley: Support big-endian targets (#9759)

* pulley: Support big-endian targets

This commit fixes the Pulley interpreter and Cranelift backend to
properly support big-endian targets. The problem be

pulley: Support big-endian targets (#9759)

* pulley: Support big-endian targets

This commit fixes the Pulley interpreter and Cranelift backend to
properly support big-endian targets. The problem beforehand was that
loads/stores in Pulley were defined as the native endianness which means
that big and little-endian platforms differed. Additionally the
previously set of understood pulley targets were implicitly always
little-endian which was not appropriate for a big-endian host where JIT
data structures are stored in big-endian format.

This commit fixes all of these issues with a few changes:

* Pulley loads/stores are always little-endian now.
* Pulley now has bswap32/bswap64 instructions.
* Wasmtime/Cranelift now understand big-endian pulley targets (e.g.
`pulley32be`).
* CLIF translation of loads/stores now properly handles the endianness
flags on `MemFlags`.

In the future if necessary we could natively add a macro-op for
big-endian loads/stores to Pulley but for now it's more minimal to just
have `bswap32` and `bswap64`.

The result of this commit is that Pulley tests are now run and executed
on s390x like all other platforms.

* Fix fuzz build

* Review comments

* Fix fuzz build

show more ...

Fix `cranelift-codegen` build script for Pulley backends (#9172)

* Fix `cranelift-codegen` build script for Pulley backends

The cargo feature is called "pulley" for both the pulley32 and pulley64
b

Fix `cranelift-codegen` build script for Pulley backends (#9172)

* Fix `cranelift-codegen` build script for Pulley backends

The cargo feature is called "pulley" for both the pulley32 and pulley64
backends, since they are the same code, but the build script was checking for
cargo features named "pulley32" and "pulley64" instead.

* Fix warnings in build

show more ...

Cranelift: Add a new backend for emitting Pulley bytecode (#9089)

* Cranelift: Add a new backend for emitting Pulley bytecode

This commit adds two new backends for Cranelift that emits 32- and 64-b

Cranelift: Add a new backend for emitting Pulley bytecode (#9089)

* Cranelift: Add a new backend for emitting Pulley bytecode

This commit adds two new backends for Cranelift that emits 32- and 64-bit Pulley
bytecode. The backends are both actually the same, with a common implementation
living in `cranelift/codegen/src/isa/pulley_shared`. Each backend configures an
ISA flag that determines the pointer size, and lowering inspects this flag's
value when lowering memory accesses.

To avoid multiple ISLE compilation units, and to avoid compiling duplicate
copies of Pulley's generated `MInst`, I couldn't use `MInst` as the `MachInst`
implementation directly. Instead, there is an `InstAndKind` type that is a
newtype over the generated `MInst` but which also carries a phantom type
parameter that implements the `PulleyTargetKind` trait. There are two
implementations of this trait, a 32- and 64-bit version. This is necessary
because there are various static trait methods for the mach backend which we
must implement, and which return the pointer width, but don't have access to any
`self`. Therefore, we are forced to monomorphize some amount of code. This type
parameter is fairly infectious, and all the "big" backend
types (`PulleyBackend<P>`, `PulleyABICallSite<P>`, etc...) are parameterized
over it. Nonetheless, not everything is parameterized over a `PulleyTargetKind`,
and we manage to avoid duplicate `MInst` definitions and lowering code.

Note that many methods are still stubbed out with `todo!`s. It is expected that
we will fill in those implementations as the work on Pulley progresses.

* Trust the `pulley-interpreter` crate, as it is part of our workspace

* fix some clippy warnings

* Fix a dead-code warning from inside generated code

* Use a helper for emitting br_if+comparison instructions

* Add a helper for converting `Reg` to `pulley_interpreter::XReg`

* Add version to pulley workspace dependency

* search the pulley directory for crates in the publish script

show more ...