Use `core::convert::Infallible` instead of our own `Uninhabited` type (#12115)

* Use `core::convert::Infallible` instead of our own `Uninhabited` type

I didn't realize that the standard library alr

Use `core::convert::Infallible` instead of our own `Uninhabited` type (#12115)

* Use `core::convert::Infallible` instead of our own `Uninhabited` type

I didn't realize that the standard library already had an uninhabited type
available for us to reuse.

* Actually remove the uninhabited module and its re-exports

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Enable the `unsafe-op-in-unsafe-fn` lint (#10559)

* Enable the `unsafe-op-in-unsafe-fn` lint

This commit enables the `unsafe-op-in-unsafe-fn` lint in rustc for the
entire workspace. This lint will

Enable the `unsafe-op-in-unsafe-fn` lint (#10559)

* Enable the `unsafe-op-in-unsafe-fn` lint

This commit enables the `unsafe-op-in-unsafe-fn` lint in rustc for the
entire workspace. This lint will be warn-by-default in the 2024 edition
so this is intended to smooth the future migration to the new edition.

Many `unsafe` blocks were added in places the lint warned about, with
two major exceptions. The `wasmtime` and `wasmtime-c-api` crates simply
expect this lint to fire and effectively disable the lint. They're too
big at this time to do through this PR. My hope is that one day in the
future they'll be migrated, but more realistically that probably won't
happen so these crates just won't benefit from this lint.

* Fix nostd fiber build

prtest:full

* Fix build on Windows

* Fix asan build

show more ...

Change `allow(missing_docs)` to `expect(..)` (#10384)

This wasn't possible when `expect` was first introduced due to a change
being required in upstream rust-lang/rust. That change
rust-lang/rust#13

Change `allow(missing_docs)` to `expect(..)` (#10384)

This wasn't possible when `expect` was first introduced due to a change
being required in upstream rust-lang/rust. That change
rust-lang/rust#130025) has now rode enough trains to be in our MSRV, so
we can expect missing docs now instead of just allowing it.

show more ...

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stor

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stores are translated to Pulley opcodes when using the
interpreter. Additionally the functionality related to memory support
has changed quite a bit with the interpreter as well. This is all based
off comments on #10102 with the end goal of folding the two Pulley
opcodes today of "do the bounds check" and "do the load" into one
opcode. This is intended to reduce the number of opcodes and overall
improve interpreter throughput by minimizing turns of the interpreter
loop.

The basic idea behind this PR is that a new basic suite of loads/stores
are added to Pulley which trap if the address is zero. This provides a
route to translate trapping loads/stores in CLIF to Pulley bytecode
without actually causing segfaults at runtime. WebAssembly translation
to CLIF is then updated to use the `select` trick for wasm loads/stores
where either 0 is loaded from or the actual address is loaded from.
Basic support for translation and such is added for this everywhere, and
this ensures that all loads/stores for wasm will be translated
successfully with Pulley.

The next step was to extend the "g32" addressing mode preexisting in
Pulley to support a bounds check as well. New pattern-matches were added
to ISLE to search for a bounds check in the address of a trapping
load/store. If found then the entire chain of operations necessary to
compute the address are folded into a single "g32" opcode which ends up
being a fallible load/store at runtime.

To fit all this into Pulley this commit contains a number of
refactorings to shuffle around existing opcodes related to memory and
extend various pieces of functionality here and there:

* Pulley now uses a `AddrFoo` types to represent addressing modes as a
single immediate rather than splitting it up into pieces for each
method. For example `AddrO32` represents "base + offset32". `AddrZ`
represents the same thing but traps if the address is zero. The
`AddrG32` mode represents a bounds-checked 32-bit linear memory access
on behalf of wasm.

* Pulley loads/stores were reduced to always using an `AddrFoo`
immediate. This means that the old `offset8` addressing mode was
removed without replacement here (to be added in the future if
necessary). Additionally the suite of sign-extension modes supported
were trimmed down to remove 8-to-64, 16-to-64, and 32-to-64 extensions
folded as part of the opcode. These can of course always be re-added
later but probably want to be added just for the `G32` addressing mode
as opposed to all addressing modes.

* The interpreter itself was refactored to have an `AddressingMode`
trait to ensure that all memory accesses, regardless of addressing
modes, are largely just copy/pastes of each other. In the future it
might make sense to implement these methods with a macro, but for now
it's copy/paste.

* In ISLE the `XLoad` generic instruction removed its `ext` field to
have extensions handled exclusively in ISLE instead of partly in
`emit.rs`.

* Float/vector loads/stores now have "g32" addressing (in addition to
the "z" that's required for wasm) since it was easy to add them.

* Translation of 1-byte accesses on Pulley from WebAssembly to CLIF no
longer has a special case for using `a >= b` instead of `a > b - 1` to
ensure that the same bounds-check instruction can be used for all
sizes of loads/stores.

* The bounds-check which folded a load-of-the-bound into the opcode is
now present as a "g32bne" addressing mode. with its of suite of
instructions to boo.

Overall this PR is not a 1:1 replacement of all previous opcodes with
exactly one opcode. For example loading 8 bits sign-extended to 64-bits
is now two opcodes instead of one. Additionally some previous opcodes
have expanded in size where for example the 8-bit offset mode was remove
in favor of only having 32-bit offsets. The goal of this PR is to reboot
how memory is handled in Pulley. All loads/stores now use a specific
addressing mode and currently all operations supported across addressing
modes are consistently supported. In the future it's expected that some
features will be added to some addressing modes and not others as
necessary, for example extending the "g32" addressing mode only instead
of all addressing modes.

For an evaluation of this PR:

* Code size: `spidermonkey.cwasm` file is reduced from 19M to 16M.
* Sightglass: `pulldown-cmark` is improved by 15%
* Sightglass: `bz2` is improved by 20%
* Sightglass: `spidermonkey` is improved by 22%
* Coremark: score improved by 40%

Overall this PR and new design looks to be a large win. This is all
driven by the reduction in opcodes both for compiled code size and
execution speed by minimizing turns of the interpreter loop. In the end
I'm also pretty happy with how this turned out and I think the
refactorings are well worth it.

* Use new `is_pulley` helper more

* Improve `addrz` helper, tighten up `memory-inbounds.wat` a bit

* Improve codegen in a few `memory-inbounds.wat` cases

* Fix test expectation

show more ...

pulley: Remove `unwrap_uninhabited` helper function (#10174)

No longer needed on our MSRV any more.

Require lint reasons in `pulley-interpreter` (#10173)

* Require lint reasons in `pulley-interpreter`

Continuing work originally started in #9696

* Add more pulley #[cfg]

pulley: Shrink frame save/restore instructions (#9999)

* pulley: Shrink frame save/restore instructions

This commit shrinks the size of the `PushFrameSave` and
`PopFrameRestore` functions which are

pulley: Shrink frame save/restore instructions (#9999)

* pulley: Shrink frame save/restore instructions

This commit shrinks the size of the `PushFrameSave` and
`PopFrameRestore` functions which are used in almost all wasm functions.
Previously these instructions allowed for 32-bits of stack space in
addition to saving/restoring all 32 X-registers. In reality though it's
quite uncommon to need more than 16-bits of stack space and ABI-wise the
most commonly saved registers are the upper 16 registers of the X
register set.

This commit therefore shrinks the frame size to 16 bits and only has the
ability to save/restore the upper 16 X-registers. Note that any
clobbered registers and frame sizes are still supported, they'll just
use more pessimal encodings which aren't a single opcode. If a function
uses >64KiB of stack space though it's probably not too important what
the dispatch cost is at the beginning.

The overall result of this change is that each instruction shaves of 4
bytes (2 from the frame size and 2 from the registers being
saved/restored). This results in a 4% faster execution time on the bz2
Sightglass benchmark, ~1% on pulldown-cmark, and while it shrinks
`spidermonkey.cwasm` slightly it's not significant.

* Remove no-longer-applicable test

* Fix clippy error

* Update test expectations

show more ...

pulley: Add immediate payloads to more opcodes (#9861)

* pulley: Add immediate payloads to more opcodes

This commit adds immediate payloads to the following instructions:

* `xmul32` - `xmul32_s8`

pulley: Add immediate payloads to more opcodes (#9861)

* pulley: Add immediate payloads to more opcodes

This commit adds immediate payloads to the following instructions:

* `xmul32` - `xmul32_s8` / `xmul32_s32`
* `xmul64` - `xmul64_s8` / `xmul64_s32`
* `xband32` - `xband32_s8` / `xband32_s32`
* `xband64` - `xband64_s8` / `xband64_s32`
* `xbor32` - `xbor32_s8` / `xbor32_s32`
* `xbor64` - `xbor64_s8` / `xbor64_s32`
* `xbxor32` - `xbxor32_s8` / `xbxor32_s32`
* `xbxor64` - `xbxor64_s8` / `xbxor64_s32`
* `xshl32` - `xshl32_u6`
* `xshl64` - `xshl64_u6`
* `xshr32_u` - `xshl32_u_u6`
* `xshr64_u` - `xshl64_u_u6`
* `xshr32_s` - `xshl32_s_u6`
* `xshr64_s` - `xshl64_s_u6`

For shifts there's no need to have 32-bit immediates (or even 8-bit)
since 6 bits is enough to encode all the immediates. This means that the
6-bit immediate is packed within `BinaryOperands` as a new `U6` type.

This commit unfortunately does not shrink `spidermonkey.cwasm`
significantly beyond the prior 29M. This is nevertheless expected to be
relatively important for performance.

* Fix test expectations

show more ...

pulley: Initial scaffold of SIMD support (#9820)

* pulley: Initial scaffold of SIMD support

This commit fills out some of the initial infrastructure necessary for
supporting the SIMD proposal to W

pulley: Initial scaffold of SIMD support (#9820)

* pulley: Initial scaffold of SIMD support

This commit fills out some of the initial infrastructure necessary for
supporting the SIMD proposal to WebAssembly in the Pulley interpreter,
namely 128-bit simd. The `VRegVal` union has been filled out with
various types, endianness questions are settled, and initial
implementations of a suite of opcodes are added to get a basic set of
tests working throughout the backend.

cc #9783

* Avoid dealing with big-endian vectors

* Change wasm `global`s to store `v128` in little-endian format.
* Change pulley stack loads/stores to work with vectors in little-endian
format.

show more ...

Pulley: tail calls (#9251)

* Document host return address and extract to const

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* Replace `Continutation` with `Co

Pulley: tail calls (#9251)

* Document host return address and extract to const

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* Replace `Continutation` with `ControlFlow`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* Tail recursive interpreter loop

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* Delete visitor based interpreter implementation

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* Move pulley tail call CI step to a job that uses nightly rust

* Record the PC at which a trap occurred

Not the initial PC passed into `run`

---------

Signed-off-by: Karl Meakin <[email protected]>
Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Use `NonNull<u8>` for `pc` instead of `*mut u8` (#9274)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

Use `split_first_chunk` instead of splitting manually (#9273)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

pulley: superinstructions for pushing/popping list of registers (#9099)

* pulley: add `push` and `pop` instructions

Add `xpush{32, 64}` and `xpop{32, 64}` for pushing/popping XRegs from the stack,

pulley: superinstructions for pushing/popping list of registers (#9099)

* pulley: add `push` and `pop` instructions

Add `xpush{32, 64}` and `xpop{32, 64}` for pushing/popping XRegs from the stack,
and `push_frame`/`pop_frame` for saving/restoring LR and FP in function
prologue/epilogue.

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* cranelift-bitset: more impls for `ScalarBitset`

Implement `Arbitrary` for `ScalarBitset`.

Also implement `DoubleEndedIterator` and `ExactSizeIterator` for `Iter`.
The `pop_min` method was added to help implement `DoubleEndedIterator`,
and `pop` was renamed to `pop_max` for consistency.

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulley: add instructions for pushing/popping list of registers

Add `xpush{32,64}_many` and `xpop{32,64}_many` for pushing/popping any
combination of all 32 XRegs.

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

---------

Signed-off-by: Karl Meakin <[email protected]>

show more ...

pulley: pack `dst`, `src1` and `src2` registers into 2 bytes (#9088)

* pulley: use enums for `{X,F,V}Reg`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulle

pulley: pack `dst`, `src1` and `src2` registers into 2 bytes (#9088)

* pulley: use enums for `{X,F,V}Reg`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulley: add `BinaryOperands`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulley: use `BinaryOperands` for binary operators

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

---------

Signed-off-by: Karl Meakin <[email protected]>

show more ...

CI: Test Pulley on 32-bit and `no_std` targets (#9013)

* CI: Check that Pulley builds for `no_std` targets

* CI: Test Pulley on 32-bit targets

This adds CI testing for Pulley on the following 32-b

CI: Test Pulley on 32-bit and `no_std` targets (#9013)

* CI: Check that Pulley builds for `no_std` targets

* CI: Test Pulley on 32-bit targets

This adds CI testing for Pulley on the following 32-bit targets:

* `i686-unknown-linux-gnu`
* `armv7-unknown-linux-gnueabihf`

The previous commit added checks that Pulley builds for `no_std` targets.

Our existing test sharding means that it is already tested on a big-endian
platform (`s390x-unknown-linux-gnu`) as well as on different OSes (Linux,
Windows, and macOS).

So altogether we are now testing a pretty wide range of portability dimensions
for Pulley in CI:

* 32- vs. 64-bit architectures
* `std` vs. `no_std`
* Little- vs. big-endian
* Linux vs. Windows vs. macOS

I think our CI coverage for Pulley is in a pretty good place now!

Co-Authored-By: Alex Crichton <[email protected]>

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Introduce the `pulley-interpreter` crate (#9008)

* Introduce the `pulley-interpreter` crate

This commit is the first step towards implementing
https://github.com/bytecodealliance/rfcs/pull/35

This

Introduce the `pulley-interpreter` crate (#9008)

* Introduce the `pulley-interpreter` crate

This commit is the first step towards implementing
https://github.com/bytecodealliance/rfcs/pull/35

This commit introduces the `pulley-interpreter` crate which contains the Pulley
bytecode definition, encoder, decoder, disassembler, and interpreter.

This is still very much a work in progress! It is expected that we will tweak
encodings and bytecode definitions, that we will overhaul the interpreter (to,
for example, optionally support the unstable Rust `explicit_tail_calls`
feature), and otherwise make large changes. This is just a starting point to get
the ball rolling.

Subsequent commits and pull requests will do things like add the Cranelift
backend to produce Pulley bytecode from Wasm as well as the runtime integration
to run the Pulley interpreter inside Wasmtime.

* remove stray fn main

* Add small tests for special x registers

* Remove now-unused import

* always generate 0 pc rel offsets in arbitrary

* Add doc_auto_cfg feature for docs.rs

* enable all optional features for docs.rs

* Consolidate `BytecodeStream::{advance,get1,get2,...}` into `BytecodeStream::read`

* fix fuzz targets build

* inherit workspace lints in pulley's fuzz crate

* Merge fuzz targets into one target; fix a couple small fuzz bugs

* Add Pulley to our cargo vet config

* Add pulley as a crate to publish

* Move Pulley fuzz target into top level fuzz directory

show more ...