cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

pulley: Implement support for `symbol_value` (#11576)

This commit fills out the Pulley lowerings of the `symbol_value` and
`func_addr` CLIF instructions. Additionally handling of relocations on
Pull

pulley: Implement support for `symbol_value` (#11576)

This commit fills out the Pulley lowerings of the `symbol_value` and
`func_addr` CLIF instructions. Additionally handling of relocations on
Pulley is improved to be more "formal" as opposed to just blindly using
an x64 relocation and assuming it works out. The intention here is to
make Pulley behave more similarly to other platforms in all these
respects while also enabling usage of Wasmtime to eventually use
`symbol_addr` to calculate an address relative to the current PC
(similar to x64).

This includes Cranelift golden tests as well as more filetests being
run, but the actual integration into Wasmtime will be deferred to a
future commit.

show more ...

Enable the `useless_conversion` Clippy lint (#10838)

* Enable the `useless_conversion` Clippy lint

We've got lots of types in Wasmtime and convert between them quite a
lot, but often over time conv

Enable the `useless_conversion` Clippy lint (#10838)

* Enable the `useless_conversion` Clippy lint

We've got lots of types in Wasmtime and convert between them quite a
lot, but often over time conversions become unnecessary through
refactorings or similar. This will hopefully enable us to clean up some
conversions as they come up to try to have as few as possible ideally.

* Review comments

show more ...

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stor

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stores are translated to Pulley opcodes when using the
interpreter. Additionally the functionality related to memory support
has changed quite a bit with the interpreter as well. This is all based
off comments on #10102 with the end goal of folding the two Pulley
opcodes today of "do the bounds check" and "do the load" into one
opcode. This is intended to reduce the number of opcodes and overall
improve interpreter throughput by minimizing turns of the interpreter
loop.

The basic idea behind this PR is that a new basic suite of loads/stores
are added to Pulley which trap if the address is zero. This provides a
route to translate trapping loads/stores in CLIF to Pulley bytecode
without actually causing segfaults at runtime. WebAssembly translation
to CLIF is then updated to use the `select` trick for wasm loads/stores
where either 0 is loaded from or the actual address is loaded from.
Basic support for translation and such is added for this everywhere, and
this ensures that all loads/stores for wasm will be translated
successfully with Pulley.

The next step was to extend the "g32" addressing mode preexisting in
Pulley to support a bounds check as well. New pattern-matches were added
to ISLE to search for a bounds check in the address of a trapping
load/store. If found then the entire chain of operations necessary to
compute the address are folded into a single "g32" opcode which ends up
being a fallible load/store at runtime.

To fit all this into Pulley this commit contains a number of
refactorings to shuffle around existing opcodes related to memory and
extend various pieces of functionality here and there:

* Pulley now uses a `AddrFoo` types to represent addressing modes as a
single immediate rather than splitting it up into pieces for each
method. For example `AddrO32` represents "base + offset32". `AddrZ`
represents the same thing but traps if the address is zero. The
`AddrG32` mode represents a bounds-checked 32-bit linear memory access
on behalf of wasm.

* Pulley loads/stores were reduced to always using an `AddrFoo`
immediate. This means that the old `offset8` addressing mode was
removed without replacement here (to be added in the future if
necessary). Additionally the suite of sign-extension modes supported
were trimmed down to remove 8-to-64, 16-to-64, and 32-to-64 extensions
folded as part of the opcode. These can of course always be re-added
later but probably want to be added just for the `G32` addressing mode
as opposed to all addressing modes.

* The interpreter itself was refactored to have an `AddressingMode`
trait to ensure that all memory accesses, regardless of addressing
modes, are largely just copy/pastes of each other. In the future it
might make sense to implement these methods with a macro, but for now
it's copy/paste.

* In ISLE the `XLoad` generic instruction removed its `ext` field to
have extensions handled exclusively in ISLE instead of partly in
`emit.rs`.

* Float/vector loads/stores now have "g32" addressing (in addition to
the "z" that's required for wasm) since it was easy to add them.

* Translation of 1-byte accesses on Pulley from WebAssembly to CLIF no
longer has a special case for using `a >= b` instead of `a > b - 1` to
ensure that the same bounds-check instruction can be used for all
sizes of loads/stores.

* The bounds-check which folded a load-of-the-bound into the opcode is
now present as a "g32bne" addressing mode. with its of suite of
instructions to boo.

Overall this PR is not a 1:1 replacement of all previous opcodes with
exactly one opcode. For example loading 8 bits sign-extended to 64-bits
is now two opcodes instead of one. Additionally some previous opcodes
have expanded in size where for example the 8-bit offset mode was remove
in favor of only having 32-bit offsets. The goal of this PR is to reboot
how memory is handled in Pulley. All loads/stores now use a specific
addressing mode and currently all operations supported across addressing
modes are consistently supported. In the future it's expected that some
features will be added to some addressing modes and not others as
necessary, for example extending the "g32" addressing mode only instead
of all addressing modes.

For an evaluation of this PR:

* Code size: `spidermonkey.cwasm` file is reduced from 19M to 16M.
* Sightglass: `pulldown-cmark` is improved by 15%
* Sightglass: `bz2` is improved by 20%
* Sightglass: `spidermonkey` is improved by 22%
* Coremark: score improved by 40%

Overall this PR and new design looks to be a large win. This is all
driven by the reduction in opcodes both for compiled code size and
execution speed by minimizing turns of the interpreter loop. In the end
I'm also pretty happy with how this turned out and I think the
refactorings are well worth it.

* Use new `is_pulley` helper more

* Improve `addrz` helper, tighten up `memory-inbounds.wat` a bit

* Improve codegen in a few `memory-inbounds.wat` cases

* Fix test expectation

show more ...

fix typo (#9906)

Signed-off-by: Jack Drogon <[email protected]>

pulley: Add macro `CallN` instructions (#9874)

* pulley: Add macro `CallN` instructions

This commit adds new macro instructions to assist with speeding up calls
between functions. Pulley's previous

pulley: Add macro `CallN` instructions (#9874)

* pulley: Add macro `CallN` instructions

This commit adds new macro instructions to assist with speeding up calls
between functions. Pulley's previous `Call` instruction was similar to
native call instructions where arguments/results are implicitly in the
right location according to the ABI, but movement between registers is
more expensive with Pulley than with native architectures. The `CallN`
instructions here enable listing a few arguments (only integer
registers) in the opcode itself. This removes the need for individual `xmov`
instructions into individual registers and instead it can all be done
within the opcode handlers.

This additionally enables passing the same argument twice to a function
to reside only in one register. Finally parallel-copies between these
registers are supported as the interpreter loads all registers and then
stores all registers.

These new instructions participate in register allocation differently
from before where the first few arguments are allowed to be in any
register and no longer use `reg_fixed_use`. All other arguments (and all
float arguments for example) continue to use `reg_fixed_use`.

Locally sightglass reports this change speeding up `pulldown-cmark` by
2-10%. On a `fib(N)` micro-benchmark it didn't help as much as I was
hoping that it was going to.

* Fix MSRV

show more ...

pulley: Add branch-with-compare-against-immediate (#9863)

This commit adds a large number of new `br_if_x*` instructions which
compare with an immediate instead of comparing two registers. This is
p

pulley: Add branch-with-compare-against-immediate (#9863)

This commit adds a large number of new `br_if_x*` instructions which
compare with an immediate instead of comparing two registers. This is
pretty common in wasm/compiled code where, for example, loop upper
bounds are often constants. This helps compress code slightly while
fusing more instructions together.

The main cost of this is that the number of opcodes added here is quite
large. Like with previous immediate-taking opcodes both 8 and 32-bit
variants of immediates are added for all comparisons. Additionally
unlike the previous set of branch-and-compare instructions it's required
to add instructions for `>` and `>=` because the operands cannot be
swapped to invert the condition, further increasing the number of
opcodes added.

This is a mild size reduction on `spidermonkey.cwasm` from 29M to 28M
but it's mostly expected to be a performance win for interpreted loops.

show more ...

pulley: Refactor conditional branches and table codegen (#9794)

This commit first fixed an issue with table access codegen to disable
spectre mitigations on Pulley targets like how spectre is disabl

pulley: Refactor conditional branches and table codegen (#9794)

This commit first fixed an issue with table access codegen to disable
spectre mitigations on Pulley targets like how spectre is disabled for
memory accesses as well. This unblocked many tests related to tables
which then led to a different error about a `trapnz` with an 8-bit value
not being supported.

In fixing `trapnz` with 8-bit values this PR went ahead and did a
general-purpose refactoring for how conditional branches are managed.
Previously conditional traps and conditional branches had some
duplicated logic and the goal was to unify everything. There is now a
single `Cond` which represents the condition of a conditional jump which
is used uniformly for all locations such as `select`, `brif`, and
`trap[n]z`. This new type represents all the sorts of conditional
branches that can be done in Pulley, for example integer comparisons and
whether or not a register is zero. This `Cond` type has various helpers
for printing it, inverting it, collecting operands, emission, etc.

The end result is that it's a bit wordy to work with `Cond` right now
due to the size of the variants but all locations working with
conditional traps are deduplicated and now it's just repetitive logic
rather than duplicated logic.

Putting all of this together gets a large batch of spec tests working.

I'll note that this does remove a feature where `trapnz` was turned into
nothing or an unconditional trap if the argument was a constant, but
that feels like an optimization perhaps best left for the middle-end
rather than doing it in the backend.

cc #9783

show more ...

pulley: Implement more of loads/stores (#9775)

* pulley: Implement more of loads/stores

This commit gets the `address.wast` spec test working by filling out
more load/store infrastructure in Pulley

pulley: Implement more of loads/stores (#9775)

* pulley: Implement more of loads/stores

This commit gets the `address.wast` spec test working by filling out
more load/store infrastructure in Pulley. In doing so I've done a
refactoring of the existing load/store methods. Changes here are:

* All load/stores are `*_offset32` now instead of optionally having no
offset, a 8-bit offset, or a 64-bit offset.
* All x-register loads/stores are prefixed with `x` now.
* All loads/stores have "le" for little-endian in their name.
* Loads/stores are refactored to have 8 and 16-bit variants.
* Sign-extending loads now either extend to 32 or 64 depending on the
opcode.
* Float loads/stores are added.
* Big-endian is handled with explicit big-endian loads/stores instead
`bswap` to handle this transparently in the backend (e.g. for stores
not ISLE-generated) and to handle floats.

* Remove pulley interpreter fuzz target

This is a bit onerous to keep updated and is probably best subsumed by
the fuzzing support we have in general for wasm.

* Update pulley tests

* Fixes from a rebase

* Review comments

* Update test expectations

show more ...

Support executing Pulley in Wasmtime (#9744)

* Support executing Pulley in Wasmtime

This commit is the initial implementation of executing the Pulley
interpreter from the `wasmtime` crate. This gi

Support executing Pulley in Wasmtime (#9744)

* Support executing Pulley in Wasmtime

This commit is the initial implementation of executing the Pulley
interpreter from the `wasmtime` crate. This gives access to all of the
`wasmtime` crate's runtime APIs backed by execution of bytecode in
Pulley. This builds on the previous PRs I've been making for support in
Pulley to culminate in testing on CI in this PR. This PR handles some
final tidbits related to producing a runnable image that can be
interpreted by the `wasmtime` crate such as:

* Pulley compilation artifacts are no longer marked as natively
executable, just read-only.
* Pulley compilation artifacts include wasm-to-array trampolines like
normal platforms (removes a pulley special-case).
* Dispatch of host calls from Pulley to the Wasmtime runtime are
implemented.
* Pulley's list of panicking wasm features is slimmed down as most are
covered by "this lowering isn't supported" errors.
* Execution of wasm code now has an `if` to see whether Pulley is
enabled within a `Store` or not.
* Traps and basic "unwinding" of the pulley stack are implemented (e.g.
a "pulley version" of `setjmp` and `longjmp`, sort of)
* Halting the interpreter has been refactored to help shrink the size of
`ControlFlow<Done>` and handle metadata with each done state.

Some minor refactorings are also included here and there along with a
few fixes here and there necessary to get tests passing.

The next major part of this commit is updates to our `wast` test suite
and executing all `*.wast` files. Pulley is now executed by default for
all files as a new execution engine. This means that all platforms in CI
will start executing Pulley tests. At this time almost all tests are
flagged as "expected to fail" but there are a small handful of
allow-listed tests which are expected to pass. This exact list will
change over time as CLIF lowerings are implemented and the interpreter
is extended.

Follow-up PRs will extend the testing strategy further such as:

* Extending `#[wasmtime_test]` to include Pulley in addition to Winch.
* Getting testing set up on CI for 32-bit platforms.

prtest:full

* Fix pulley fuzz build

* Fix clippy lints

* Shuffle around some `#[cfg]`'d code

* Remove unused imports

* Update feature sets testing MIRI

Enable pulley for wasmtime/wasmtime-cli and also enable all features for
wasmtime-environ

* Round up pulley's page size to 64k

* Skip pulley tests on s390x for now

* Add a safety rail for matching a pulley target to the host

* Fix more pulley tests on s390x

* Review comments

* Fix fuzz build

show more ...

pulley: Remove virtual sp offset (#9722)

This fixes the output of the pulley cranelift backend with respect to
virtual stack offsets and adjustments for calls with stack parameters
and results. Prev

pulley: Remove virtual sp offset (#9722)

This fixes the output of the pulley cranelift backend with respect to
virtual stack offsets and adjustments for calls with stack parameters
and results. Previously the virtual stack pointer was updated but the
rest of the backend assumed the real stack pointer was updated. This
updates the pulley backend to match the other backends in this respects
and fixes a test on my "more complete pulley integration" branch.

show more ...

pulley: Implement `get_frame_pointer` (#9658)

* pulley: Implement `get_frame_pointer`

This commit implements the CLIF `get_frame_pointer` instruction which
will be needed by trampolines in Wasmtime

pulley: Implement `get_frame_pointer` (#9658)

* pulley: Implement `get_frame_pointer`

This commit implements the CLIF `get_frame_pointer` instruction which
will be needed by trampolines in Wasmtime. This is implemented by
generalizing the preexisting `GetSp` `MInst` into a "get special". This
the additionally removes the extended `get_sp` opcode in Pulley as it's
not necessary as `xmov` can operate on this register.

* Review comments

show more ...

pulley: Remove special registers as candidates from regalloc (#9666)

* pulley: Remove special registers as candidates from regalloc

This commit fixes a minor mistake where regalloc was allowed to a

pulley: Remove special registers as candidates from regalloc (#9666)

* pulley: Remove special registers as candidates from regalloc

This commit fixes a minor mistake where regalloc was allowed to allocate
"special registers" such as the stack pointer, the frame pointer, etc.
These register shouldn't participate in general-purpose register
allocation, so they're removed from the list of non-preferred registers
when metadata is provided to regalloc2.

* Assert everything before SPECIAL_START is not special

show more ...

Add support for stack limit checks in Cranelift's Pulley backend (#9238)

pulley: pack `dst`, `src1` and `src2` registers into 2 bytes (#9088)

* pulley: use enums for `{X,F,V}Reg`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulle

pulley: pack `dst`, `src1` and `src2` registers into 2 bytes (#9088)

* pulley: use enums for `{X,F,V}Reg`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulley: add `BinaryOperands`

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* pulley: use `BinaryOperands` for binary operators

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

---------

Signed-off-by: Karl Meakin <[email protected]>

show more ...

Cranelift: Add a new backend for emitting Pulley bytecode (#9089)

* Cranelift: Add a new backend for emitting Pulley bytecode

This commit adds two new backends for Cranelift that emits 32- and 64-b

Cranelift: Add a new backend for emitting Pulley bytecode (#9089)

* Cranelift: Add a new backend for emitting Pulley bytecode

This commit adds two new backends for Cranelift that emits 32- and 64-bit Pulley
bytecode. The backends are both actually the same, with a common implementation
living in `cranelift/codegen/src/isa/pulley_shared`. Each backend configures an
ISA flag that determines the pointer size, and lowering inspects this flag's
value when lowering memory accesses.

To avoid multiple ISLE compilation units, and to avoid compiling duplicate
copies of Pulley's generated `MInst`, I couldn't use `MInst` as the `MachInst`
implementation directly. Instead, there is an `InstAndKind` type that is a
newtype over the generated `MInst` but which also carries a phantom type
parameter that implements the `PulleyTargetKind` trait. There are two
implementations of this trait, a 32- and 64-bit version. This is necessary
because there are various static trait methods for the mach backend which we
must implement, and which return the pointer width, but don't have access to any
`self`. Therefore, we are forced to monomorphize some amount of code. This type
parameter is fairly infectious, and all the "big" backend
types (`PulleyBackend<P>`, `PulleyABICallSite<P>`, etc...) are parameterized
over it. Nonetheless, not everything is parameterized over a `PulleyTargetKind`,
and we manage to avoid duplicate `MInst` definitions and lowering code.

Note that many methods are still stubbed out with `todo!`s. It is expected that
we will fill in those implementations as the work on Pulley progresses.

* Trust the `pulley-interpreter` crate, as it is part of our workspace

* fix some clippy warnings

* Fix a dead-code warning from inside generated code

* Use a helper for emitting br_if+comparison instructions

* Add a helper for converting `Reg` to `pulley_interpreter::XReg`

* Add version to pulley workspace dependency

* search the pulley directory for crates in the publish script

show more ...