Cranelift: preserve_all: disallow return values. (#12447)

In #12399 a fuzzbug uncovered an issue whereby a function with
`preserve_all` and with many return values cannot be called, because
`emit_re

Cranelift: preserve_all: disallow return values. (#12447)

In #12399 a fuzzbug uncovered an issue whereby a function with
`preserve_all` and with many return values cannot be called, because
`emit_retval_loads` cannot codegen memory-to-memory moves (from on-stack
return value slots directly to spillslots) without a
temporary/clobberable register, and `preserve_all` implies no such
registers exist.

I thought about trying to support this by shuffling registers -- such a
case implies many return values, and at least some of them will be in
registers, so we might be able to temporarily spill one of those and use
it as a scratch to move other values memory-to-memory. But the
complexity doesn't seem worthwhile, and this path is not actually needed
at the moment.

Thinking more broadly, anyone using `preserve_all` for hooks meant to
minimally perturb register state will likely not want to use many return
values anyway (that defeats the point). We could allow *one* return
value, with the knowledge that this always fits in a register in our
existing ABIs, but that also feels somewhat arbitrary, and I could make
the argument that "preserve all" should really mean preserve *all*.
Someone wanting to return a value anyway from such a hook could use
indirect means (pass in a pointer to a stackslot, for example). I'm
happy to tweak this limit if others have more thoughts, however.

Fixes #12399.

show more ...

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable AB

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`.

As discussed in this week's Cranelift meeting, we've discovered a need
to generalize the `patchable_call` mechanism and corresponding
`patchable` ABI slightly. In particular, we will need patchable
`try_call` callsites as well in order to allow breakpoint handlers to
throw exceptions (desirable functionality eventually) and have this work
in the presence of inlining. Also, it's just a nice generalization to
say that patchability is an orthogonal dimension to the call ABI and the
other restrictions we initially imposed, and works as long as the basic
requirement (no return values) is met.

This also renames the `patchable` ABI to `preserve_all`, to make it
clear that its purpose is actually orthogonal, and it can be used
independently of patchable callsites. It also deletes the `cold` ABI,
which never actually did anything and is misleading in the presence of
an actual cold-ish (subzero temperature, actually) ABI like
`preserve_all`.

* Review feedback.

show more ...

Cranelift: add a "patchable call" ABI. (#12061)

This ABI is intended for use in scenarios where we want a very
lightweight callsite that can be turned on and off by patching in one
instruction. (The

Cranelift: add a "patchable call" ABI. (#12061)

This ABI is intended for use in scenarios where we want a very
lightweight callsite that can be turned on and off by patching in one
instruction. (The actual patchable call instruction is not in this PR;
that will be a separate PR.)

The idea is that we define a call to clobber *no* registers -- not
even the arguments! And we restrict signatures such that on all of our
supported architectures, all arguments go into registers only. Those
two requirements together mean that all callsites for this ABI should
have only a raw call instruction, with no loads/stores to stackslots;
and have the minimum possible impact on regalloc, by only imposing
constraints on args to ensure they are in certain registers but not
altering those registers.

Given this, we could implement, e.g., breakpoints with patchable
callsites (off by default) at every sequence point in compiled
code. In a typical use-case with Wasmtime-compiled Wasm, that would
put a bunch of uses of vmctx constrained to the first argument
register in every code path, but vmctx likely already sits there most
of the time anyway (for any call to other Wasm functions or for
libcalls). Thus, the impact is just the one instruction and nothing
else.

This PR adds the calling convention itself and tests that show
that *two* consecutive callsites can be compiled with no register
setup re-occurring from one call to the next (thus demonstrating no
clobbers).

show more ...

Replace setjmp/longjmp usage in Wasmtime (#11592)

Since Wasmtime's inception it's used the `setjmp` and `longjmp`
functions in C to implement handling of traps. While this solution was
easy to imple

Replace setjmp/longjmp usage in Wasmtime (#11592)

Since Wasmtime's inception it's used the `setjmp` and `longjmp`
functions in C to implement handling of traps. While this solution was
easy to implement, relatively portable, and performant enough, there are
a number of downsides that have evolved over time to make this an
unattractive approach in the long run:

* Using `setjmp` fundamentally requires using C because Rust does not
understand a function that returns twice. It's fundamentally unsound
to invoke `setjmp` in Rust meaning that Wasmtime has forever needed a
C compiler configured and set up to build. This notably means that
`cargo check` cannot check other targets easily.

* Using `longjmp` means that Rust function frames are unwound on the
stack without running destructors. This is a dangerous operation of
which we get no protection from the compiler about. Both frames
entering wasm and frames exiting wasm are all skipped. Absolutely
minimizing this has been beneficial for portability to platforms such
as Pulley.

* Currently the no_std implementation of Wasmtime requires embedders to
provide `wasmtime_{setjmp,longjmp}` which is a thorn in the side of
what is otherwise a mostly entirely independent implementation of
Wasmtime.

* There is a performance floor to using `setjmp` and `longjmp`. Calling
`setjmp` requires using C but Wasmtime is otherwise written in Rust
meaning that there's a Rust->C->Rust->Wasm boundary which
fundamentally can't be inlined without cross-language LTO which is
difficult to configure.

* With the implementation of the WebAssembly exceptions proposal
Wasmtime now has two means of unwinding the stack. Ideally Wasmtime
would only have one, and the more general one is the method of
exceptions.

* Jumping out of a signal handler on Unix is tricky business. While
we've made it work it's generally most robust of the signal handler
simply returns which it now does.

With all of that in mind the purpose of this commit is to replace the
setjmp/longjmp mechanism of handling traps with the recently implemented
support for exceptions in Cranelift. That is intended to resolve all of
the above points in one swoop.

One point in particular though that's nice about setjmp/longjmp is that
unwinding the stack on a trap is an O(1) operation. For situations such
as stack overflow that's a particularly nice property to have as we can
guarantee embedders that traps are a constant time (albeit somewhat
expensive with signals) operation. Exceptions naively require unwinding
the entire stack, and although frame pointers mean we're just traversing
a linked list I wanted to preserve the O(1) property here nonetheless.
To achieve this a solution is implemented where the array-to-wasm
(host-to-wasm) trampolines setup state in `VMStoreContext` so looking up
the current trap handler frame is an O(1) operation. Namely the sp/fp/pc
values for a `Handler` are stored inline.

Implementing this feature required supporting
relocations-to-offsets-in-functions which was not previously supported
by Wasmtime. This required Cranelift refactorings such as #11570, #11585,
and #11576. This then additionally required some more refactoring in
this commit which was difficult to split out as it otherwise wouldn't be
tested.

Apart from the relocation-related business much of this change is about
updating the platform signal handlers to use exceptions instead of
longjmp to return. For example on Unix this means updating the
`ucontext_t` with register values that the handler specifies. Windows
involves updating similar contexts, and macOS mach ports ended up not
needing too many changes.

In terms of overall performance the relevant benchmark from this
repository, compared to before this commit, is:

sync/no-hook/core - host-to-wasm - typed - nop
time: [10.552 ns 10.561 ns 10.571 ns]
change: [−7.5238% −7.4011% −7.2786%] (p = 0.00 < 0.05)
Performance has improved.

Closes #3927
cc #10923

prtest:full

show more ...

Clarify docs around exceptions in Cranelift (#11601)

* Clarify docs around exceptions in Cranelift

This is a result of today's Cranelift meeting with some of my questions
around the ABI bits here a

Clarify docs around exceptions in Cranelift (#11601)

* Clarify docs around exceptions in Cranelift

This is a result of today's Cranelift meeting with some of my questions
around the ABI bits here and there. Notably:

* Cranelift is audited and now documented to always consider the callee
calling convention in `try_call`, disregarding the caller calling convention.

* Wasmtime's exception throw now explicitly names the `tailcc` in the
name to indicate that it's only compatible with the tail calling convention.

* The verifier test for exceptions is expanded with a few more cases
here and there that I could think of.

* Backends now assert that the size of the calling-convention list of
payload types is the same as the size of the list of registers the
backend places results into.

* Fix a typo

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Some fixes for try_call (#10593)

* Fix cranelift-frontend handling of try_call

* Implement eliminate_unreachable_code for exception tables

* Ensure try_call is considered a memory fence

* Don't e

Some fixes for try_call (#10593)

* Fix cranelift-frontend handling of try_call

* Implement eliminate_unreachable_code for exception tables

* Ensure try_call is considered a memory fence

* Don't error on try_call in the verifier if no TargetIsa is passed

* Don't clobber all registers for try_call unless the tail call conv is used

This way other consumers of Cranelift don't have to pay the cost of the
way Wasmtime will implement unwinding on exceptions.

* Allow SystemV call conv with try_call

show more ...

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect`

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect` instructions, and
lowerings on four of five ISAs (x86-64, aarch64, riscv64, pulley; s390x
has its own non-shared ABI code that will need separate work).

It extends CLIF to support these instructions as new kinds of branches,
and extends block-calls to accept `retN` and `exnN` block-call args that
carry the normal return values or exception payloads (respectively) into
the appropriate successor blocks.

It wires up the "normal return path" so that it continues to work.
It updates the ABI so that unwinding is possible without an initial
register state at throw: specifically, as per our RFC, all registers are
clobbered. It also includes metadata in the `MachBuffer` that describes
exception-catch destinations. However, no unwinder exists to interpret
these catch-destinations yet, so they are untested.

* Add try_call_indirect lowering as well.

show more ...

Couple of cleanups to the ABI computation (#9253)

* Remove WasmtimeSystemV call conv

It is no longer used by Wasmtime

* Remove CallConv::extends_* methods

They all match only a single call conv a

Couple of cleanups to the ABI computation (#9253)

* Remove WasmtimeSystemV call conv

It is no longer used by Wasmtime

* Remove CallConv::extends_* methods

They all match only a single call conv anyway.

* Avoid isa:: prefix for referencing CallConv where possible

* Update callconv list in the clif ir reference

* Use push_non_formal where expected

show more ...

cranelift: Add a winch calling convention (#8056)

* Add a winch calling convention to cranelift

* Add some tests to exercise the winch calling convention

* Enable fuzzing for the winch calling con

cranelift: Add a winch calling convention (#8056)

* Add a winch calling convention to cranelift

* Add some tests to exercise the winch calling convention

* Enable fuzzing for the winch calling convention

* Share the list of clobbers with the tail calling convention

* Fix the doc build

show more ...

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parall

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parallel to `serde_derive`, allowing it to finish compilation possibly
even before `serde_derive`, unblocking all the crates waiting for
`serde` to start compiling much sooner.

As it turns out the main deciding factor for how long the compile time of a
project is, is primarly determined by the depth of dependencies rather
than the width. In other words, a crate's compile times aren't affected
by how many crates it depends on, but rather by the longest chain of
dependencies that it needs to wait on. In many cases `serde` is part of
that long chain, as it is part of a long chain if the `derive` feature
is active:

`proc-macro2` compile build script > `proc-macro2` run build script >
`proc-macro2` > `quote` > `syn` > `serde_derive` > `serde` >
`serde_json` (or any crate that depends on serde)

By decoupling it from `serde_derive`, the chain is shortened and compile
times get much better.

Check this issue for a deeper elaboration:
https://github.com/serde-rs/serde/issues/2584

For `wasmtime` I'm seeing a reduction from 24.75s to 22.45s when
compiling in `release` mode. This is because wasmtime through `gimli`
has a dependency on `indexmap` which can only start compiling when
`serde` is finished, which you want to happen as early as possible so
some of wasmtime's dependencies can start compiling.

To measure the full effect, the dependencies can't by themselves
activate the `derive` feature. I've upstreamed a patch for
`fxprof-processed-profile` which was the only dependency that activated
it for `wasmtime` (not yet published to crates.io). `wasmtime-cli` and
co. may need patches for their dependencies to see a similar
improvement.

show more ...

Remove Wasmtime ABIs from Cranelift (#6649)

* Remove Wasmtime ABIs from Cranelift

This commit removes the `Wasmtime*` family of ABIs from Cranelift. These
were originally added to support multi-val

Remove Wasmtime ABIs from Cranelift (#6649)

* Remove Wasmtime ABIs from Cranelift

This commit removes the `Wasmtime*` family of ABIs from Cranelift. These
were originally added to support multi-value in Wasmtime via the
`TypedFunc` API, but they should now no longer be necessary. In general
this is a higher-level Wasmtime concern than something all backends of
Cranelift should have to deal with.

Today with recent refactorings it's possible to remove the reliance on
ABI details for multi-value and instead codify it directly into the
Cranelift IR generated. For example wasm calls are able to have a
"purely internal" ABI which Wasmtime's Rust code doesn't see at all, and
the Rust code only interacts with the native ABI. The native ABI is
redefined to be what the previous Wasmtime ABIs were, which is to return
the first of a 2+ value return through a register (native return value)
and everything else through a return pointer.

* Remove some wasmtime_system_v usage in tests

* Add back WasmtimeSystemV for s390x

* Fix some docs and references in winch

* Fix another doc link

show more ...

Cranelift: Get non-tail calls working for the "tail" calling convention (#6500)

Co-authored-by: Jamey Sharp <[email protected]>

Cranelift: Introduce the `return_call` and `return_call_indirect` instructions (#5679)

* Cranelift: Introduce the `tail` calling convention

This is an unstable-ABI calling convention that we will

Cranelift: Introduce the `return_call` and `return_call_indirect` instructions (#5679)

* Cranelift: Introduce the `tail` calling convention

This is an unstable-ABI calling convention that we will eventually use to
support Wasm tail calls.

Co-Authored-By: Jamey Sharp <[email protected]>

* Cranelift: Introduce the `return_call` and `return_call_indirect` instructions

These will be used to implement tail calls for Wasm and any other language
targeting CLIF. The `return_call_indirect` instruction differs from the Wasm
instruction of the same name by taking a native address callee rather than a
Wasm function index.

Co-Authored-By: Jamey Sharp <[email protected]>

* Cranelift: Implement verification rules for `return_call[_indirect]`

They must:

* have the same return types between the caller and callee,
* have the same calling convention between caller and callee,
* and that calling convention must support tail calls.

Co-Authored-By: Jamey Sharp <[email protected]>

* cargo fmt

---------

Co-authored-by: Jamey Sharp <[email protected]>

show more ...

Cranellift: remove Baldrdash support and related features. (#4571)

* Cranellift: remove Baldrdash support and related features.

As noted in Mozilla's bugzilla bug 1781425 [1], the SpiderMonkey te

Cranellift: remove Baldrdash support and related features. (#4571)

* Cranellift: remove Baldrdash support and related features.

As noted in Mozilla's bugzilla bug 1781425 [1], the SpiderMonkey team
has recently determined that their current form of integration with
Cranelift is too hard to maintain, and they have chosen to remove it
from their codebase. If and when they decide to build updated support
for Cranelift, they will adopt different approaches to several details
of the integration.

In the meantime, after discussion with the SpiderMonkey folks, they
agree that it makes sense to remove the bits of Cranelift that exist
to support the integration ("Baldrdash"), as they will not need
them. Many of these bits are difficult-to-maintain special cases that
are not actually tested in Cranelift proper: for example, the
Baldrdash integration required Cranelift to emit function bodies
without prologues/epilogues, and instead communicate very precise
information about the expected frame size and layout, then stitched
together something post-facto. This was brittle and caused a lot of
incidental complexity ("fallthrough returns", the resulting special
logic in block-ordering); this is just one example. As another
example, one particular Baldrdash ABI variant processed stack args in
reverse order, so our ABI code had to support both traversal
orders. We had a number of other Baldrdash-specific settings as well
that did various special things.

This PR removes Baldrdash ABI support, the `fallthrough_return`
instruction, and pulls some threads to remove now-unused bits as a
result of those two, with the understanding that the SpiderMonkey folks
will build new functionality as needed in the future and we can perhaps
find cleaner abstractions to make it all work.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1781425

* Review feedback.

* Fix (?) DWARF debug tests: add `--disable-cache` to wasmtime invocations.

The debugger tests invoke `wasmtime` from within each test case under
the control of a debugger (gdb or lldb). Some of these tests started to
inexplicably fail in CI with unrelated changes, and the failures were
only inconsistently reproducible locally. It seems to be cache related:
if we disable cached compilation on the nested `wasmtime` invocations,
the tests consistently pass.

* Review feedback.

show more ...

cranelift: introduce a new WasmtimeAppleAarch64 calling convention

The previous choice to use the WasmtimeSystemV calling convention for
apple-aarch64 devices was incorrect: padding of arguments was

cranelift: introduce a new WasmtimeAppleAarch64 calling convention

The previous choice to use the WasmtimeSystemV calling convention for
apple-aarch64 devices was incorrect: padding of arguments was
incorrectly computed. So we have to use some flavor of the apple-aarch64
ABI there.

Since we want to support the wasmtime custom convention for multiple
returns on apple-aarch64 too, a new custom Wasmtime calling convention
was introduced to support this.

show more ...

Fully support multiple returns in Wasmtime (#2806)

* Fully support multiple returns in Wasmtime

For quite some time now Wasmtime has "supported" multiple return values,
but only in the mose bare

Fully support multiple returns in Wasmtime (#2806)

* Fully support multiple returns in Wasmtime

For quite some time now Wasmtime has "supported" multiple return values,
but only in the mose bare bones ways. Up until recently you couldn't get
a typed version of functions with multiple return values, and never have
you been able to use `Func::wrap` with functions that return multiple
values. Even recently where `Func::typed` can call functions that return
multiple values it uses a double-indirection by calling a trampoline
which calls the real function.

The underlying reason for this lack of support is that cranelift's ABI
for returning multiple values is not possible to write in Rust. For
example if a wasm function returns two `i32` values there is no Rust (or
C!) function you can write to correspond to that. This commit, however
fixes that.

This commit adds two new ABIs to Cranelift: `WasmtimeSystemV` and
`WasmtimeFastcall`. The intention is that these Wasmtime-specific ABIs
match their corresponding ABI (e.g. `SystemV` or `WindowsFastcall`) for
everything *except* how multiple values are returned. For multiple
return values we simply define our own version of the ABI which Wasmtime
implements, which is that for N return values the first is returned as
if the function only returned that and the latter N-1 return values are
returned via an out-ptr that's the last parameter to the function.

These custom ABIs provides the ability for Wasmtime to bind these in
Rust meaning that `Func::wrap` can now wrap functions that return
multiple values and `Func::typed` no longer uses trampolines when
calling functions that return multiple values. Although there's lots of
internal changes there's no actual changes in the API surface area of
Wasmtime, just a few more impls of more public traits which means that
more types are supported in more places!

Another change made with this PR is a consolidation of how the ABI of
each function in a wasm module is selected. The native `SystemV` ABI,
for example, is more efficient at returning multiple values than the
wasmtime version of the ABI (since more things are in more registers).
To continue to take advantage of this Wasmtime will now classify some
functions in a wasm module with the "fast" ABI. Only functions that are
not reachable externally from the module are classified with the fast
ABI (e.g. those not exported, used in tables, or used with `ref.func`).
This should enable purely internal functions of modules to have a faster
calling convention than those which might be exposed to Wasmtime itself.

Closes #1178

* Tweak some names and add docs

* "fix" lightbeam compile

* Fix TODO with dummy environ

* Unwind info is a property of the target, not the ABI

* Remove lightbeam unused imports

* Attempt to fix arm64

* Document new ABIs aren't stable

* Fix filetests to use the right target

* Don't always do 64-bit stores with cranelift

This was overwriting upper bits when 32-bit registers were being stored
into return values, so fix the code inline to do a sized store instead
of one-size-fits-all store.

* At least get tests passing on the old backend

* Fix a typo

* Add some filetests with mixed abi calls

* Get `multi` example working

* Fix doctests on old x86 backend

* Add a mixture of wasmtime/system_v tests

show more ...

cranelift: add support for the Mac aarch64 calling convention

This bumps target-lexicon and adds support for the AppleAarch64 calling
convention. Specifically for WebAssembly support, we only have t

cranelift: add support for the Mac aarch64 calling convention

This bumps target-lexicon and adds support for the AppleAarch64 calling
convention. Specifically for WebAssembly support, we only have to worry
about the new stack slots convention. Stack slots don't need to be at
least 8-bytes, they can be as small as the data type's size. For
instance, if we need stack slots for (i32, i32), they can be located at
offsets (+0, +4). Note that they still need to be properly aligned on
the data type they're containing, though, so if we need stack slots for
(i32, i64), we can't start the i64 slot at the +4 offset (it must start
at the +8 offset).

Added one test that was failing on the Mac M1, as well as other tests
stressing different yet similar situations.

show more ...

Support for SpiderMonkey's "Wasm ABI 2020".

As part of a Wasm JIT update, SpiderMonkey is changing its internal
WebAssembly function ABI. The new ABI's frame format includes "caller
TLS" and "callee

Support for SpiderMonkey's "Wasm ABI 2020".

As part of a Wasm JIT update, SpiderMonkey is changing its internal
WebAssembly function ABI. The new ABI's frame format includes "caller
TLS" and "callee TLS" slots. The details of where these come from are
not important; from Cranelift's point of view, the only relevant
requirement is that we have two on-stack args that are always present
(offsetting other on-stack args), and that we define special argument
purposes so that we can supply values for these slots.

Note that this adds a *new* ABI (a variant of the Baldrdash ABI) because
we do not want to tightly couple the landing of this PR to the landing
of the changes in SpiderMonkey; it's better if both the old and new
behavior remain available in Cranelift, so SpiderMonkey can continue to
vendor Cranelift even if it does not land (or backs out) the ABI change.

Furthermore, note that this needs to be a Cranelift-level change (i.e.
cannot be done purely from the translator environment implementation)
because the special TLS arguments must always go on the stack, which
would not otherwise happen with the usual argument-placement logic; and
there is no primitive to push a value directly in CLIF code (the notion
of a stack frame is a lower-level concept).

show more ...

machinst x64: implement float Floor/Ceil/Trunc/Nearest as VM calls;

Write .debug_frame information (#53)

* Write .debug_frame information

* mv map_reg

Fix clippy warnings.

This commit fixes the current set of (stable) clippy warnings in the repo.

upgrade to target-lexicon 0.8.0

* the target-lexicon crate no longer has or needs the std feature
in cargo, so we can delete all default-features=false, any mentions
of its std feature, and the

upgrade to target-lexicon 0.8.0

* the target-lexicon crate no longer has or needs the std feature
in cargo, so we can delete all default-features=false, any mentions
of its std feature, and the nostd configs in many lib.rs files
* the representation of arm architectures has changed, so some case
statements needed refactoring

show more ...

Implement a Windows Baldrdash calling convention;

Adds the libcall_call_conv setting and use it for libcall calls expansion;