deprecate `[Typed]Func::post_return[_async]` and make them no-ops (#12498)

* deprecate `[Typed]Func::post_return[_async]` and make them no-ops

With the advent of the Component Model concurrency ABI

deprecate `[Typed]Func::post_return[_async]` and make them no-ops (#12498)

* deprecate `[Typed]Func::post_return[_async]` and make them no-ops

With the advent of the Component Model concurrency ABI and it's `task.return`
intrinsic, post-return functions have been informally deprecated and are
expected to be removed for WASI 1.0 and the corresponding stable edition of the
Component Model. Consequently, it does not make sense anymore to require
embedders to explicitly call the post-return function after using
`[Typed]Func::call[_async]`.

As of this commit, `[Typed]Func::post_return[_async]` are no-ops. Instead, the
post-return function is called automatically as part of
`[Typed]Func::call[_async]` if present, which is how
`[Typed]Func::call_concurrent` has worked all along. In addition, this commit
fixes and tests a couple of cases where the task and/or thread was being
disposed of before the post-return function was called.

* address review feedback

* test post-return function in more scenarios

Specifically, I've split the `invoke_post_return` test into multiple tests:

- using `TypedFunc::call`
- using `TypedFunc::call_async` with concurrency support enabled
- using `TypedFunc::call_async` with concurrency support disabled
- using `Func::call_async` with concurrency support disabled
- using `TypedFunc::call_concurrent`

* remove GCC/clang-specific deprecation attribute

This broke the MSVC build.

* bless bindgen output

* remove obsolete post-return functions and fields

Now that post-return calls are handled internally without requiring explicit
action by the embedder, we can avoid unnecessary bookkeeping.

show more ...

Migrate the Wasmtime CLI to `wasmtime::error` (#12295)

* Migrate wasmtime-cli to `wasmtime::error`

* migrate benches to `wasmtime::error` as well

* Remove new usage of anyhow that snuck in

disallow exiting a component instance during a post-return call (#11688)

* disallow exiting a component instance during a post-return call

This is a relatively recent change to the spec.

In order

disallow exiting a component instance during a post-return call (#11688)

* disallow exiting a component instance during a post-return call

This is a relatively recent change to the spec.

In order to check the `may_leave` flag in all the relevant intrinsics, I had to
plumb the caller `RuntimeComponentInstanceIndex` through a bunch of trampolines
that didn't previously need it, hence the large diff.

Note that I've added a slightly tweaked version of `trap-in-post-return.wast`
and left the upstream version disabled in `test-util/src/wast.rs` due to #11683.

Fixes #11676.

Signed-off-by: Joel Dice <[email protected]>

* fix test regressions

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

style: simplify string formatting (#9047)

* style: simplify string formatting

* fix: formatting in `benches/call.rs`

Implement component model resources in Wasmtime (#6691)

* Fix signatures registered with modules-in-components

This commit fixes a minor issue in
`FunctionIndices::link_and_append_code` which previ

Implement component model resources in Wasmtime (#6691)

* Fix signatures registered with modules-in-components

This commit fixes a minor issue in
`FunctionIndices::link_and_append_code` which previously ended up only
filling out the `wasm_to_native_trampolines` field for the first module
rather than all the modules. Additionally the first module might have
too many entries that encompass all modules instead of just its own
entries. The fix in this commit is to refactor this logic to ensure that
the necessary maps are present for all translations.

While technically a bug that can be surfaced through the embedder API
it's pretty obscure. The given test here panics beforehand but succeeds
afterwards, but this is moreso prep for some future resource-related
work where this map will need persisting into the component metadata
side of things.

* Initial support for resources

Lots of bits and pieces squashed into this commit. Much to be done
still.

* Start supporting destructors

* Get some basic drop tests working

Also add a test which requires host-defined drop to be called which
isn't working.

* Fix rebase issue

* Fix a failing test

* I am zorthax, destroyer of resources

* Remove a branch in compiled code

No need to check for a null funcref when we already know ahead of time
if it's ever going to be null or not.

* Fix the test suite

* Add embedder API to destroy resources

* Add TODO for factc

* Fix a warning and leave a comment

* Integrate resources into `Type`

Plumb around dynamic information about resource types.

* Implement `Val::Own`

* Implement reentrance check for destructors

Implemented both in the raw wasm intrinsic as well as the host.

* Use cast instead of transmute

* Fill out some cranelift-shared comments

* Update codegen for resource.drop shim

The MAY_ENTER flag must always be checked, regardless of whether there's
an actual destructor or not.

* Update wasm-tools crates to latest `main`

* Update resource.drop binary format

* Add some docs

* Implement dynamic tracking for borrow resources

Not actually hooked up anywhere but this should at least be a first stab
at an implementation of the spec.

* Remove git overrides

* Remove no-longer-needed arms in wit-bindgen

* Prepare for mutability in `LiftContext`

* Change `&LiftContext` to `&mut LiftContext`
* Remove `store: &'a StoreOpaque` from `LiftContext`, instead storing
just `memory: &'a [u8]`
* Refactor methods to avoid needing the entire `StoreOpaque`

This'll enable `LiftContext` to store `&'a mut ResourceTable` in an
upcoming commit to refer to the host's resources.

* Lowering a borrow is infallible

* Use `ResourceAny` for both own/borrow

Rename `Val::Own` to `Val::Resource` accordingly.

* Initial implementation of borrowed resources

Lots of juggling of contexts here and there to try and get everything
working but this is hopefully a faithful implementation. Tests not
implemented yet and will come next and additionally likely update
implementation details as issues are weeded out.

* Add a suite of tests for borrowing resources

Code coverage was used to ensure that almost all of the various paths
through the code are taken to ensure all the basic bases are covered.
There's probably still lurking bugs, but this should be a solid enough
base to start from hopefully.

* Fill in an issue for bindgen todo

* Add docs, still more to go

* Fill out more documentation

* Fill out a test TODO

* Update the host `Resource<T>` type

* Add docs everywhere
* Don't require a `Store` for creating the resource or getting the
representation.

The latter point is the main refactoring in this commit. This is done in
preparation for `bindgen!` to use this type where host bindings
generally do not have access to the store.

* Document `ResourceAny`

* Debug assert dtor is non-null

* Review comments on loading libcalls

* Update some comments

* Update a comment

* Fix some typos

* Add a test that host types are the same when guest types differ

* Fix some typos

* Thread things through a bit less

* Undo CompileKey-related changes

* Gate an async function on the async feature

* Fix doc links

* Skip resources tests in miri

They all involve compilation which takes too long and doesn't currently
work

show more ...

Run some tests in MIRI on CI (#6332)

* Run some tests in MIRI on CI

This commit is an implementation of getting at least chunks of Wasmtime
to run in MIRI on CI. The full test suite is not possible

Run some tests in MIRI on CI (#6332)

* Run some tests in MIRI on CI

This commit is an implementation of getting at least chunks of Wasmtime
to run in MIRI on CI. The full test suite is not possible to run in MIRI
because MIRI cannot run Cranelift-produced code at runtime (aka it
doesn't support JITs). Running MIRI, however, is still quite valuable if
we can manage it because it would have trivially detected
GHSA-ch89-5g45-qwc7, our most recent security advisory. The goal of this
PR is to select a subset of the test suite to execute in CI under MIRI
and boost our confidence in the copious amount of `unsafe` code in
Wasmtime's runtime.

Under MIRI's default settings, which is to use the [Stacked
Borrows][stacked] model, much of the code in `Instance` and `VMContext`
is considered invalid. Under the optional [Tree Borrows][tree] model,
however, this same code is accepted. After some [extremely helpful
discussion][discuss] on the Rust Zulip my current conclusion is that
what we're doing is not fundamentally un-sound but we need to model it
in a different way. This PR, however, uses the Tree Borrows model for
MIRI to get something onto CI sooner rather than later, and I hope to
follow this up with something that passed Stacked Borrows. Additionally
that'll hopefully make this diff smaller and easier to digest.

Given all that, the end result of this PR is to get 131 separate unit
tests executing on CI. These unit tests largely exercise the embedding
API where wasm function compilation is not involved. Some tests compile
wasm functions but don't run them, but compiling wasm through Cranelift
in MIRI is so slow that it doesn't seem worth it at this time. This does
mean that there's a pretty big hole in MIRI's test coverage, but that's
to be expected as we're a JIT compiler after all.

To get tests working in MIRI this PR uses a number of strategies:

* When platform-specific code is involved there's now `#[cfg(miri)]` for
MIRI's version. For example there's a custom-built "mmap" just for
MIRI now. Many of these are simple noops, some are `unimplemented!()`
as they shouldn't be reached, and some are slightly nontrivial
implementations such as mmaps and trap handling (for native-to-native
function calls).

* Many test modules are simply excluded via `#![cfg(not(miri))]` at the
top of the file. This excludes the entire module's worth of tests from
MIRI. Other modules have `#[cfg_attr(miri, ignore)]` annotations to
ignore tests by default on MIRI. The latter form is used in modules
where some tests work and some don't. This means that all future test
additions will need to be effectively annotated whether they work in
MIRI or not. My hope though is that there's enough precedent in the
test suite of what to do to not cause too much burden.

* A number of locations are fixed with respect to MIRI's analysis. For
example `ComponentInstance`, the component equivalent of
`wasmtime_runtime::Instance`, was actually left out from the fix for
the CVE by accident. MIRI dutifully highlighted the issues here and
I've fixed them locally. Some locations fixed for MIRI are changed to
something that looks similar but is subtly different. For example
retaining items in a `Store<T>` is now done with a Wasmtime-specific
`StoreBox<T>` type. This is because, with MIRI's analyses, moving a
`Box<T>` invalidates all pointers derived from this `Box<T>`. We don't
want these semantics, so we effectively have a custom `Box<T>` to suit
our needs in this regard.

* Some default configuration is different under MIRI. For example most
linear memories are dynamic with no guards and no space reserved for
growth. Settings such as parallel compilation are disabled. These are
applied to make MIRI "work by default" in more places ideally. Some
tests which perform N iterations of something perform fewer iterations
on MIRI to not take quite so long.

This PR is not intended to be a one-and-done-we-never-look-at-it-again
kind of thing. Instead this is intended to lay the groundwork to
continuously run MIRI in CI to catch any soundness issues. This feels,
to me, overdue given the amount of `unsafe` code inside of Wasmtime. My
hope is that over time we can figure out how to run Wasm in MIRI but
that may take quite some time. Regardless this will be adding nontrivial
maintenance work to contributors to Wasmtime. MIRI will be run on CI for
merges, MIRI will have test failures when everything else passes,
MIRI's errors will need to be deciphered by those who have probably
never run MIRI before, things like that. Despite all this to me it seems
worth the cost at this time. Just getting this running caught two
possible soundness bugs in the component implementation that could have
had a real-world impact in the future!

[stacked]: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
[tree]: https://perso.crans.org/vanille/treebor/
[discuss]: https://rust-lang.zulipchat.com/#narrow/stream/269128-miri/topic/Tree.20vs.20Stacked.20Borrows.20.26.20a.20debugging.20question

* Update alignment comment

show more ...

Remove explicit `S` type from component functions (#5722)

I ended up forgetting this as part of #5275.

Return `anyhow::Error` from host functions instead of `Trap`, redesign `Trap` (#5149)

* Return `anyhow::Error` from host functions instead of `Trap`

This commit refactors how errors are modeled w

Return `anyhow::Error` from host functions instead of `Trap`, redesign `Trap` (#5149)

* Return `anyhow::Error` from host functions instead of `Trap`

This commit refactors how errors are modeled when returned from host
functions and additionally refactors how custom errors work with `Trap`.
At a high level functions in Wasmtime that previously worked with
`Result<T, Trap>` now work with `Result<T>` instead where the error is
`anyhow::Error`. This includes functions such as:

* Host-defined functions in a `Linker<T>`
* `TypedFunc::call`
* Host-related callbacks like call hooks

Errors are now modeled primarily as `anyhow::Error` throughout Wasmtime.
This subsequently removes the need for `Trap` to have the ability to
represent all host-defined errors as it previously did. Consequently the
`From` implementations for any error into a `Trap` have been removed
here and the only embedder-defined way to create a `Trap` is to use
`Trap::new` with a custom string.

After this commit the distinction between a `Trap` and a host error is
the wasm backtrace that it contains. Previously all errors in host
functions would flow through a `Trap` and get a wasm backtrace attached
to them, but now this only happens if a `Trap` itself is created meaning
that arbitrary host-defined errors flowing from a host import to the
other side won't get backtraces attached. Some internals of Wasmtime
itself were updated or preserved to use `Trap::new` to capture a
backtrace where it seemed useful, such as when fuel runs out.

The main motivation for this commit is that it now enables hosts to
thread a concrete error type from a host function all the way through to
where a wasm function was invoked. Previously this could not be done
since the host error was wrapped in a `Trap` that didn't provide the
ability to get at the internals.

A consequence of this commit is that when a host error is returned that
isn't a `Trap` we'll capture a backtrace and then won't have a `Trap` to
attach it to. To avoid losing the contextual information this commit
uses the `Error::context` method to attach the backtrace as contextual
information to ensure that the backtrace is itself not lost.

This is a breaking change for likely all users of Wasmtime, but it's
hoped to be a relatively minor change to workaround. Most use cases can
likely change `-> Result<T, Trap>` to `-> Result<T>` and otherwise
explicit creation of a `Trap` is largely no longer necessary.

* Fix some doc links

* add some tests and make a backtrace type public (#55)

* Trap: avoid a trailing newline in the Display impl

which in turn ends up with three newlines between the end of the
backtrace and the `Caused by` in the anyhow Debug impl

* make BacktraceContext pub, and add tests showing downcasting behavior of anyhow::Error to traps or backtraces

* Remove now-unnecesary `Trap` downcasts in `Linker::module`

* Fix test output expectations

* Remove `Trap::i32_exit`

This commit removes special-handling in the `wasmtime::Trap` type for
the i32 exit code required by WASI. This is now instead modeled as a
specific `I32Exit` error type in the `wasmtime-wasi` crate which is
returned by the `proc_exit` hostcall. Embedders which previously tested
for i32 exits now downcast to the `I32Exit` value.

* Remove the `Trap::new` constructor

This commit removes the ability to create a trap with an arbitrary error
message. The purpose of this commit is to continue the prior trend of
leaning into the `anyhow::Error` type instead of trying to recreate it
with `Trap`. A subsequent simplification to `Trap` after this commit is
that `Trap` will simply be an `enum` of trap codes with no extra
information. This commit is doubly-motivated by the desire to always use
the new `BacktraceContext` type instead of sometimes using that and
sometimes using `Trap`.

Most of the changes here were around updating `Trap::new` calls to
`bail!` calls instead. Tests which assert particular error messages
additionally often needed to use the `:?` formatter instead of the `{}`
formatter because the prior formats the whole `anyhow::Error` and the
latter only formats the top-most error, which now contains the
backtrace.

* Merge `Trap` and `TrapCode`

With prior refactorings there's no more need for `Trap` to be opaque or
otherwise contain a backtrace. This commit parse down `Trap` to simply
an `enum` which was the old `TrapCode`. All various tests and such were
updated to handle this.

The main consequence of this commit is that all errors have a
`BacktraceContext` context attached to them. This unfortunately means
that the backtrace is printed first before the error message or trap
code, but given all the prior simplifications that seems worth it at
this time.

* Rename `BacktraceContext` to `WasmBacktrace`

This feels like a better name given how this has turned out, and
additionally this commit removes having both `WasmBacktrace` and
`BacktraceContext`.

* Soup up documentation for errors and traps

* Fix build of the C API

Co-authored-by: Pat Hickey <[email protected]>

show more ...

unsplat component::Linker::func_wrap args (#5065)

* component::Linker::func_wrap: replace IntoComponentFunc with directly accepting a closure

We find that this makes the Linker::func_wrap type si

unsplat component::Linker::func_wrap args (#5065)

* component::Linker::func_wrap: replace IntoComponentFunc with directly accepting a closure

We find that this makes the Linker::func_wrap type signature much easier
to read. The IntoComponentFunc abstraction was adding a lot of weight to
"splat" a set of arguments from a tuple of types into individual
arguments to the closure. Additionally, making the StoreContextMut
argument optional, or the Result<return> optional, wasn't very
worthwhile.

* Fixes for the new style of closure required by component::Linker::func_wrap

* fix fuzzing generator

show more ...

Upgrade wasm-tools crates, namely the component model (#4715)

* Upgrade wasm-tools crates, namely the component model

This commit pulls in the latest versions of all of the `wasm-tools`
family

Upgrade wasm-tools crates, namely the component model (#4715)

* Upgrade wasm-tools crates, namely the component model

This commit pulls in the latest versions of all of the `wasm-tools`
family of crates. There were two major changes that happened in
`wasm-tools` in the meantime:

* bytecodealliance/wasm-tools#697 - this commit introduced a new API for
more efficiently reading binary operators from a wasm binary. The old
`Operator`-based reading was left in place, however, and continues to
be what Wasmtime uses. I hope to update Wasmtime in a future PR to use
this new API, but for now the biggest change is...

* bytecodealliance/wasm-tools#703 - this commit was a major update to
the component model AST. This commit almost entirely deals with the
fallout of this change.

The changes made to the component model were:

1. The `unit` type no longer exists. This was generally a simple change
where the `Unit` case in a few different locations were all removed.
2. The `expected` type was renamed to `result`. This similarly was
relatively lightweight and mostly just a renaming on the surface. I
took this opportunity to rename `val::Result` to `val::ResultVal` and
`types::Result` to `types::ResultType` to avoid clashing with the
standard library types. The `Option`-based types were handled with
this as well.
3. The payload type of `variant` and `result` types are now optional.
This affected many locations that calculate flat type
representations, ABI information, etc. The `#[derive(ComponentType)]`
macro now specifically handles Rust-defined `enum` types which have
no payload to the equivalent in the component model.
4. Functions can now return multiple parameters. This changed the
signature of invoking component functions because the return value is
now bound by `ComponentNamedList` (renamed from `ComponentParams`).
This had a large effect in the tests, fuzz test case generation, etc.
5. Function types with 2-or-more parameters/results must uniquely name
all parameters/results. This mostly affected the text format used
throughout the tests.

I haven't added specifically new tests for multi-return but I changed a
number of tests to use it. Additionally I've updated the fuzzers to all
exercise multi-return as well so I think we should get some good
coverage with that.

* Update version numbers

* Use crates.io

show more ...

Update may_enter flag handling in components (#4354)

This commit updates the management of the `may_enter` flag in line with
WebAssembly/component-model#57. Namely the `may_enter` flag is now
excl

Update may_enter flag handling in components (#4354)

This commit updates the management of the `may_enter` flag in line with
WebAssembly/component-model#57. Namely the `may_enter` flag is now
exclusively managed in the `canon lift` function (which is
`TypedFunc::call`) and is only unset after post-return completes
successfully. This implements semantics where if any trap happens for
any reason (lifting, lowering, execution, imports, etc) then the
instance is considered permanently poisoned and can no longer be
entered.

Tests needed many updates to create new instances where previously the
same instance was reused after it had an erroneous state.

show more ...

Implement the post-return attribute (#4297)

This commit implements the `post-return` feature of the canonical ABI in
the component model. This attribute is an optionally-specified function
which i

Implement the post-return attribute (#4297)

This commit implements the `post-return` feature of the canonical ABI in
the component model. This attribute is an optionally-specified function
which is to be executed after the return value has been processed by the
caller to optionally clean-up the return value. This enables, for
example, returning an allocated string and the host then knows how to
clean it up to prevent memory leaks in the original module.

The API exposed in this PR changes the prior `TypedFunc::call` API in
behavior but not in its signature. Previously the `TypedFunc::call`
method would set the `may_enter` flag on the way out, but now that
operation is deferred until a new `TypedFunc::post_return` method is
called. This means that once a method on an instance is invoked then
nothing else can be done on the instance until the `post_return` method
is called. Note that the method must be called irrespective of whether
the `post-return` canonical ABI option was specified or not. Internally
wasm will be invoked if necessary.

This is a pretty wonky and unergonomic API to work with. For now I
couldn't think of a better alternative that improved on the ergonomics.
In the theory that the raw Wasmtime bindings for a component may not be
used all that heavily (instead `wit-bindgen` would largely be used) I'm
hoping that this isn't too much of an issue in the future.

cc #4185

show more ...