Remove subtask reparenting (#12570)

This commit updates the implementation of component-model-async
primitives to remove the manual subtask reparenting process. This is
required to fix #12544 at a s

Remove subtask reparenting (#12570)

This commit updates the implementation of component-model-async
primitives to remove the manual subtask reparenting process. This is
required to fix #12544 at a semantic level because a subtask isn't ever
actually reparented, even if its parent exits. The first part of this
change is to remove the `GuestTask::subtasks` field and all relevant
manipulations of it.

This field, however, powered the `TaskExit` abstraction returned from
`call_concurrent`. This commit then subsequently deletes `TaskExit` and
all related infrastructure as it's no longer directly applicable as-is.
The rest of this change is then updating tests/bindings/etc to account
for these two changes.

The main semantic changes related to tests are:

* `wasmtime run`, with and without `--invoke`, no longer waits for all
subtasks. This instead only waits for the main task returning before
exiting. Whether or not this is the correct behavior is under
discussion in WebAssembly/component-model#608

* `wasmtime serve` has been updated to keep the store alive at least
until the response body has been fully transmitted. This is also part
of WebAssembly/component-model#608.

* Some `component-async-tests`-related tests were updated to either
avoid blocking the store as it wasn't needed or yield enough times to
ensure that the test passes.

Closes #12544

prtest:full

show more ...

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtim

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtime_environ::error` as the `wasmtime::error`
module, updates the prelude to include these new error-handling types, redirects
our top-level `wasmtime::{Error, Result}` re-exports to re-export
`wasmtime::error::{Error, Result}`, and updates various use sites that were
directly using `anyhow` to use the new `wasmtime` versions.

This process also required updating the component macro and wit-bindgen macro to
use the new error types instead of `anyhow`.

Part of https://github.com/bytecodealliance/wasmtime/issues/12069

* Replace wasmtime::error::Thing with wasmtime::Thing where it makes sense

* cargo fmt

* Move `crate::error::Thing` to `crate::Thing` where it makes sense

show more ...

add `task_exit` option to `wasmtime-wit-bindgen` (#11665)

This builds on #11662 by optionally exposing the `TaskExit` return value from
`[Typed]Func::call_concurrent` in the bindings generated for e

add `task_exit` option to `wasmtime-wit-bindgen` (#11665)

This builds on #11662 by optionally exposing the `TaskExit` return value from
`[Typed]Func::call_concurrent` in the bindings generated for exported functions.

Note that the first two commits are shared with #11662.

Fixes #11600

Signed-off-by: Joel Dice <[email protected]>

show more ...

return `TaskExit` future from `[Typed]Func::call_concurrent` (#11662)

* return `TaskExit` future from `[Typed]Func::call_concurrent`

In addition to returning the value produced by the callee, these

return `TaskExit` future from `[Typed]Func::call_concurrent` (#11662)

* return `TaskExit` future from `[Typed]Func::call_concurrent`

In addition to returning the value produced by the callee, these functions now
also return a `TaskExit` future which resolves once the subtask (and any
transitively-created subtasks) have exited. This partially addresses #11600;
the next step will be to add a `wasmtime-wit-bindgen` option to expose the
`TaskExit` value in generated bindings.

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

`TaskExit` now has an `async fn block` instead of closing over an `impl
AsAccessor` and implementing `Future`.

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` m

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` macro. The main goal of this redesign is to
better support WASIp3 and component model async functions. Prior to this
redesign there was a mish mash of mechanisms to configure behavior of
imports/exports:

* The `async` configuration could turn everything async, nothing async,
only some imports async, or everything except some imports async.

* The `concurrent_{imports,exports}` keys were required to explicitly
opt-in to component model async signatures and applied to all
imports/exports.

* The `trappable_imports` configuration would indicate a list of imports
allowed to trap and it had special configuration for everything,
nothing, and only a certain list.

* The `tracing` and `verbose_tracing` keys could be applied to either
nothing or all functions.

Overall the previous state of configuration in `bindgen!` was clearly a
hodgepodge of systems that organically grew over time. In my personal
opinion it was in dire need of a refresh to take into account how
component-model-async ended up being implemented as well as
consolidating the one-off systems amongst all of these configuration
keys. A major motivation of this redesign, for example, was to inherit
behavior from WIT files by default. An `async` function in WIT should
not require `concurrent_*` keys to be configured, but rather it should
generate correct bindings by default.

In this commit, all of the above keys were removed. All keys have been
replaced with `imports` and `exports` configuration keys. Each behaves
the same way and looks like so:

bindgen!({
// ...
imports: {
// enable tracing for just this function
"my:local/interface/func": tracing,

// enable verbose tracing for just this function
"my:local/interface/other-func": tracing | verbose_tracing,

// this is blocking in WIT, but generate async bindings for
// it
"my:local/interface/[method]io.block": async,

// like above, but use "concurrent" bindings which have
// access to the store.
"my:local/interface/[method]io.block-again": async | store,

// everything else is, by default, trappable
default: trappable,
},
});

Effectively all the function-level configuration items are now bitflags.
These bitflags are by default inherited from the WIT files itself (e.g.
`async` functions are `async | store` by default). Further configuration
is then layered on top at the desires of the embedder. Supported keys are:

* `async` - this means that a Rust-level `async` function should be
generated. This is either `CallStyle::Async` or
`CallStyle::Concurrent` as it was prior, depending on ...

* `store` - this means that the generated function will have access to
the store on the host. This is only implemented right now for `async |
store` functions which map to `CallStyle::Concurrent`. In the future
I'd like to support just-`store` functions which means that you could
define a synchronous function with access to the store in addition to
an asynchronous function.

* `trappable` - this means that the function returns a
`wasmtime::Result<TheWitBindingType>`. If `trappable_errors` is
applicable then it means just a `Result<TheWitOkType,
TrappableErrorType>` is returned (like before)

* `tracing` - this enables `tracing!` integration for this function.

* `verbose_tracing` - this logs all argument values for this function
(including lists).

* `ignore_wit` - this ignores the WIT-level defaults of the function
(e.g. ignoring WIT `async`).

The way this then works is all modeled is that for any WIT function
being generated there are a set of flags associated with that function.
To calculate the flags the algorithm looks like:

1. Find the first matching rule in the `imports` or `exports` map
depending on if the function is imported or exported. If there is no
matching rule then use the `default` rule if present. This is the
initial set of flags for the function (or empty if nothing was
found).

2. If `ignore_wit` is present, return the flags from step 1. Otherwise
add in `async | store` if the function is `async` in WIT.

The resulting set of flags are then used to control how everything is
generated. For example the same split traits of today are still
generated and it's controlled based on the flags. Note though that the
previous `HostConcurrent` trait was renamed to `HostWithStore` to make
space for synchronous functions in this trait in the future too.

The end result of all these changes is that configuring imports/exports
now uses the exact same selection system as the `with` replacement map,
meaning there's only one system of selecting functions instead of 3.
WIT-level `async` is now respected by default meaning that bindings work
by default without further need to configure anything (unless more
functionality is desired).

One final minor change made here as well is that auto-generated
`instantiate` methods are now always synchronous and an
`instantiate_async` method is unconditionally generated for async mode.
This means that bindings always generate both functions and it's up to
the embedder to choose the appropriate one.

Closes #11246
Closes #11247

* Update expanded test expectations

prtest:full

* Fix the min platform embedding example

* Fix doc tests

* Always generate `*WithStore` traits

This helps when using the `with` mapping since that can always assume
that `HostWithStore` is available in the generated bindings, avoiding
the need to duplicate configuration options.

* Update test expectations

* Review comments

show more ...

Start to use `&Accessor<T, D>` more in concurrent code (#11238)

* Start to use `&Accessor<T, D>` more in concurrent code

After discussion with Joel we've concluded that while `&mut Accessor<T, D>`

Start to use `&Accessor<T, D>` more in concurrent code (#11238)

* Start to use `&Accessor<T, D>` more in concurrent code

After discussion with Joel we've concluded that while `&mut Accessor<T, D>`
was originally added to model host functions it is also appropriate to
use it to model embedder-rooted invocations of items such as wasm as
well. Effectively the conclusion we reached was that
`*::call_concurrent` should be taking `&Accessor`, not
`StoreContextMut`. This has a number of benefits to it over the previous
iteration:

* This makes exports behave more like imports where `Accessor` means
"you're in the concurrent world".

* This makes exports have an `async fn` signature which is easier to
read and understand.

* This automatically enforces the guarantee that the returned future is
only polled within the main event loop because the future is always
considered to close over the `&Accessor` provided meaning it
statically cannot live outside of the event loop.

* This paves the way forward to future refactorings to avoid storing so
much state within a `Store<T>` and instead try to store state directly
in futures themselves. This should make cancellation more natural and
eventually also remove `'static` bounds on params/results. Furthermore
this should make it easier to avoid spawning tasks internally by
storing state in futures instead of spawned tasks.

In doing this one of the main questions we were faced with was what to
do about `&mut Accessor<T, D>`, namely the `mut` part. With a mutable
accessor it would be only possible to call one function concurrently.
One option considered was to add combinators like `Accessor::join` and
`Accessor::race` but in the end we decided to avoid going that direction
and instead switch to `&Accessor<T, D>` everywhere, freely enabling
aliasing of the accessor. This has the downside that `Accessor::with` is
now a relatively dangerous function in that it can panic, but idiomatic
usage of it is not expected to panic as the distinction between the
`async` and sync boundary of `Accessor` vs `StoreContextMut` is expected
to naturally make the recursive panic condition of `with` rare to come
up in practice.

Concrete changes in this commit are:

* `Accessor::with` now requires `&self`.
* `Accessor::spawn` now requires `&self`.
* Host functions are now given `&Accessor`, not `&mut Accessor`.
* `{Typed,}Func::call_concurrent` is now an `async fn` which takes an
`&Accessor` instead of `StoreContextMut`.
* Guest bindings generation for concurrent invocations now looks exactly
like async bindings generation except for replacing `StoreContextMut`
with `Accessor`.

Note that this commit does not yet update the internal implementations
of these functions to benefit from the new abilities that taking
`&Accessor` implies. Instead that's deferred to a future update as
necessary. Instead this is only updating the public API of the
`wasmtime` crate to enable these refactorings in the future.

Also note that this does not yet update all functions to take
`&Accessor`. Notably futures and streams still need to be updated.

cc #11224

* Review comments

---------

Co-authored-by: Joel Dice <[email protected]>

show more ...

Re-enable concurrent bindings generation tests (#10972)

* Re-enable concurrent bindings generation tests

This commit re-enables tests for bindings generation for concurrent
calls in the main repo

Re-enable concurrent bindings generation tests (#10972)

* Re-enable concurrent bindings generation tests

This commit re-enables tests for bindings generation for concurrent
calls in the main repo after all syncs have now finished with wasip3.
This additionally adds some skeleton APIs that the bindings generator
uses which are necessary to get tests passing.

* Update test expectations

show more ...

Add `T: 'static` to `Store<T>` (#10760)

* Add `T: 'static` to `Store<T>

Since the beginning the `T` type parameter on `Store<T>` has had no
bounds on it. This was intended for maximal flexibility i

Add `T: 'static` to `Store<T>` (#10760)

* Add `T: 'static` to `Store<T>

Since the beginning the `T` type parameter on `Store<T>` has had no
bounds on it. This was intended for maximal flexibility in terms of what
embedders place within a `Store<T>` and I've personally advocated that
we need to keep it this way. In the development of the WASIp3 work,
however, I've at least personally reached the conclusion that this is no
longer tenable and proceeding will require adding a `'static` bound to
data within a store.

Wasmtime today [already] carries unsafe `transmute`s to work around this
lack of `'static` bound, and while the number of `unsafe` parts is
relatively small right now we're still fundamentally lying to the
compiler about lifetime bounds internally. With the WASIp3 async work
this degree of "lying" has become even worse. Joel has written up some
examples [on Zulip] about how the Rust compiler is requiring `'static`
bounds in surprising ways. These patterns are cropping up quite
frequently in the WASIp3 work and it's becoming particularly onerous
maintaining all of the `unsafe` and ensuring that everything is in sync.

In the WASIp3 repository I've additionally [prototyped a change] which
would additionally practically require `T: 'static` in more locations.
This change is one I plan on landing in Wasmtime in the near future and
while its main motivations are for enabling WASIp3 work it is also a
much nicer system than what we have today, in my opinion.

Overall the cost of not having `T: 'static` on `Store<T>` is effectively
becoming quite costly, in particular with respect to WASIp3 work. This
is coupled with all known embedders already using `T: 'static` data
within a `Store<T>` so the expectation of the impact of this change is
not large. The main downside of this change as a result is that when and
where to place `'static` bounds is sort of a game of whack-a-mole with
the compiler. For example I changed `Store<T>` to require `'static`
here, but the rest of the change is basically "hit compile until rustc
says it's ok". There's not necessarily a huge amount of rhyme-or-reason
to where `'static` bounds crop up, which can be surprising or difficult
to work with for users.

In the end I feel that this change is necessary and one we can't shy
away from. If problems crop up we'll need to figure out how to thread
that needle at that time, but I'm coming around to thinking that
`T: 'static` is just a fundamental constraint we'll have to take on at
this time. Maybe a future version of Rust that fixes some of Joel's
examples (if they can be fixed, we're not sure of that) we could
consider relaxing this but that's left for future work.

[already]: https://github.com/bytecodealliance/wasmtime/blob/35053d6d8d1a5d4692cf636cba0c920b4a79a44b/crates/wasmtime/src/runtime/store.rs#L602-L611
[on Zulip]: https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/.22type.20may.20not.20live.20long.20enough.22.20for.20generic.20closure/near/473862072
[prototyped a change]: https://github.com/bytecodealliance/wasip3-prototyping/pull/158

* Remove a no-longer-necessary `unsafe` block

* Update test expectations

* Fix gc-disabled builds

show more ...

wasmtime-wit-bindgen: Typecheck exports at {Foo}Indices construction (#10610)

* wasmtime::component: make it possible to typecheck export funcs

* wasmtime-wit-bindgen: add typechecking on construct

wasmtime-wit-bindgen: Typecheck exports at {Foo}Indices construction (#10610)

* wasmtime::component: make it possible to typecheck export funcs

* wasmtime-wit-bindgen: add typechecking on construction of Indices struct

* wit-bindgen: reduce to a single Indices constructor which takes InstancePre

* bless bindgen output

show more ...

Component and Instance have corresponding get_export, get_export_index (#10597)

* component::Instance: get_export gives a ComponentItem, ExportIndex pair

just like Component::export_index does

* a

Component and Instance have corresponding get_export, get_export_index (#10597)

* component::Instance: get_export gives a ComponentItem, ExportIndex pair

just like Component::export_index does

* align Instance and Component with get_export and get_export_index

* fix example

* component macro expanded: bless output

* fix component model tests

* cli only needs get_export_index

* code review: deduplicate export to typedef transformation into ComponentItem::from_export

show more ...

async/stream/future support for wasmtime-wit-bindgen (#10044)

* async/stream/future support for wasmtime-wit-bindgen

I've split this out of #9582 to make review easier.

This patch adds async/strea

async/stream/future support for wasmtime-wit-bindgen (#10044)

* async/stream/future support for wasmtime-wit-bindgen

I've split this out of #9582 to make review easier.

This patch adds async/stream/future/error-context support to the host binding
generator, along with placeholder type and function definitions in the
`wasmtime` crate which the generated bindings can refer to. See
https://github.com/dicej/rfcs/blob/component-async/accepted/component-model-async.md#componentbindgen-updates
for the design and rationale.

Note that I've added temporary `[patch.crates-io]` overrides in Cargo.toml until
https://github.com/bytecodealliance/wit-bindgen/pull/1130 and
https://github.com/bytecodealliance/wasm-tools/pull/1978 have been released.

Also note that we emit a `T: 'static` bound for `AsContextMut<Data = T>` when
generating bindings with `concurrent_imports: true`. This is only because
`rustc` insists that the closure we're passing to
`LinkerInstance::func_wrap_concurrent` captures the lifetime of `T` despite my
best efforts to convince it otherwise. Alex and I suspect this is a limitation
in the compiler, and I asked about it on the rust-lang Zulip, but we haven't
been able to determine a workaround so far.

Signed-off-by: Joel Dice <[email protected]>

remove obsolete TODO comment

Signed-off-by: Joel Dice <[email protected]>

make `futures` dep optional

Signed-off-by: Joel Dice <[email protected]>

update `wasm-tools` and `wit-bindgen`

Signed-off-by: Joel Dice <[email protected]>

* run cargo vet

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...