Relax panics in async/futures to traps/errors (#12688)

* Relax panics in async/futures to traps/errors

This commit is an admittance that I don't believe we're going to get
to a point where we are c

Relax panics in async/futures to traps/errors (#12688)

* Relax panics in async/futures to traps/errors

This commit is an admittance that I don't believe we're going to get
to a point where we are confident enough in the fuzzing of
component-model-async such that we could confidently say we're
exercising the vast majority of possible panics. Development of
component-model-async has shown a steady trickle of panics over the
course of the development of the feature, and this trend has been
persistent over time as well.

An attempt was made in #12119 to add a fuzzer dedicated to async events
but that didn't actually find anything in development and it has missed
a number of panics present before and discovered after its introduction.
Overall I do not know how to improve the fuzzer to the point that it
would find pretty much all of the existing async-related panics over
time.

To help address this concern of the `concurrent.rs` implementation this
commit goes through and replaces things like `unwrap()`, `assert!`,
`panic!`, and `unreachable!` with an error-producing form. The benefit
of this is that a bug in the implementation is less likely to result in
a panic and instead just results in a non-spec-compliant trap. The
downside of doing this though is that it can become unclear what errors
are "first class traps", or expected to be guest reachable, and which
are expected to be bugs in Wasmtime. To help address this I've performed
a few refactorings here as well.

* Some traps previously present as error strings are now promoted to
using `Trap::Foo` instead. This has some refactoring of the Rust/C
side as well to make it easier to define new variants. Tests were
additionally added for any trap messages that weren't previously
tested as being reachable.

* A new `bail_bug!` macro was added (internally) for Wasmtime. This is
coupled with a concrete `WasmtimeBug` error type (exported as
`wasmtime::WasmtimeBug`). The intention is that `bail!` continues to
be "here's a string and I'm a bit too lazy to make a concrete error"
while `bail_bug!` indicates "this is a bug in wasmtime please report
this if you see it".

The rough vision is that if an error condition is reached, and the system
is not broken in such a way that panicking is required, then `bail_bug!`
can be used to indicate a bug in Wasmtime as opposed to panicking. This
reduces the real-world impact of hitting these scenarios by downgrading a
CVE-worthy `panic!` into a bug-worthy non-spec-compliant trap. Not all
panics are able to be transitioned to this as some are load bearing from
a safety perspective or similar (or indicate something equally broken),
but the vast majority of cases are suitable for "return a trap, lock
down the store, and let destructors take care of everything else".

This change additionally has resulted in API changes for `FutureReader`
and `StreamReader`. For example creation of these types now returns a
`Result` for when the `ResourceTable` is full, for example, instead of
panicking.

* Fix CI build

* Translate `WasmtimeBug` to panics in debug mode

* Review comments

* Refactor some stream methods for fewer panics

show more ...

Migrate component model async tests to `wasmtime::error` (#12270)

* Migrate component model async tests to `wasmtime::error`

* fix and review feedback

Use `.` instead of `/` for interface members. (#11947)

* Use `.` instead of `/` for interface members.

In the `wasmtime::component::generate` macro, change the syntax for
referencing functions and

Use `.` instead of `/` for interface members. (#11947)

* Use `.` instead of `/` for interface members.

In the `wasmtime::component::generate` macro, change the syntax for
referencing functions and types inside interfaces to use `.` instead
of `/`.

For example, this changes strings like
`wasi:http/types/outgoing-body`
to
`wasi:http/types.outgoing-body`
.

This makes the syntax more consistent with the syntax of
[WIT `use` statements], which use `.` for this purpose.

And, it avoids an incompatibility with the future nested namespaces syntax
([��]), where the `/d` in `a:b/c/d` is for traversing a component export
rather than an interface member.

[WIT `use` statements]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md#wit-packages-and-use
[��]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md#gated-features

* Use the new syntax in more places.

* Revert changes to vendored WIT files.

* Revert more changes to vendored files.

* Update syntax in more places.

show more ...

move `ConcurrentState` from `ComponentInstance` to `Store` (#11796)

* move `ConcurrentState` from `ComponentInstance` to `Store`

This has a few benefits:

- No need to specify an instance when crea

move `ConcurrentState` from `ComponentInstance` to `Store` (#11796)

* move `ConcurrentState` from `ComponentInstance` to `Store`

This has a few benefits:

- No need to specify an instance when creating or piping from a stream or future.
- No need to track the instance in an `Accessor`.
- You may now execute tasks for multiple instances in a single event loop.

The main drawback is that, if one of several instances within a single store
traps, it effectively means all instances have trapped, and the store can't be
used to create new instances. The way to avoid that is to use separate stores
for instances which must be isolated from others.

As a result of this change, a lot of code had to move from e.g. `impl Instance`
to e.g. `impl StoreOpaque`, so the diff is pretty huge, but the changes
themselves are almost entirely non-functional.

Fixes #11226

Signed-off-by: Joel Dice <[email protected]>

* fix non-component-model-async build

Signed-off-by: Joel Dice <[email protected]>

* fix outdated doc comment

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

- restore `ComponentStoreData` encapsulation
- avoid conditional code duplication in `LiftContext::new`

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendez

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendezvous"-style API to a callback-oriented one.

Previously you would create e.g. a `StreamReader`/`StreamWriter` pair and call
their `read` and `write` methods, respectively, and those methods would return
`Future`s that resolved when the operation was matched with a corresponding
`write` or `read` operation on the other end.

With the new API, you instead provide a `StreamProducer` trait implementation
whe creating the stream, whose `produce` method will be called as soon as a read
happens, giving the implementation a chance to respond immediately without
making the reader wait for a rendezvous. Likewise, you can match the read end
of a stream to a `StreamConsumer` to respond immediately to writes. This model
should reduce scheduling overhead and make it easier to e.g. pipe items to/from
`AsyncWrite`/`AsyncRead` or `Sink`/`Stream` implementations without needing to
explicitly spawn background tasks. In addition, the new API provides direct
access to guest read and write buffers for `stream<u8>` operations, enabling
zero-copy operations.

Other changes:

- I've removed the `HostTaskOutput`; we were using it to run extra code with
access to the store after a host task completes, but we can do that more
elegantly inside the future using `tls::get`. This also allowed me to
simplify `Instance::poll_until` a bit.

- I've removed the `watch_{reader,writer}` functionality; it's not needed now
given that the runtime will automatically dispose of the producer or consumer
when the other end of the stream or future is closed -- no need for embedder
code to manage that.

- In order to make `UntypedWriteBuffer` `Send`, I had to wrap its raw pointer
`buf` field in a `SendSyncPtr`.

- I've removed `{Future,Stream}Writer` entirely and moved
`Instance::{future,stream}` to `{Future,Stream}Reader::new`, respectively.

- I've added a bounds check to the beginnings of `Instance::guest_read` and
`Instance::guest_write` so that we need not do it later in
`Guest{Source,Destination}::remaining`, meaning those functions can be
infallible.

Note that I haven't updated `wasmtime-wasi` yet to match; that will happen in
one or more follow-up commits.

Signed-off-by: Joel Dice <[email protected]>

* Add `Accessor::getter`, rename `with_data` to `with_getter`

* fixup bindgen invocation

Signed-off-by: Roman Volosatovs <[email protected]>

* add support for zero-length writes/reads to/from host

I've added a test to cover this; it also tests direct buffer access for
`stream<u8>`, which I realized I forgot to cover earlier. And of course there
was a bug :facepalm:.

Signed-off-by: Joel Dice <[email protected]>

* add `{Destination,Source}::remaining` methods

This can help `Stream{Producer,Consumer}` implementations determine how many
items to write or read, respectively.

Signed-off-by: Joel Dice <[email protected]>

* wasi: migrate sockets to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* tests: read the socket stream until EOF

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: account for cancellation

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: mostly ensure byte buffer cancellation-safety

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-filesystem: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3-sockets: mostly ensure byte buffer cancellation-safety

* p3-cli: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: limit maximum buffer size

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: remove reuseaddr test loop workaround

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: drive I/O in `when_ready`

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3: drive I/O in `when_ready`

* Refine `Stream{Producer,Consumer}` APIs

Per conversations last week with Roman, Alex, and Lann, I've updated these
traits to present a lower-level API based on `poll_{consume,produce}` functions
and have documented the implementation requirements for various scenarios which
have come up in `wasmtime-wasi`, particularly around graceful cancellation. See
the doc comments for those functions for details.

Signed-off-by: Joel Dice <[email protected]>

* being integration of new API

Signed-off-by: Roman Volosatovs <[email protected]>

* update wasi/src/p3/filesystem to use new stream API

This is totally untested so far; I'll run the tests once we have everything else
compiling.

Signed-off-by: Joel Dice <[email protected]>

* update wasi/src/p3/cli to use new stream API

This is totally untested and doesn't even compile yet due to a lifetime issue I
don't have time to address yet. I'll follow up later with a fix.

Signed-off-by: Joel Dice <[email protected]>

* fix: remove `'a` bound on `&self`

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:sockets` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:cli` adaptation

Note, that this removes the read optimization - let's get the
implementation complete first and optimize later

Signed-off-by: Roman Volosatovs <[email protected]>

* remove redundant loop in sockets

Signed-off-by: Roman Volosatovs <[email protected]>

* wasi: buffer on 0-length reads

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:filesystem` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* remove `MAX_BUFFER_CAPACITY`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor `Cursor` usage

Signed-off-by: Roman Volosatovs <[email protected]>

* impl Default for VecBuffer

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: use consistent import styling

Signed-off-by: Roman Volosatovs <[email protected]>

* feature-gate fs Arc accessors

Signed-off-by: Roman Volosatovs <[email protected]>

* Update test expectations

---------

Signed-off-by: Joel Dice <[email protected]>
Signed-off-by: Roman Volosatovs <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>
Co-authored-by: Roman Volosatovs <[email protected]>

show more ...

Simplify WASI internal implementations (#11365)

* Simplify WASI internal implementations

This commit migrates the WASIp2 implementation to be closer to the
upcoming WASIp3 implementation in terms o

Simplify WASI internal implementations (#11365)

* Simplify WASI internal implementations

This commit migrates the WASIp2 implementation to be closer to the
upcoming WASIp3 implementation in terms of how things are implemented
internally. Previously the way things worked with WASIp2 is:

* Embedders call `add_to_linker` with `T: WasiView`
* Internally `add_to_linker` is called which creates `WasiImpl<&mut T>`
* All internal implementations were `impl<T> Host for WasiImpl<T> where T: WasiView`
* A forwarding impl of `impl<T: WasiView> WasiView for &mut T` was
required

While this all worked it's a bit complicated for a few reasons:

1. Dealing with generically named structures like `WasiImpl` (or
`IoImpl` or `WasiHttpImpl`) is a bit baroque and not always obvious
as to what's going on.
2. The extra layer of generics in `impl<T> Host for WasiImpl<T>` adds a
layer of conceptual indirection which is non-obvious.
3. Other WASI proposal implementations do not use this strategy and
instead use "view" types or `impl Host for TheType` for example.
4. Internal incantations of `add_to_linker` had to deal with mixtures of
`IoImpl` and `WasiImpl` and aligning everything just right.
5. An extra layer of generics on all impls meant that everything was
generic meaning that `wasmtime-wasi`-the-crate didn't generate much
code, causing longer codegen times for consumers.

The goal of this commit is to migrate towards the style of what WASIp3
is prototyping for how impls are modeled. This is done to increase the
amount of code that can be shared between WASIp2 and WASIp3. This has a
number of benefits such as being easier to understand and also being
more modular where `wasi:clocks` implementations of traits don't require
filesystem context to be present (as is the case today). This in theory
helps a more mix-and-match paradigm of blending together various bits
and pieces of `wasmtime-wasi` implementations.

Concretely the changes made here are:

* `WasiView` no longer inherits from `IoView`, they're unrelated traits
now.
* `WasiView` now returns `WasiViewCtx<'a>` which has `ctx: &'a mut WasiCtx`
and `table: &'a mut ResourceTable`. That means it basically does the
same thing before but in a slightly different fashion.
* Implementations of `Host` traits are now directly for
`WasiCtxView<'_>` and don't involve any generics at all. These are
hopefully easier to understand and also better from a
codegen/compile-time perspective.
* Embedders no longer need to implement `IoView` directly and instead
fold that functionality into `WasiView`.
* `WasiHttpView` no longer inherits from `IoView` and instead has a
direct `fn table` method. Additionally `WasiHttpImpl` no longer embeds
`IoImpl` inside of it.
* Host traits for `wasi:io` are now implemented directly for
`ResourceTable` instead of `IoImpl<T>`.

The immediate goal of this refactoring is to enable more sharing along
the lines of #11362. This was not possible prior because WASIp3 requires
a simultaneous borrow on the table/ctx while the trait hierarchy
previously gave you one-or-the-other. With this new organization it will
be possible to get both at the same time meaning more
structure/contexts/etc can be shared between implementations.

prtest:full

* CI fixes

* More CI fixes

* More CI fixes

show more ...

refactor `{Stream,Future}|{Reader,Writer}` APIs and internals (#11325)

* refactor `{Stream,Future}|{Reader,Writer}` APIs and internals

This makes a several changes to how `{Stream,Future}|{Reader,W

refactor `{Stream,Future}|{Reader,Writer}` APIs and internals (#11325)

* refactor `{Stream,Future}|{Reader,Writer}` APIs and internals

This makes a several changes to how `{Stream,Future}|{Reader,Writer}` work to
make them more efficient and, in some ways, more ergonomic:

- The background tasks have been removed, allowing reads and writes to complete without task context switching. We now only allocate and use oneshot channels lazily when the other end is not yet ready; this improves real world performance benchmarks (e.g. wasi-http request handling) considerably.

- Instances of `{Stream,Future}Reader` can now be lifted and lowered directly; no need for `Host{Stream,Future}` anymore.

- The type parameter for `Stream{Reader,Writer}` no longer refers to the buffer type -- just the payload type (i.e. `StreamReader<u8>` instead of `StreamReader<Vec<u8>>`), meaning any buffer type may be used for a given read or write operation. This also means the compiler needs help with type inference less often when calling `Instance::stream`.

- Instances of `{Stream,Future}|{Reader,Writer}` now require access to the store in order to be disposed of properly. I've added RAII wrapper structs (`WithAccessor[AndValue]`) to help with this, and also updated `Store::drop` and `Instance::run_concurrent` to ensure the store thread-local is set when dropping futures closing over `&Accessor`s.

- In order to ensure that resources containing `{Stream,Future}|{Reader,Writer}` instances are disposed of properly, I've added `LinkerInstance::resource_concurrent` and have updated `wasmtime-wit-bindgen` to use it. This gives resource drop functions access to a `StoreContextMut` via an `Accessor`, allowing the stream and future handles to be disposed of.
- In order to make this work, I had to change `Accessor::instance` from a `Instance` to an `Option<Instance>`, which is awkward but temporary since we're planning to remove `Accessor::instance` entirely once we've moved concurrent state from `ComponentInstance` to `Store`.

That problem of disposal is definitely the most awkward part of all this. In
simple cases, it's easy enough to ensure that read and write handles are
disposed of properly, but both `wasmtime-wasi` and `wasmtime-wasi-http` have
some pretty complicated functions where handles are passed between tasks and/or
stored inside resources, so it can be tricky to ensure proper disposal on all
code paths. I'm open to ideas for improving this, but I suspect we'll need new
Rust language features (e.g. linear types) to make it truly ergonomic, robust,
and efficient.

While testing the above, I discovered an issue with `Instance::poll_until` such
that it would prematurely give up and return a "deadlock" trap error, believing
that there was no further work to do, even though the future passed to it was
ready to resolve the next time it was polled. I've fixed this by polling it one
last time and only trapping if it returns pending.

Note that I've moved a few associated functions from `ConcurrentState` to
`Instance` (e.g. `guest_drop_writable` and others) since they now need access to
the store; they're unchanged otherwise. Apologies for the diff noise.

Finally, I've tweaked how `wasmtime serve` to poll the guest for content before
handing the response to Hyper, which helps performance by ensuring the first
content chunk can be sent with the same TCP packet as the beginning of the
response.

Signed-off-by: Joel Dice <[email protected]>

fix wasi p3 build and test failures

Signed-off-by: Joel Dice <[email protected]>

use `ManuallyDrop` instead of `Option` in `Dropper`

This allows us to drop its `value` field in-place, i.e. without moving it,
thereby upholding the `Pin` guarantee.

Signed-off-by: Joel Dice <[email protected]>

address review comments

- Remove `DropWithStoreAndValue` and friends; go back to taking a `fn() -> T` parameter in `Instance::future` instead
- Make `DropWithStore::drop[_with]` take `&mut self` instead of `self`
- Make `WithAccessor` and `DropWithStore` private
- Instead, I've added public `Guarded{Stream,Future}{Reader,Writer}` types for RAII
- and also `{Stream,Future}{Reader,Writer}::close[_with]` methods
- Use RAII in `FutureReader::read` and `FutureWriter::write` to ensure handles are dropped if the `Future` is dropped

Signed-off-by: Joel Dice <[email protected]>

* lower host stream/future writes in background task

This avoids unsoundness due to guest realloc calls while there are host embedder
frames on the stack.

Signed-off-by: Joel Dice <[email protected]>

* fix `tcp.rs` regressions

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` m

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` macro. The main goal of this redesign is to
better support WASIp3 and component model async functions. Prior to this
redesign there was a mish mash of mechanisms to configure behavior of
imports/exports:

* The `async` configuration could turn everything async, nothing async,
only some imports async, or everything except some imports async.

* The `concurrent_{imports,exports}` keys were required to explicitly
opt-in to component model async signatures and applied to all
imports/exports.

* The `trappable_imports` configuration would indicate a list of imports
allowed to trap and it had special configuration for everything,
nothing, and only a certain list.

* The `tracing` and `verbose_tracing` keys could be applied to either
nothing or all functions.

Overall the previous state of configuration in `bindgen!` was clearly a
hodgepodge of systems that organically grew over time. In my personal
opinion it was in dire need of a refresh to take into account how
component-model-async ended up being implemented as well as
consolidating the one-off systems amongst all of these configuration
keys. A major motivation of this redesign, for example, was to inherit
behavior from WIT files by default. An `async` function in WIT should
not require `concurrent_*` keys to be configured, but rather it should
generate correct bindings by default.

In this commit, all of the above keys were removed. All keys have been
replaced with `imports` and `exports` configuration keys. Each behaves
the same way and looks like so:

bindgen!({
// ...
imports: {
// enable tracing for just this function
"my:local/interface/func": tracing,

// enable verbose tracing for just this function
"my:local/interface/other-func": tracing | verbose_tracing,

// this is blocking in WIT, but generate async bindings for
// it
"my:local/interface/[method]io.block": async,

// like above, but use "concurrent" bindings which have
// access to the store.
"my:local/interface/[method]io.block-again": async | store,

// everything else is, by default, trappable
default: trappable,
},
});

Effectively all the function-level configuration items are now bitflags.
These bitflags are by default inherited from the WIT files itself (e.g.
`async` functions are `async | store` by default). Further configuration
is then layered on top at the desires of the embedder. Supported keys are:

* `async` - this means that a Rust-level `async` function should be
generated. This is either `CallStyle::Async` or
`CallStyle::Concurrent` as it was prior, depending on ...

* `store` - this means that the generated function will have access to
the store on the host. This is only implemented right now for `async |
store` functions which map to `CallStyle::Concurrent`. In the future
I'd like to support just-`store` functions which means that you could
define a synchronous function with access to the store in addition to
an asynchronous function.

* `trappable` - this means that the function returns a
`wasmtime::Result<TheWitBindingType>`. If `trappable_errors` is
applicable then it means just a `Result<TheWitOkType,
TrappableErrorType>` is returned (like before)

* `tracing` - this enables `tracing!` integration for this function.

* `verbose_tracing` - this logs all argument values for this function
(including lists).

* `ignore_wit` - this ignores the WIT-level defaults of the function
(e.g. ignoring WIT `async`).

The way this then works is all modeled is that for any WIT function
being generated there are a set of flags associated with that function.
To calculate the flags the algorithm looks like:

1. Find the first matching rule in the `imports` or `exports` map
depending on if the function is imported or exported. If there is no
matching rule then use the `default` rule if present. This is the
initial set of flags for the function (or empty if nothing was
found).

2. If `ignore_wit` is present, return the flags from step 1. Otherwise
add in `async | store` if the function is `async` in WIT.

The resulting set of flags are then used to control how everything is
generated. For example the same split traits of today are still
generated and it's controlled based on the flags. Note though that the
previous `HostConcurrent` trait was renamed to `HostWithStore` to make
space for synchronous functions in this trait in the future too.

The end result of all these changes is that configuring imports/exports
now uses the exact same selection system as the `with` replacement map,
meaning there's only one system of selecting functions instead of 3.
WIT-level `async` is now respected by default meaning that bindings work
by default without further need to configure anything (unless more
functionality is desired).

One final minor change made here as well is that auto-generated
`instantiate` methods are now always synchronous and an
`instantiate_async` method is unconditionally generated for async mode.
This means that bindings always generate both functions and it's up to
the embedder to choose the appropriate one.

Closes #11246
Closes #11247

* Update expanded test expectations

prtest:full

* Fix the min platform embedding example

* Fix doc tests

* Always generate `*WithStore` traits

This helps when using the `with` mapping since that can always assume
that `HostWithStore` is available in the generated bindings, avoiding
the need to duplicate configuration options.

* Update test expectations

* Review comments

show more ...

Take `&mut self` in `Stream{Reader,Writer}` methods (#11261)

This commit updates the various methods of `Stream{Reader,Writer}` to
take a mutable reference to `self` instead of `self`-by-value. This

Take `&mut self` in `Stream{Reader,Writer}` methods (#11261)

This commit updates the various methods of `Stream{Reader,Writer}` to
take a mutable reference to `self` instead of `self`-by-value. This is
something which I personally feel is more idiomatic and avoids
cumbersome and possibly frequent move-in-and-move-out style code. This
is additionally coupled with a new `is_closed` method on each of these
types to determine if the stream's other end has been closed after the
last read/write operation.

Closes #11219

show more ...

Require `Accessor` on all future/stream functions (#11250)

* Require `Accessor` on all future/stream functions

This is a follow-up to #11238 which adds `&Accessor` arguments to all
functions for fu

Require `Accessor` on all future/stream functions (#11250)

* Require `Accessor` on all future/stream functions

This is a follow-up to #11238 which adds `&Accessor` arguments to all
functions for futures and streams. Like #11238 this is done to make
future refactorings easier for the internal implementation but the
internal implementations are not updated at this time. Many functions,
for example, do not use the argument at all just yet. The purpose of
this is to ensure host usage of these functions always provides a store
context.

This change required large refactorings of the upcoming
wasmtime-wasi-http implementation in the wasip3-prototyping repository.
That's all been sorted out now though so the changes are being pulled
back here into the Wasmtime repository as well.

This commit additionally changes the `watch_*` functions on the various
stream/future types to take `&mut self` instead of `self`-by-value. This is
mostly a stylistic change and is more API-driven than anything else.
Functionally this behaves the same as before where, while watching, the
stream/future cannot be read/written to otherwise.

* Review comments

show more ...

Start to use `&Accessor<T, D>` more in concurrent code (#11238)

* Start to use `&Accessor<T, D>` more in concurrent code

After discussion with Joel we've concluded that while `&mut Accessor<T, D>`

Start to use `&Accessor<T, D>` more in concurrent code (#11238)

* Start to use `&Accessor<T, D>` more in concurrent code

After discussion with Joel we've concluded that while `&mut Accessor<T, D>`
was originally added to model host functions it is also appropriate to
use it to model embedder-rooted invocations of items such as wasm as
well. Effectively the conclusion we reached was that
`*::call_concurrent` should be taking `&Accessor`, not
`StoreContextMut`. This has a number of benefits to it over the previous
iteration:

* This makes exports behave more like imports where `Accessor` means
"you're in the concurrent world".

* This makes exports have an `async fn` signature which is easier to
read and understand.

* This automatically enforces the guarantee that the returned future is
only polled within the main event loop because the future is always
considered to close over the `&Accessor` provided meaning it
statically cannot live outside of the event loop.

* This paves the way forward to future refactorings to avoid storing so
much state within a `Store<T>` and instead try to store state directly
in futures themselves. This should make cancellation more natural and
eventually also remove `'static` bounds on params/results. Furthermore
this should make it easier to avoid spawning tasks internally by
storing state in futures instead of spawned tasks.

In doing this one of the main questions we were faced with was what to
do about `&mut Accessor<T, D>`, namely the `mut` part. With a mutable
accessor it would be only possible to call one function concurrently.
One option considered was to add combinators like `Accessor::join` and
`Accessor::race` but in the end we decided to avoid going that direction
and instead switch to `&Accessor<T, D>` everywhere, freely enabling
aliasing of the accessor. This has the downside that `Accessor::with` is
now a relatively dangerous function in that it can panic, but idiomatic
usage of it is not expected to panic as the distinction between the
`async` and sync boundary of `Accessor` vs `StoreContextMut` is expected
to naturally make the recursive panic condition of `with` rare to come
up in practice.

Concrete changes in this commit are:

* `Accessor::with` now requires `&self`.
* `Accessor::spawn` now requires `&self`.
* Host functions are now given `&Accessor`, not `&mut Accessor`.
* `{Typed,}Func::call_concurrent` is now an `async fn` which takes an
`&Accessor` instead of `StoreContextMut`.
* Guest bindings generation for concurrent invocations now looks exactly
like async bindings generation except for replacing `StoreContextMut`
with `Accessor`.

Note that this commit does not yet update the internal implementations
of these functions to benefit from the new abilities that taking
`&Accessor` implies. Instead that's deferred to a future update as
necessary. Instead this is only updating the public API of the
`wasmtime` crate to enable these refactorings in the future.

Also note that this does not yet update all functions to take
`&Accessor`. Notably futures and streams still need to be updated.

cc #11224

* Review comments

---------

Co-authored-by: Joel Dice <[email protected]>

show more ...

add Component Model async ABI tests (#11136)

* add Component Model async ABI tests

This pulls in the tests from the `wasip3-prototyping` repo, minus the ones
requiring WASIp3 support in `wasmtime-w

add Component Model async ABI tests (#11136)

* add Component Model async ABI tests

This pulls in the tests from the `wasip3-prototyping` repo, minus the ones
requiring WASIp3 support in `wasmtime-wasi[-http]`, which will be PR'd
separately.

* add audits and exemptions for new `component-async-tests` deps

In order to convince `cargo vet` that we only needed these deps to be
`safe-to-run` (not necessarily `safe-to-deploy`, since it's test code), I've
moved the `wasm-compose` dep to the `dev-dependencies` section of the
`Cargo.toml` file, which required rearranging some code.

I've exempted `wasm-compose` since it's a BA project, and also exempted all but
one of the remaining new deps since they each get well over 10,000 downloads per
day from crates.io. I've audited and certified the remaining dep, `im-rc`,
which came in a bit shy of the 10,000-per-day mark.

Signed-off-by: Joel Dice <[email protected]>

* simplify `component_async_tests::util::sleep`

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...