Yield instead of sleep in component-async-tests (#12567)

* Yield instead of sleep in component-async-tests

Use cooperative yields instead of sleeps to make tests more
deterministic and also avoid t

Yield instead of sleep in component-async-tests (#12567)

* Yield instead of sleep in component-async-tests

Use cooperative yields instead of sleeps to make tests more
deterministic and also avoid them unnecessarily taking up test
parallelism by sleeping. Yielding should have the same effect in terms
of testing by exercising behavior returning `Pending` in futures, so
there's no expected loss in test coverage here.

* Yield fewer times

show more ...

make async tests using `ready` interface more robust (#12360)

This changes the `ready` interface used by `component-async-tests` from:

```
interface ready {
// Set the `ready` state
set-ready:

make async tests using `ready` interface more robust (#12360)

This changes the `ready` interface used by `component-async-tests` from:

```
interface ready {
// Set the `ready` state
set-ready: func(ready: bool);
// Block until `ready` is `true`
when-ready: async func();
}
```
to:
```
interface ready {
resource thing {
constructor();
set-ready: func(ready: bool);
when-ready: async func();
}
}
```

The problem with the original version was that it required global state and thus
caused cross-talk across concurrent tasks. Due to implementation details inside
Wasmtime, the tests worked anyway, but
https://github.com/bytecodealliance/wasmtime/pull/12357 perturbed that and
revealed how fragile tests based on that interface were.

The new version puts the state inside a resource type, allowing each task create
its own instance of that resource type and thereby avoid crosstalk.

show more ...

trap on blocking call in sync task before return (#12043)

* trap on blocking call in sync task before return

This implements a spec change (PR pending) such that tasks created for calls to
synchron

trap on blocking call in sync task before return (#12043)

* trap on blocking call in sync task before return

This implements a spec change (PR pending) such that tasks created for calls to
synchronous exports may not call potentially-blocking imports or return `wait`
or `poll` callback codes prior to returning a value. Specifically, the
following are prohibited in that scenario:

- returning callback-code.{wait,poll}
- sync calling an async import
- sync calling subtask.cancel
- sync calling {stream,future}.{read,write}
- sync calling {stream,future}.cancel-{read,write}
- calling waitable-set.{wait,poll}
- calling thread.suspend

This breaks a number of tests, which will be addressed in follow-up commits:

- The `{tcp,udp}-socket.bind` implementation in `wasmtime-wasi` is implemented
using `Linker::func_wrap_concurrent` and thus assumed to be async, whereas the
WIT interface says they're sync, leading to a type mismatch error at runtime.
Alex and I have discussed this and have a general plan to address it.

- A number of tests in the tests/component-model submodule that points to the
spec repo are failing. Those will presumably be fixed as part of the upcoming
spec PR (although some could be due to bugs in this implementation, in which
case I'll fix them).

- A number of tests in tests/misc_testsuite are failing. I'll address those in
a follow-up commit.

Signed-off-by: Joel Dice <[email protected]>

* call `check_may_leave` before `check_blocking`

`check_blocking` needs access to the current task, but that's not set for
post-return functions since those should not be calling _any_ imports at all, so
first check for that.

Signed-off-by: Joel Dice <[email protected]>

* fix `misc_testsuite` test regressions

This amounts to adding `async` to any exported component functions that might
need to block.

Signed-off-by: Joel Dice <[email protected]>

* simplify code in `ConcurrentState::check_blocking`

Signed-off-by: Joel Dice <[email protected]>

* make `thread.yield` a no-op in non-blocking contexts

Per the proposed spec changes, `thread.yield` should return control to the guest
immediately without allowing any other thread to run. Similarly, when an
async-lifted export or callback returns `CALLBACK_CODE_YIELD`, we should call
the callback again immediately without allowing another thread to run.

Signed-off-by: Joel Dice <[email protected]>

* fix build when `component-model-async` feature disabled

Signed-off-by: Joel Dice <[email protected]>

* fix more test regressions

Signed-off-by: Joel Dice <[email protected]>

* fix more test regressions

Note that this temporarily updates the `tests/component-model` submodule to the
branch for https://github.com/WebAssembly/component-model/pull/577 until that PR
is merged.

Signed-off-by: Joel Dice <[email protected]>

* tweak `Trap::CannotBlockSyncTask` message

This clarifies that such a task cannot block prior to returning.

Signed-off-by: Joel Dice <[email protected]>

* fix cancel_host_future test

Signed-off-by: Joel Dice <[email protected]>

* trap sync-lowered, guest->guest async calls in sync tasks

I somehow forgot to address this earlier. Thanks to Luke for catching this.

Note that this commit doesn't include test coverage, but Luke's forthecoming
tests in the `component-model` repo will cover it, and we'll pull that in with
the next submodule update.

Signed-off-by: Joel Dice <[email protected]>

* switch back to `main` branch of `component-model` repo

...and skip or `should_fail` the tests that won't pass until
https://github.com/WebAssembly/component-model/pull/578 is merged.

Signed-off-by: Joel Dice <[email protected]>

* add `trap-if-block-and-sync.wast`

We'll remove this again in favor of the upstream version once
https://github.com/WebAssembly/component-model/pull/578 has been merged.

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

- Assert that `StoreOpaque::suspend` is not called in a non-blocking context except in specific circumstances

- Typecheck async-ness for dynamic host functions

- Use type parameter instead of value parameter in `call_host[_dynamic]`

Signed-off-by: Joel Dice <[email protected]>

* add explanation comments to `check_blocking` calls

Signed-off-by: Joel Dice <[email protected]>

* fix fuzz test oracle for async functions

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

fix a couple of partial read/write bugs (#11981)

* reset read/write state back to `Open` on event delivery

If one end of a stream does a partial read or write, we leave the other end in a
`GuestRea

fix a couple of partial read/write bugs (#11981)

* reset read/write state back to `Open` on event delivery

If one end of a stream does a partial read or write, we leave the other end in a
`GuestReady` state, allowing further reads or writes to proceed until the buffer
has been drained or filled, respectively. However, once we've delivered the
event regarding the partial operation, we need to set the state back to `Open`,
since we'll have released the buffer back to the guest at that point.

Signed-off-by: Joel Dice <[email protected]>

* delay returning `Dropped` until producer buffer drained

If the `StreamProducer` calls `Destination::set_buffer`, we need to make sure
all the items in that buffer have been delivered to the receiver (or the
receiver closes its end) before telling it the write end has been dropped.

Signed-off-by: Joel Dice <[email protected]>

* add short reads tests

These cover a couple of scenarios where the guest and/or host read owned
resource items one-at-a-time from writes of more than one item, forcing the
writer to re-take ownership of the unwritten items between writes.

This also covers the case where the host's `StreamProducer` returns
`StreamResult::Dropped` after calling `Destination::set_buffer`, in which case
Wasmtime must delay telling the other end about the dropped stream until that
buffer has been drained.

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Implement `backpressure.{inc,dec}` (#11661)

Added to the async specification in WebAssembly/component-model#560
these are minor adaptations to the preexisting `backpressure.set`
intrinsic and are in

Implement `backpressure.{inc,dec}` (#11661)

Added to the async specification in WebAssembly/component-model#560
these are minor adaptations to the preexisting `backpressure.set`
intrinsic and are intended to replace it. The `backpressure.set`
intrinsic will remain until tooling propagates to understand
`backpressure.{inc,dec}`.

show more ...

fix panic in `Instance::set_consumer` when write end is already dropped (#11669)

* fix panic in `Instance::set_consumer` when write end is already dropped

In this case, we can drop the whole stream

fix panic in `Instance::set_consumer` when write end is already dropped (#11669)

* fix panic in `Instance::set_consumer` when write end is already dropped

In this case, we can drop the whole stream or future immediately since there's
nothing left to do with it.

Fixes #11621

Signed-off-by: Joel Dice <[email protected]>

* add test for piping from a stream whose write end is already dropped

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

return `TaskExit` future from `[Typed]Func::call_concurrent` (#11662)

* return `TaskExit` future from `[Typed]Func::call_concurrent`

In addition to returning the value produced by the callee, these

return `TaskExit` future from `[Typed]Func::call_concurrent` (#11662)

* return `TaskExit` future from `[Typed]Func::call_concurrent`

In addition to returning the value produced by the callee, these functions now
also return a `TaskExit` future which resolves once the subtask (and any
transitively-created subtasks) have exited. This partially addresses #11600;
the next step will be to add a `wasmtime-wit-bindgen` option to expose the
`TaskExit` value in generated bindings.

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

`TaskExit` now has an `async fn block` instead of closing over an `impl
AsAccessor` and implementing `Future`.

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

support and test synchronous `{stream,future}.cancel-{read,write}` (#11645)

* support and test synchronous `{stream,future}.cancel-{read,write}`

Previously, we only supported async calls to those i

support and test synchronous `{stream,future}.cancel-{read,write}` (#11645)

* support and test synchronous `{stream,future}.cancel-{read,write}`

Previously, we only supported async calls to those intrinsics; now we support
blocking, synchronous calls as well.

Signed-off-by: Joel Dice <[email protected]>

* update future-read.wast test

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

support non-async `{stream,future}.cancel-{read,write}` (#11625)

* support non-async `{stream,future}.cancel-{read,write}`

During my earlier stream API refactoring, I had forgotten to support or te

support non-async `{stream,future}.cancel-{read,write}` (#11625)

* support non-async `{stream,future}.cancel-{read,write}`

During my earlier stream API refactoring, I had forgotten to support or test
synchronous cancellation; this commit does both. In the process, I realized the
future API ought to be updated to support blocking cancellation just like the
stream API, so I made that change as well.

This also adds `{Source,Destination}::reborrow` functions, allowing instances of
those types to be reborrowed, such that they may be passed as parameters but
also used again.

Note that I had to move some functions from `impl ConcurrentState` to `impl
Instance` in order to access the store and suspend the current fiber when
synchronously cancelling.

Signed-off-by: Joel Dice <[email protected]>

* reduce code duplication

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendez

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendezvous"-style API to a callback-oriented one.

Previously you would create e.g. a `StreamReader`/`StreamWriter` pair and call
their `read` and `write` methods, respectively, and those methods would return
`Future`s that resolved when the operation was matched with a corresponding
`write` or `read` operation on the other end.

With the new API, you instead provide a `StreamProducer` trait implementation
whe creating the stream, whose `produce` method will be called as soon as a read
happens, giving the implementation a chance to respond immediately without
making the reader wait for a rendezvous. Likewise, you can match the read end
of a stream to a `StreamConsumer` to respond immediately to writes. This model
should reduce scheduling overhead and make it easier to e.g. pipe items to/from
`AsyncWrite`/`AsyncRead` or `Sink`/`Stream` implementations without needing to
explicitly spawn background tasks. In addition, the new API provides direct
access to guest read and write buffers for `stream<u8>` operations, enabling
zero-copy operations.

Other changes:

- I've removed the `HostTaskOutput`; we were using it to run extra code with
access to the store after a host task completes, but we can do that more
elegantly inside the future using `tls::get`. This also allowed me to
simplify `Instance::poll_until` a bit.

- I've removed the `watch_{reader,writer}` functionality; it's not needed now
given that the runtime will automatically dispose of the producer or consumer
when the other end of the stream or future is closed -- no need for embedder
code to manage that.

- In order to make `UntypedWriteBuffer` `Send`, I had to wrap its raw pointer
`buf` field in a `SendSyncPtr`.

- I've removed `{Future,Stream}Writer` entirely and moved
`Instance::{future,stream}` to `{Future,Stream}Reader::new`, respectively.

- I've added a bounds check to the beginnings of `Instance::guest_read` and
`Instance::guest_write` so that we need not do it later in
`Guest{Source,Destination}::remaining`, meaning those functions can be
infallible.

Note that I haven't updated `wasmtime-wasi` yet to match; that will happen in
one or more follow-up commits.

Signed-off-by: Joel Dice <[email protected]>

* Add `Accessor::getter`, rename `with_data` to `with_getter`

* fixup bindgen invocation

Signed-off-by: Roman Volosatovs <[email protected]>

* add support for zero-length writes/reads to/from host

I've added a test to cover this; it also tests direct buffer access for
`stream<u8>`, which I realized I forgot to cover earlier. And of course there
was a bug :facepalm:.

Signed-off-by: Joel Dice <[email protected]>

* add `{Destination,Source}::remaining` methods

This can help `Stream{Producer,Consumer}` implementations determine how many
items to write or read, respectively.

Signed-off-by: Joel Dice <[email protected]>

* wasi: migrate sockets to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* tests: read the socket stream until EOF

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: account for cancellation

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: mostly ensure byte buffer cancellation-safety

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-filesystem: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3-sockets: mostly ensure byte buffer cancellation-safety

* p3-cli: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: limit maximum buffer size

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: remove reuseaddr test loop workaround

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: drive I/O in `when_ready`

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3: drive I/O in `when_ready`

* Refine `Stream{Producer,Consumer}` APIs

Per conversations last week with Roman, Alex, and Lann, I've updated these
traits to present a lower-level API based on `poll_{consume,produce}` functions
and have documented the implementation requirements for various scenarios which
have come up in `wasmtime-wasi`, particularly around graceful cancellation. See
the doc comments for those functions for details.

Signed-off-by: Joel Dice <[email protected]>

* being integration of new API

Signed-off-by: Roman Volosatovs <[email protected]>

* update wasi/src/p3/filesystem to use new stream API

This is totally untested so far; I'll run the tests once we have everything else
compiling.

Signed-off-by: Joel Dice <[email protected]>

* update wasi/src/p3/cli to use new stream API

This is totally untested and doesn't even compile yet due to a lifetime issue I
don't have time to address yet. I'll follow up later with a fix.

Signed-off-by: Joel Dice <[email protected]>

* fix: remove `'a` bound on `&self`

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:sockets` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:cli` adaptation

Note, that this removes the read optimization - let's get the
implementation complete first and optimize later

Signed-off-by: Roman Volosatovs <[email protected]>

* remove redundant loop in sockets

Signed-off-by: Roman Volosatovs <[email protected]>

* wasi: buffer on 0-length reads

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:filesystem` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* remove `MAX_BUFFER_CAPACITY`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor `Cursor` usage

Signed-off-by: Roman Volosatovs <[email protected]>

* impl Default for VecBuffer

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: use consistent import styling

Signed-off-by: Roman Volosatovs <[email protected]>

* feature-gate fs Arc accessors

Signed-off-by: Roman Volosatovs <[email protected]>

* Update test expectations

---------

Signed-off-by: Joel Dice <[email protected]>
Signed-off-by: Roman Volosatovs <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>
Co-authored-by: Roman Volosatovs <[email protected]>

show more ...

fix sending default value when closing host-owned future writer (#11361)

* fix sending default value when closing host-owned future writer

My earlier commit didn't handle all the cases, which cause

fix sending default value when closing host-owned future writer (#11361)

* fix sending default value when closing host-owned future writer

My earlier commit didn't handle all the cases, which caused a regression for the
wasi-http tests in the `wasip3-prototyping` repo.

Signed-off-by: Joel Dice <[email protected]>

* add test for specific close-future-writer-with-default-value scenario

This covers the case where we're closing the future write end from the host
without having written a value and while the guest has already started a read
(which I fixed in the previous commit).

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

move waitable to original set after moving it to temporary one (#11357)

There are a few places in `concurrent.rs` where we use
`GuestTask::sync_call_set` to wait on waitables during synchronous call

move waitable to original set after moving it to temporary one (#11357)

There are a few places in `concurrent.rs` where we use
`GuestTask::sync_call_set` to wait on waitables during synchronous calls.
However, they may have been members of another set before we joined them to
`sync_call_set`, in which case we need to move them back when we're done (or at
least remove them from `sync_call_set`).

Prior to this fix, we would panic when dropping a task which had subtasks which
had been synchronously cancelled. I've updated `async_cancel_callee.rs` to
cover that case.

Signed-off-by: Joel Dice <[email protected]>

show more ...

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` m

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` macro. The main goal of this redesign is to
better support WASIp3 and component model async functions. Prior to this
redesign there was a mish mash of mechanisms to configure behavior of
imports/exports:

* The `async` configuration could turn everything async, nothing async,
only some imports async, or everything except some imports async.

* The `concurrent_{imports,exports}` keys were required to explicitly
opt-in to component model async signatures and applied to all
imports/exports.

* The `trappable_imports` configuration would indicate a list of imports
allowed to trap and it had special configuration for everything,
nothing, and only a certain list.

* The `tracing` and `verbose_tracing` keys could be applied to either
nothing or all functions.

Overall the previous state of configuration in `bindgen!` was clearly a
hodgepodge of systems that organically grew over time. In my personal
opinion it was in dire need of a refresh to take into account how
component-model-async ended up being implemented as well as
consolidating the one-off systems amongst all of these configuration
keys. A major motivation of this redesign, for example, was to inherit
behavior from WIT files by default. An `async` function in WIT should
not require `concurrent_*` keys to be configured, but rather it should
generate correct bindings by default.

In this commit, all of the above keys were removed. All keys have been
replaced with `imports` and `exports` configuration keys. Each behaves
the same way and looks like so:

bindgen!({
// ...
imports: {
// enable tracing for just this function
"my:local/interface/func": tracing,

// enable verbose tracing for just this function
"my:local/interface/other-func": tracing | verbose_tracing,

// this is blocking in WIT, but generate async bindings for
// it
"my:local/interface/[method]io.block": async,

// like above, but use "concurrent" bindings which have
// access to the store.
"my:local/interface/[method]io.block-again": async | store,

// everything else is, by default, trappable
default: trappable,
},
});

Effectively all the function-level configuration items are now bitflags.
These bitflags are by default inherited from the WIT files itself (e.g.
`async` functions are `async | store` by default). Further configuration
is then layered on top at the desires of the embedder. Supported keys are:

* `async` - this means that a Rust-level `async` function should be
generated. This is either `CallStyle::Async` or
`CallStyle::Concurrent` as it was prior, depending on ...

* `store` - this means that the generated function will have access to
the store on the host. This is only implemented right now for `async |
store` functions which map to `CallStyle::Concurrent`. In the future
I'd like to support just-`store` functions which means that you could
define a synchronous function with access to the store in addition to
an asynchronous function.

* `trappable` - this means that the function returns a
`wasmtime::Result<TheWitBindingType>`. If `trappable_errors` is
applicable then it means just a `Result<TheWitOkType,
TrappableErrorType>` is returned (like before)

* `tracing` - this enables `tracing!` integration for this function.

* `verbose_tracing` - this logs all argument values for this function
(including lists).

* `ignore_wit` - this ignores the WIT-level defaults of the function
(e.g. ignoring WIT `async`).

The way this then works is all modeled is that for any WIT function
being generated there are a set of flags associated with that function.
To calculate the flags the algorithm looks like:

1. Find the first matching rule in the `imports` or `exports` map
depending on if the function is imported or exported. If there is no
matching rule then use the `default` rule if present. This is the
initial set of flags for the function (or empty if nothing was
found).

2. If `ignore_wit` is present, return the flags from step 1. Otherwise
add in `async | store` if the function is `async` in WIT.

The resulting set of flags are then used to control how everything is
generated. For example the same split traits of today are still
generated and it's controlled based on the flags. Note though that the
previous `HostConcurrent` trait was renamed to `HostWithStore` to make
space for synchronous functions in this trait in the future too.

The end result of all these changes is that configuring imports/exports
now uses the exact same selection system as the `with` replacement map,
meaning there's only one system of selecting functions instead of 3.
WIT-level `async` is now respected by default meaning that bindings work
by default without further need to configure anything (unless more
functionality is desired).

One final minor change made here as well is that auto-generated
`instantiate` methods are now always synchronous and an
`instantiate_async` method is unconditionally generated for async mode.
This means that bindings always generate both functions and it's up to
the embedder to choose the appropriate one.

Closes #11246
Closes #11247

* Update expanded test expectations

prtest:full

* Fix the min platform embedding example

* Fix doc tests

* Always generate `*WithStore` traits

This helps when using the `with` mapping since that can always assume
that `HostWithStore` is available in the generated bindings, avoiding
the need to duplicate configuration options.

* Update test expectations

* Review comments

show more ...

add Component Model async ABI tests (#11136)

* add Component Model async ABI tests

This pulls in the tests from the `wasip3-prototyping` repo, minus the ones
requiring WASIp3 support in `wasmtime-w

add Component Model async ABI tests (#11136)

* add Component Model async ABI tests

This pulls in the tests from the `wasip3-prototyping` repo, minus the ones
requiring WASIp3 support in `wasmtime-wasi[-http]`, which will be PR'd
separately.

* add audits and exemptions for new `component-async-tests` deps

In order to convince `cargo vet` that we only needed these deps to be
`safe-to-run` (not necessarily `safe-to-deploy`, since it's test code), I've
moved the `wasm-compose` dep to the `dev-dependencies` section of the
`Cargo.toml` file, which required rearranging some code.

I've exempted `wasm-compose` since it's a BA project, and also exempted all but
one of the remaining new deps since they each get well over 10,000 downloads per
day from crates.io. I've audited and certified the remaining dep, `im-rc`,
which came in a bit shy of the 10,000-per-day mark.

Signed-off-by: Joel Dice <[email protected]>

* simplify `component_async_tests::util::sleep`

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...