Add support for map type (#12216)

* Add support for map type

Signed-off-by: Yordis Prieto <[email protected]>

* Add Map and MapEntry classes to support key/value pairs in component model

Th

Add support for map type (#12216)

* Add support for map type

Signed-off-by: Yordis Prieto <[email protected]>

* Add Map and MapEntry classes to support key/value pairs in component model

This commit introduces the Map and MapEntry classes, enabling the representation of map values in the component model. The Map class allows for the creation and iteration of key/value pairs, enhancing the functionality of the wasmtime component API. Additionally, the .gitignore file is updated to exclude build artifacts from the crates/c-api directory.

* Add wasm_component_model_map configuration support

* Format code

* Format C code

* Enhance component model to support HashMap<K, V> type

This commit introduces support for HashMap<K, V> in the component model, allowing maps to be represented as list<tuple<K, V>> in the canonical ABI. It includes implementations for the ComponentType, Lower, and Lift traits for HashMap, enabling type checking, lowering to flat representations, and lifting from memory. Additionally, the maximum depth for type generation in the fuzzing utility is updated to accommodate the new map type.

* Refactor component configuration to introduce map support

This commit removes the previous wasm features configuration and adds new functions for creating a map-configured engine. The `map_config` and `map_engine` functions are introduced to facilitate the use of the component model with maps in tests, ensuring that the engine is properly configured for map types in the component model.

* Add new WAST test for map types and remove map type definitions from existing tests

This commit introduces a new WAST test file specifically for testing various map types in the component model. Additionally, it removes the redundant map type definitions from the existing types.wast file to streamline the test suite.

* Update component fuzzing and dynamic tests to replace call_and_post_return with call

* Format code

* Refactor HashMap usage in typed.rs to use wasmtime_environ collections

* Fix HashMap initialization and insertion to handle potential errors in typed.rs

* Refactor HashMap handling in typed.rs to use lower_map_iter for improved iteration and memory management. Introduce new implementations for ComponentType, Lower, and Lift traits for std::collections::HashMap, enhancing support for map types in the component model.

* Fix map adapter trampoline compilation and alignment bugs

The translate_map function had two categories of bugs preventing map
adapter trampolines from working:

1. Wasm stack discipline: local_set_new_tmp emits LocalSet which pops
from the stack, but was called when the stack was empty (to
"pre-allocate" locals). Fixed by computing values first, then
calling local_set_new_tmp to consume them—matching translate_list's
pattern. Also removed an erroneous LocalTee that left an orphan
value on the stack. Affected: src_byte_len, dst_byte_len,
cur_src_ptr, cur_dst_ptr.

2. Pointer advancement: after value translation, the pointer still
points at the value start. The code only advanced by trailing
padding instead of value_size + trailing_padding, causing every
loop iteration to re-read the same memory.

Also fixes entry layout to use proper record alignment rules (entry
align = max(key_align, value_align), value at aligned offset).

* Refactor map entry layout calculations to use canonical ABI

* Remove unnecessary clone of map pairs during lowering

Val::Map already holds Vec<(Val, Val)> which derefs to &[(Val, Val)],
matching lower_map's signature directly. The intermediate Vec allocation
and deep clone of every key/value pair was redundant.

* Deduplicate map lift logic between HashMap implementations

* Deduplicate list and map sequence translation scaffolding

* Fix cargo fmt formatting issues

* Deduplicate map typecheck logic

* Deduplicate map lowering with linear_lower_map_to_flat and linear_lower_map_to_memory helpers

* Clean up lift_try_map: use drop, move TryHashMap import to module scope

* Fix CI: arbtest overflow and no-std HashMap lift_map

- component_fuzz: use saturating_sub in generate_hashable_key to prevent
underflow when fuel is 0 and Enum variant is chosen
- typed: remove incorrect ? operators in lift_map for hashbrown::HashMap
(with_capacity and insert don't return Result)

* Store map tuple layout in TypeMap

Compute map entry ABI and value offsets once during type building, and reuse that metadata in runtime map lift/lower paths instead of recalculating tuple layout at each call site.

* Refactor map ABI argument passing

Bundle map lift/lower layout and type metadata into a small MapAbi32 helper so map helper calls stay concise without changing behavior.

* Fix CI: enable component_model_map in fuzzing and handle map in arbitrary_val

The fuzzer's component_api oracle was generating map types but the engine
didn't have the map feature enabled, and arbitrary_val had no arm for
Type::Map. Enable component_model_map in the store helper (matching how
component_model_async is forced on) and implement arbitrary value generation
for map types.

---------

Signed-off-by: Yordis Prieto <[email protected]>

show more ...

Fix some minor `bindgen!` issues with `anyhow: true` (#12657)

* Fix some minor `bindgen!` issues with `anyhow: true`

Found when updating Spin to Wasmtime 42.0.0 where some cases weren't
handled:

*

Fix some minor `bindgen!` issues with `anyhow: true` (#12657)

* Fix some minor `bindgen!` issues with `anyhow: true`

Found when updating Spin to Wasmtime 42.0.0 where some cases weren't
handled:

* Trappable functions with no result.
* Resource destructors.
* Functions with custom trappable errors.

* Shuffle some features

show more ...

Remove subtask reparenting (#12570)

This commit updates the implementation of component-model-async
primitives to remove the manual subtask reparenting process. This is
required to fix #12544 at a s

Remove subtask reparenting (#12570)

This commit updates the implementation of component-model-async
primitives to remove the manual subtask reparenting process. This is
required to fix #12544 at a semantic level because a subtask isn't ever
actually reparented, even if its parent exits. The first part of this
change is to remove the `GuestTask::subtasks` field and all relevant
manipulations of it.

This field, however, powered the `TaskExit` abstraction returned from
`call_concurrent`. This commit then subsequently deletes `TaskExit` and
all related infrastructure as it's no longer directly applicable as-is.
The rest of this change is then updating tests/bindings/etc to account
for these two changes.

The main semantic changes related to tests are:

* `wasmtime run`, with and without `--invoke`, no longer waits for all
subtasks. This instead only waits for the main task returning before
exiting. Whether or not this is the correct behavior is under
discussion in WebAssembly/component-model#608

* `wasmtime serve` has been updated to keep the store alive at least
until the response body has been fully transmitted. This is also part
of WebAssembly/component-model#608.

* Some `component-async-tests`-related tests were updated to either
avoid blocking the store as it wasn't needed or yield enough times to
ensure that the test passes.

Closes #12544

prtest:full

show more ...

Update to wasm-tools 245 (#12571)

* Intrinsics updates

* Remove ds_store

* Update to wasm-tools 245

* Add vets

* Fixup tests

* Add missing feature for `indexmap`

---------

Co-authored-by: Sy

Update to wasm-tools 245 (#12571)

* Intrinsics updates

* Remove ds_store

* Update to wasm-tools 245

* Add vets

* Fixup tests

* Add missing feature for `indexmap`

---------

Co-authored-by: Sy Brand <[email protected]>

show more ...

deprecate `[Typed]Func::post_return[_async]` and make them no-ops (#12498)

* deprecate `[Typed]Func::post_return[_async]` and make them no-ops

With the advent of the Component Model concurrency ABI

deprecate `[Typed]Func::post_return[_async]` and make them no-ops (#12498)

* deprecate `[Typed]Func::post_return[_async]` and make them no-ops

With the advent of the Component Model concurrency ABI and it's `task.return`
intrinsic, post-return functions have been informally deprecated and are
expected to be removed for WASI 1.0 and the corresponding stable edition of the
Component Model. Consequently, it does not make sense anymore to require
embedders to explicitly call the post-return function after using
`[Typed]Func::call[_async]`.

As of this commit, `[Typed]Func::post_return[_async]` are no-ops. Instead, the
post-return function is called automatically as part of
`[Typed]Func::call[_async]` if present, which is how
`[Typed]Func::call_concurrent` has worked all along. In addition, this commit
fixes and tests a couple of cases where the task and/or thread was being
disposed of before the post-return function was called.

* address review feedback

* test post-return function in more scenarios

Specifically, I've split the `invoke_post_return` test into multiple tests:

- using `TypedFunc::call`
- using `TypedFunc::call_async` with concurrency support enabled
- using `TypedFunc::call_async` with concurrency support disabled
- using `Func::call_async` with concurrency support disabled
- using `TypedFunc::call_concurrent`

* remove GCC/clang-specific deprecation attribute

This broke the MSVC build.

* bless bindgen output

* remove obsolete post-return functions and fields

Now that post-return calls are handled internally without requiring explicit
action by the embedder, we can avoid unnecessary bookkeeping.

show more ...

Forbid rustdoc warnings in CI (#12420)

* Forbid rustdoc warnings in CI

This commit corrects our handling of rustdoc flags in CI to ensure that
warnings indeed fire. Additionally this changes our fl

Forbid rustdoc warnings in CI (#12420)

* Forbid rustdoc warnings in CI

This commit corrects our handling of rustdoc flags in CI to ensure that
warnings indeed fire. Additionally this changes our flags to pass
`-Dwarnings` to ensure that we have warning-free doc builds when all
features are enabled at least.

There were quite a lot of preexisting issues to fix, so this
additionally goes through and fixes all the warnings that cropped up.

* Update nightly toolchain again

prtest:full

* Update another nightly

* Fix a warning in generated code

show more ...

refactor recursive reentrance checks (#12349)

* refactor recursive reentrance checks

This commit makes a few changes related to recursive reentrance checks, instance
poisoning, etc.:

- Implements

refactor recursive reentrance checks (#12349)

* refactor recursive reentrance checks

This commit makes a few changes related to recursive reentrance checks, instance
poisoning, etc.:

- Implements the more restrictive lift/lower rules described in https://github.com/WebAssembly/component-model/pull/589 such that a component instance may not lower a function lifted by one of its ancestors, nor vice-versa. Any such lower will result in a fused adapter which traps unconditionally, preventing guest-to-guest recursive reentrance without requiring data flow analysis.
- Note that this required updating several WAST tests which were violating the new rule, including some in the `tests/component-model` Git submodule, which I've updated.
- This is handled entirely in the `fact` module now; I've removed the `AlwaysTrap` case previously handled by `wasmtime-cranelift`.
- Removes `FLAG_MAY_ENTER` from `InstanceFlags`. It is no longer needed for guest-to-guest calls due to the above, and for guest-to-host-to-guest calls we can rely on either `FLAG_NEEDS_POST_RETURN` for sync-lifted functions or the `GuestTask` call stack for async-lifted functions.
- Adds a `StoreOpaque::trapped` field which is set when _any_ instance belonging to that store traps, at which point the entire store is considered poisoned, meaning no instance belonging to it may be entered. This prevents indeterminant concurrent task state left over from the trapping instance from leaking into other instances.

Note that this does _not_ include code to push and pop `GuestTask` instances for
guest-to-guest sync-to-sync calls, nor for host-to-guest calls using e.g. the
synchronous `Func::call` API, so certain intrinsics which expect a `GuestTask`
to be present such as `backpressure.inc` will still fail in such cases. I'll
address that in a later PR.

Also note that I made a small change to `wasmtime-wit-bindgen`, adding a `Send`
bound on the `T` type parameter for `store | async` functions. This allowed me
to recursively call `{Typed}Func::call_concurrent` from inside a host function,
and it doesn't have any downsides AFAICT.

Fixes #12128

* bless bindgen expansions

* bless disas tests

* address review feedback

* sync `trap.h` with `trap_encoding.rs`

...and add const assertions to `trap.rs` to help avoid future divergence.

show more ...

Fix generated code for `anyhow: true` in wit-bindgen (#12334)

* Fix generated code for `anyhow: true` in wit-bindgen

The trait was still declared as using `wasmtime::Result`. It now uses
`wasmtime:

Fix generated code for `anyhow: true` in wit-bindgen (#12334)

* Fix generated code for `anyhow: true` in wit-bindgen

The trait was still declared as using `wasmtime::Result`. It now uses
`wasmtime::anyhow::Result` and we re-export `anyhow` through `wasmtime::anyhow`
-- with `doc(hidden)` since it is only for macro-generated code -- when the
`anyhow` cargo feature is enabled.

* Fix macro expansion test expectations

show more ...

`wit-bindgen`: Add an option to use `anyhow::Result` instead of `wasmtime::Result` (#12331)

* wit-bindgen: Add an option to use `anyhow::Result` instead of `wasmtime::Result`

* Add option to refere

`wit-bindgen`: Add an option to use `anyhow::Result` instead of `wasmtime::Result` (#12331)

* wit-bindgen: Add an option to use `anyhow::Result` instead of `wasmtime::Result`

* Add option to reference docs

* Rename `anyhow` dev dependency to `anyhow-for-testing`

show more ...

Update wasm-tools dependencies (#12254)

* Update wasm-tools dependencies

Brings in binary-parsing support for the WIT `map` type as well as the
wasm compact-imports proposal. Neither of these are e

Update wasm-tools dependencies (#12254)

* Update wasm-tools dependencies

Brings in binary-parsing support for the WIT `map` type as well as the
wasm compact-imports proposal. Neither of these are enabled by default
and will continue to be rejected, but it'll now be possible to support
them without needing to update dependencies.

* Add vets

* Update test expectations, add tests

Closes #12166

show more ...

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtim

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtime_environ::error` as the `wasmtime::error`
module, updates the prelude to include these new error-handling types, redirects
our top-level `wasmtime::{Error, Result}` re-exports to re-export
`wasmtime::error::{Error, Result}`, and updates various use sites that were
directly using `anyhow` to use the new `wasmtime` versions.

This process also required updating the component macro and wit-bindgen macro to
use the new error types instead of `anyhow`.

Part of https://github.com/bytecodealliance/wasmtime/issues/12069

* Replace wasmtime::error::Thing with wasmtime::Thing where it makes sense

* cargo fmt

* Move `crate::error::Thing` to `crate::Thing` where it makes sense

show more ...

Add a new fuzzer focused on component-model-async events (#12119)

* Add a new fuzzer focused on component-model-async events

This commit adds a new fuzzer mode to the `misc` fuzzer of Wasmtime
whic

Add a new fuzzer focused on component-model-async events (#12119)

* Add a new fuzzer focused on component-model-async events

This commit adds a new fuzzer mode to the `misc` fuzzer of Wasmtime
which is focused on async events and interleavings of components using
the component-model-async proposal. This fuzzer works by having a
precompiled guest program which serves as the component to run. This
precompiled component has a custom `fuzz.wit` which is used to interface
with the fuzzer itself. The fuzzer is then a fuzz-generated sequence of
commands to send to the component which verifies that everything
executes correctly, has no panics, etc.

This fuzzer intends to stress async communication and task
infrastructure with component-model-async. Notably this does not stress
lifting/lowering or arbitrary type signatures. This does, however,
permute all of the following:

* Guest/host interactions (also guest/guest, host/host, etc).
* Async functions, both ready and pending.
* Future operations: reads, writes, cancellation, transfers, etc.
* Stream operations: reads, writes, cancellation, transfers, etc.

This is all throwing into a large "soup" and then asserted to work
correctly. There's a few gotchas here and there for how this fuzzer is
designed, such as some events requiring "yield N times to await this
event happening". This is required because Wasmtime is allowed to
non-deterministically select between a number of "ready events" and what
to dispatch.

This is not intended to be a one-size-fits-all fuzzer for
component-model-async. The recent enhancements to the `component_api`
fuzzer are intended to complement this fuzzer in terms of what's
stressed where internally.

* Review comments

show more ...

Relax required host capabilities in wasip3 tcp/udp bindings (#12085)

Don't require `async | store` when it's not necessary as this'll soon
have ramifications on the type of the function being bound

Relax required host capabilities in wasip3 tcp/udp bindings (#12085)

Don't require `async | store` when it's not necessary as this'll soon
have ramifications on the type of the function being bound in the
component model.

show more ...

Use `.` instead of `/` for interface members. (#11947)

* Use `.` instead of `/` for interface members.

In the `wasmtime::component::generate` macro, change the syntax for
referencing functions and

Use `.` instead of `/` for interface members. (#11947)

* Use `.` instead of `/` for interface members.

In the `wasmtime::component::generate` macro, change the syntax for
referencing functions and types inside interfaces to use `.` instead
of `/`.

For example, this changes strings like
`wasi:http/types/outgoing-body`
to
`wasi:http/types.outgoing-body`
.

This makes the syntax more consistent with the syntax of
[WIT `use` statements], which use `.` for this purpose.

And, it avoids an incompatibility with the future nested namespaces syntax
([��]), where the `/d` in `a:b/c/d` is for traversing a component export
rather than an interface member.

[WIT `use` statements]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md#wit-packages-and-use
[��]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md#gated-features

* Use the new syntax in more places.

* Revert changes to vendored WIT files.

* Revert more changes to vendored files.

* Update syntax in more places.

show more ...

Fix(wit_bindgen): use declared defaults to decide on `Send` Bounds in `Wastime::world_add_to_linker` (#11761)

* feat: make FonctionConfig.default pub(crate)

Signed-off-by: Antoine Lavandier <antoin

Fix(wit_bindgen): use declared defaults to decide on `Send` Bounds in `Wastime::world_add_to_linker` (#11761)

* feat: make FonctionConfig.default pub(crate)

Signed-off-by: Antoine Lavandier <[email protected]>

* fix: use declared imports/exports defaults when deciding on Send bounds

Signed-off-by: Antoine Lavandier <[email protected]>

* fix: add Send bound on relevant test output

Signed-off-by: Antoine Lavandier <[email protected]>

* fix(tests): add test using imports to define send bounds

Signed-off-by: Antoine Lavandier <[email protected]>

* fix(fmt): make rustfmt happy

Signed-off-by: Antoine Lavandier <[email protected]>

* fix(fmt): actually make rustfmt happy this time

Signed-off-by: Antoine Lavandier <[email protected]>

---------

Signed-off-by: Antoine Lavandier <[email protected]>

show more ...

add `task_exit` option to `wasmtime-wit-bindgen` (#11665)

This builds on #11662 by optionally exposing the `TaskExit` return value from
`[Typed]Func::call_concurrent` in the bindings generated for e

add `task_exit` option to `wasmtime-wit-bindgen` (#11665)

This builds on #11662 by optionally exposing the `TaskExit` return value from
`[Typed]Func::call_concurrent` in the bindings generated for exported functions.

Note that the first two commits are shared with #11662.

Fixes #11600

Signed-off-by: Joel Dice <[email protected]>

show more ...

return `TaskExit` future from `[Typed]Func::call_concurrent` (#11662)

* return `TaskExit` future from `[Typed]Func::call_concurrent`

In addition to returning the value produced by the callee, these

return `TaskExit` future from `[Typed]Func::call_concurrent` (#11662)

* return `TaskExit` future from `[Typed]Func::call_concurrent`

In addition to returning the value produced by the callee, these functions now
also return a `TaskExit` future which resolves once the subtask (and any
transitively-created subtasks) have exited. This partially addresses #11600;
the next step will be to add a `wasmtime-wit-bindgen` option to expose the
`TaskExit` value in generated bindings.

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

`TaskExit` now has an `async fn block` instead of closing over an `impl
AsAccessor` and implementing `Future`.

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Support store-access in `bindgen!` generated imports (#11628)

* Support store-access in `bindgen!` generated imports

This commit adds support to accessing the store in `bindgen!`-generated
import

Support store-access in `bindgen!` generated imports (#11628)

* Support store-access in `bindgen!` generated imports

This commit adds support to accessing the store in `bindgen!`-generated
import functions in traits. Since the inception of `bindgen!` this has
never been possible and access to the store requires manually working
with `Linker`, for example. This is not easy to do because it requires
surgically editing code or working around what bindings generation parts
you do want.

The implementation here is a small step away from what
component-model-async has already implemented for async functions.
Effectively it's a small extension of the `*WithStore` traits to also
have synchronous functions with `Access` parameters instead of `async`
functions with an `Accessor` parameter.

This is something we're going to want for the WASIp3 implementation
where I've noticed some resource destructors are going to want access to
the store to close out streams and such and this'll provide the bindings
necessary for that.

Closes #11590

* Update test expectations

show more ...

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendez

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendezvous"-style API to a callback-oriented one.

Previously you would create e.g. a `StreamReader`/`StreamWriter` pair and call
their `read` and `write` methods, respectively, and those methods would return
`Future`s that resolved when the operation was matched with a corresponding
`write` or `read` operation on the other end.

With the new API, you instead provide a `StreamProducer` trait implementation
whe creating the stream, whose `produce` method will be called as soon as a read
happens, giving the implementation a chance to respond immediately without
making the reader wait for a rendezvous. Likewise, you can match the read end
of a stream to a `StreamConsumer` to respond immediately to writes. This model
should reduce scheduling overhead and make it easier to e.g. pipe items to/from
`AsyncWrite`/`AsyncRead` or `Sink`/`Stream` implementations without needing to
explicitly spawn background tasks. In addition, the new API provides direct
access to guest read and write buffers for `stream<u8>` operations, enabling
zero-copy operations.

Other changes:

- I've removed the `HostTaskOutput`; we were using it to run extra code with
access to the store after a host task completes, but we can do that more
elegantly inside the future using `tls::get`. This also allowed me to
simplify `Instance::poll_until` a bit.

- I've removed the `watch_{reader,writer}` functionality; it's not needed now
given that the runtime will automatically dispose of the producer or consumer
when the other end of the stream or future is closed -- no need for embedder
code to manage that.

- In order to make `UntypedWriteBuffer` `Send`, I had to wrap its raw pointer
`buf` field in a `SendSyncPtr`.

- I've removed `{Future,Stream}Writer` entirely and moved
`Instance::{future,stream}` to `{Future,Stream}Reader::new`, respectively.

- I've added a bounds check to the beginnings of `Instance::guest_read` and
`Instance::guest_write` so that we need not do it later in
`Guest{Source,Destination}::remaining`, meaning those functions can be
infallible.

Note that I haven't updated `wasmtime-wasi` yet to match; that will happen in
one or more follow-up commits.

Signed-off-by: Joel Dice <[email protected]>

* Add `Accessor::getter`, rename `with_data` to `with_getter`

* fixup bindgen invocation

Signed-off-by: Roman Volosatovs <[email protected]>

* add support for zero-length writes/reads to/from host

I've added a test to cover this; it also tests direct buffer access for
`stream<u8>`, which I realized I forgot to cover earlier. And of course there
was a bug :facepalm:.

Signed-off-by: Joel Dice <[email protected]>

* add `{Destination,Source}::remaining` methods

This can help `Stream{Producer,Consumer}` implementations determine how many
items to write or read, respectively.

Signed-off-by: Joel Dice <[email protected]>

* wasi: migrate sockets to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* tests: read the socket stream until EOF

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: account for cancellation

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: mostly ensure byte buffer cancellation-safety

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-filesystem: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3-sockets: mostly ensure byte buffer cancellation-safety

* p3-cli: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: limit maximum buffer size

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: remove reuseaddr test loop workaround

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: drive I/O in `when_ready`

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3: drive I/O in `when_ready`

* Refine `Stream{Producer,Consumer}` APIs

Per conversations last week with Roman, Alex, and Lann, I've updated these
traits to present a lower-level API based on `poll_{consume,produce}` functions
and have documented the implementation requirements for various scenarios which
have come up in `wasmtime-wasi`, particularly around graceful cancellation. See
the doc comments for those functions for details.

Signed-off-by: Joel Dice <[email protected]>

* being integration of new API

Signed-off-by: Roman Volosatovs <[email protected]>

* update wasi/src/p3/filesystem to use new stream API

This is totally untested so far; I'll run the tests once we have everything else
compiling.

Signed-off-by: Joel Dice <[email protected]>

* update wasi/src/p3/cli to use new stream API

This is totally untested and doesn't even compile yet due to a lifetime issue I
don't have time to address yet. I'll follow up later with a fix.

Signed-off-by: Joel Dice <[email protected]>

* fix: remove `'a` bound on `&self`

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:sockets` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:cli` adaptation

Note, that this removes the read optimization - let's get the
implementation complete first and optimize later

Signed-off-by: Roman Volosatovs <[email protected]>

* remove redundant loop in sockets

Signed-off-by: Roman Volosatovs <[email protected]>

* wasi: buffer on 0-length reads

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:filesystem` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* remove `MAX_BUFFER_CAPACITY`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor `Cursor` usage

Signed-off-by: Roman Volosatovs <[email protected]>

* impl Default for VecBuffer

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: use consistent import styling

Signed-off-by: Roman Volosatovs <[email protected]>

* feature-gate fs Arc accessors

Signed-off-by: Roman Volosatovs <[email protected]>

* Update test expectations

---------

Signed-off-by: Joel Dice <[email protected]>
Signed-off-by: Roman Volosatovs <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>
Co-authored-by: Roman Volosatovs <[email protected]>

show more ...

Use `trappable_error_type` in WASIp3 sockets (#11373)

This commit updates to using the `trappable_error_type` in bindings
generated for WASIp3 to mirror what happens in WASIp2, removing an extra
`Re

Use `trappable_error_type` in WASIp3 sockets (#11373)

This commit updates to using the `trappable_error_type` in bindings
generated for WASIp3 to mirror what happens in WASIp2, removing an extra
`Result<Result<..>>` layer to only have one layer of results.

show more ...

docs: fix typos (#11358)

Account for concurrent resource destructors (#11350)

* Account for concurrent resource destructors

This fixes a minor merge conflict between #11325 and #11328 which isn't
currently exercised by in-

Account for concurrent resource destructors (#11350)

* Account for concurrent resource destructors

This fixes a minor merge conflict between #11325 and #11328 which isn't
currently exercised by in-repo WASI bindings but will be soon once
wasip3-prototyping is finished merging.

* Update expanded test expectations

show more ...

refactor `{Stream,Future}|{Reader,Writer}` APIs and internals (#11325)

* refactor `{Stream,Future}|{Reader,Writer}` APIs and internals

This makes a several changes to how `{Stream,Future}|{Reader,W

refactor `{Stream,Future}|{Reader,Writer}` APIs and internals (#11325)

* refactor `{Stream,Future}|{Reader,Writer}` APIs and internals

This makes a several changes to how `{Stream,Future}|{Reader,Writer}` work to
make them more efficient and, in some ways, more ergonomic:

- The background tasks have been removed, allowing reads and writes to complete without task context switching. We now only allocate and use oneshot channels lazily when the other end is not yet ready; this improves real world performance benchmarks (e.g. wasi-http request handling) considerably.

- Instances of `{Stream,Future}Reader` can now be lifted and lowered directly; no need for `Host{Stream,Future}` anymore.

- The type parameter for `Stream{Reader,Writer}` no longer refers to the buffer type -- just the payload type (i.e. `StreamReader<u8>` instead of `StreamReader<Vec<u8>>`), meaning any buffer type may be used for a given read or write operation. This also means the compiler needs help with type inference less often when calling `Instance::stream`.

- Instances of `{Stream,Future}|{Reader,Writer}` now require access to the store in order to be disposed of properly. I've added RAII wrapper structs (`WithAccessor[AndValue]`) to help with this, and also updated `Store::drop` and `Instance::run_concurrent` to ensure the store thread-local is set when dropping futures closing over `&Accessor`s.

- In order to ensure that resources containing `{Stream,Future}|{Reader,Writer}` instances are disposed of properly, I've added `LinkerInstance::resource_concurrent` and have updated `wasmtime-wit-bindgen` to use it. This gives resource drop functions access to a `StoreContextMut` via an `Accessor`, allowing the stream and future handles to be disposed of.
- In order to make this work, I had to change `Accessor::instance` from a `Instance` to an `Option<Instance>`, which is awkward but temporary since we're planning to remove `Accessor::instance` entirely once we've moved concurrent state from `ComponentInstance` to `Store`.

That problem of disposal is definitely the most awkward part of all this. In
simple cases, it's easy enough to ensure that read and write handles are
disposed of properly, but both `wasmtime-wasi` and `wasmtime-wasi-http` have
some pretty complicated functions where handles are passed between tasks and/or
stored inside resources, so it can be tricky to ensure proper disposal on all
code paths. I'm open to ideas for improving this, but I suspect we'll need new
Rust language features (e.g. linear types) to make it truly ergonomic, robust,
and efficient.

While testing the above, I discovered an issue with `Instance::poll_until` such
that it would prematurely give up and return a "deadlock" trap error, believing
that there was no further work to do, even though the future passed to it was
ready to resolve the next time it was polled. I've fixed this by polling it one
last time and only trapping if it returns pending.

Note that I've moved a few associated functions from `ConcurrentState` to
`Instance` (e.g. `guest_drop_writable` and others) since they now need access to
the store; they're unchanged otherwise. Apologies for the diff noise.

Finally, I've tweaked how `wasmtime serve` to poll the guest for content before
handing the response to Hyper, which helps performance by ensuring the first
content chunk can be sent with the same TCP packet as the beginning of the
response.

Signed-off-by: Joel Dice <[email protected]>

fix wasi p3 build and test failures

Signed-off-by: Joel Dice <[email protected]>

use `ManuallyDrop` instead of `Option` in `Dropper`

This allows us to drop its `value` field in-place, i.e. without moving it,
thereby upholding the `Pin` guarantee.

Signed-off-by: Joel Dice <[email protected]>

address review comments

- Remove `DropWithStoreAndValue` and friends; go back to taking a `fn() -> T` parameter in `Instance::future` instead
- Make `DropWithStore::drop[_with]` take `&mut self` instead of `self`
- Make `WithAccessor` and `DropWithStore` private
- Instead, I've added public `Guarded{Stream,Future}{Reader,Writer}` types for RAII
- and also `{Stream,Future}{Reader,Writer}::close[_with]` methods
- Use RAII in `FutureReader::read` and `FutureWriter::write` to ensure handles are dropped if the `Future` is dropped

Signed-off-by: Joel Dice <[email protected]>

* lower host stream/future writes in background task

This avoids unsoundness due to guest realloc calls while there are host embedder
frames on the stack.

Signed-off-by: Joel Dice <[email protected]>

* fix `tcp.rs` regressions

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>

show more ...

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` m

Redesign function configuration in `bindgen!` (#11328)

* Redesign function configuration in `bindgen!`

This commit is a redesign of how function-level configuration works in
Wasmtime's `bindgen!` macro. The main goal of this redesign is to
better support WASIp3 and component model async functions. Prior to this
redesign there was a mish mash of mechanisms to configure behavior of
imports/exports:

* The `async` configuration could turn everything async, nothing async,
only some imports async, or everything except some imports async.

* The `concurrent_{imports,exports}` keys were required to explicitly
opt-in to component model async signatures and applied to all
imports/exports.

* The `trappable_imports` configuration would indicate a list of imports
allowed to trap and it had special configuration for everything,
nothing, and only a certain list.

* The `tracing` and `verbose_tracing` keys could be applied to either
nothing or all functions.

Overall the previous state of configuration in `bindgen!` was clearly a
hodgepodge of systems that organically grew over time. In my personal
opinion it was in dire need of a refresh to take into account how
component-model-async ended up being implemented as well as
consolidating the one-off systems amongst all of these configuration
keys. A major motivation of this redesign, for example, was to inherit
behavior from WIT files by default. An `async` function in WIT should
not require `concurrent_*` keys to be configured, but rather it should
generate correct bindings by default.

In this commit, all of the above keys were removed. All keys have been
replaced with `imports` and `exports` configuration keys. Each behaves
the same way and looks like so:

bindgen!({
// ...
imports: {
// enable tracing for just this function
"my:local/interface/func": tracing,

// enable verbose tracing for just this function
"my:local/interface/other-func": tracing | verbose_tracing,

// this is blocking in WIT, but generate async bindings for
// it
"my:local/interface/[method]io.block": async,

// like above, but use "concurrent" bindings which have
// access to the store.
"my:local/interface/[method]io.block-again": async | store,

// everything else is, by default, trappable
default: trappable,
},
});

Effectively all the function-level configuration items are now bitflags.
These bitflags are by default inherited from the WIT files itself (e.g.
`async` functions are `async | store` by default). Further configuration
is then layered on top at the desires of the embedder. Supported keys are:

* `async` - this means that a Rust-level `async` function should be
generated. This is either `CallStyle::Async` or
`CallStyle::Concurrent` as it was prior, depending on ...

* `store` - this means that the generated function will have access to
the store on the host. This is only implemented right now for `async |
store` functions which map to `CallStyle::Concurrent`. In the future
I'd like to support just-`store` functions which means that you could
define a synchronous function with access to the store in addition to
an asynchronous function.

* `trappable` - this means that the function returns a
`wasmtime::Result<TheWitBindingType>`. If `trappable_errors` is
applicable then it means just a `Result<TheWitOkType,
TrappableErrorType>` is returned (like before)

* `tracing` - this enables `tracing!` integration for this function.

* `verbose_tracing` - this logs all argument values for this function
(including lists).

* `ignore_wit` - this ignores the WIT-level defaults of the function
(e.g. ignoring WIT `async`).

The way this then works is all modeled is that for any WIT function
being generated there are a set of flags associated with that function.
To calculate the flags the algorithm looks like:

1. Find the first matching rule in the `imports` or `exports` map
depending on if the function is imported or exported. If there is no
matching rule then use the `default` rule if present. This is the
initial set of flags for the function (or empty if nothing was
found).

2. If `ignore_wit` is present, return the flags from step 1. Otherwise
add in `async | store` if the function is `async` in WIT.

The resulting set of flags are then used to control how everything is
generated. For example the same split traits of today are still
generated and it's controlled based on the flags. Note though that the
previous `HostConcurrent` trait was renamed to `HostWithStore` to make
space for synchronous functions in this trait in the future too.

The end result of all these changes is that configuring imports/exports
now uses the exact same selection system as the `with` replacement map,
meaning there's only one system of selecting functions instead of 3.
WIT-level `async` is now respected by default meaning that bindings work
by default without further need to configure anything (unless more
functionality is desired).

One final minor change made here as well is that auto-generated
`instantiate` methods are now always synchronous and an
`instantiate_async` method is unconditionally generated for async mode.
This means that bindings always generate both functions and it's up to
the embedder to choose the appropriate one.

Closes #11246
Closes #11247

* Update expanded test expectations

prtest:full

* Fix the min platform embedding example

* Fix doc tests

* Always generate `*WithStore` traits

This helps when using the `with` mapping since that can always assume
that `HostWithStore` is available in the generated bindings, avoiding
the need to duplicate configuration options.

* Update test expectations

* Review comments

show more ...

Start to use `&Accessor<T, D>` more in concurrent code (#11238)

* Start to use `&Accessor<T, D>` more in concurrent code

After discussion with Joel we've concluded that while `&mut Accessor<T, D>`

Start to use `&Accessor<T, D>` more in concurrent code (#11238)

* Start to use `&Accessor<T, D>` more in concurrent code

After discussion with Joel we've concluded that while `&mut Accessor<T, D>`
was originally added to model host functions it is also appropriate to
use it to model embedder-rooted invocations of items such as wasm as
well. Effectively the conclusion we reached was that
`*::call_concurrent` should be taking `&Accessor`, not
`StoreContextMut`. This has a number of benefits to it over the previous
iteration:

* This makes exports behave more like imports where `Accessor` means
"you're in the concurrent world".

* This makes exports have an `async fn` signature which is easier to
read and understand.

* This automatically enforces the guarantee that the returned future is
only polled within the main event loop because the future is always
considered to close over the `&Accessor` provided meaning it
statically cannot live outside of the event loop.

* This paves the way forward to future refactorings to avoid storing so
much state within a `Store<T>` and instead try to store state directly
in futures themselves. This should make cancellation more natural and
eventually also remove `'static` bounds on params/results. Furthermore
this should make it easier to avoid spawning tasks internally by
storing state in futures instead of spawned tasks.

In doing this one of the main questions we were faced with was what to
do about `&mut Accessor<T, D>`, namely the `mut` part. With a mutable
accessor it would be only possible to call one function concurrently.
One option considered was to add combinators like `Accessor::join` and
`Accessor::race` but in the end we decided to avoid going that direction
and instead switch to `&Accessor<T, D>` everywhere, freely enabling
aliasing of the accessor. This has the downside that `Accessor::with` is
now a relatively dangerous function in that it can panic, but idiomatic
usage of it is not expected to panic as the distinction between the
`async` and sync boundary of `Accessor` vs `StoreContextMut` is expected
to naturally make the recursive panic condition of `with` rare to come
up in practice.

Concrete changes in this commit are:

* `Accessor::with` now requires `&self`.
* `Accessor::spawn` now requires `&self`.
* Host functions are now given `&Accessor`, not `&mut Accessor`.
* `{Typed,}Func::call_concurrent` is now an `async fn` which takes an
`&Accessor` instead of `StoreContextMut`.
* Guest bindings generation for concurrent invocations now looks exactly
like async bindings generation except for replacing `StoreContextMut`
with `Accessor`.

Note that this commit does not yet update the internal implementations
of these functions to benefit from the new abilities that taking
`&Accessor` implies. Instead that's deferred to a future update as
necessary. Instead this is only updating the public API of the
`wasmtime` crate to enable these refactorings in the future.

Also note that this does not yet update all functions to take
`&Accessor`. Notably futures and streams still need to be updated.

cc #11224

* Review comments

---------

Co-authored-by: Joel Dice <[email protected]>

show more ...