Refactor `wasi:http` headers' host representation (#12754)

* Refactor `wasi:http` headers' host representation

This commit is a follow-on/extension of #12748 and extends the changes
made for WASIp2

Refactor `wasi:http` headers' host representation (#12754)

* Refactor `wasi:http` headers' host representation

This commit is a follow-on/extension of #12748 and extends the changes
made for WASIp2 headers in #12652 to the WASIp3 implementation as well.
This is done through a number of refactorings to make the WASIp2 and
WASIp3 implementations more similar in terms of how they represent
headers. Changes here are:

* `FieldMap` now has its own dedicated module at the crate root instead
of intermingling with other WASIp2 types.
* `FieldMap` is now internally-`Arc`'d and is cheaply clonable.
`FieldMap` itself now tracks whether it's mutable or immutable (WASI
semantics) and doesn't need different wrappers in WASIp2 and WASIp3.
* Creation of an immutable `FieldMap` can be done without needing a size
limit. Flagging a `FieldMap` as mutable, however, requires a size limit.
* `FieldMap::set` was added to be a bit more efficient w.r.t. clones.
* `FieldMapError` is a new error type that covers all of the possible
error modes of operating with a `FieldMap`. Conversions from this to
WASIp{2,3} `header-error` types are now implemented as well.
* WASIp2 now flags `header-error` as a trappable-error-type, allowing
the use of `?` in implementing header functions (like WASIp3).
* Much of WASIp2's header implementation was refactored with `?`, moving
methods around, shuffling where headers are made vs `FieldMap`, some
minor idioms, etc.
* WASIp3 no longer uses `MaybeMutable` for headers and instead uses
`FieldMap` directly.

cc #12674

* Clippy warnings

show more ...

Refactor WASIp2 `wasi:http` implementation (#12748)

* Sequester WASIp2 in `wasmtime-wasi-http` to a module

This mirrors the `wasmtime-wasi` crate's organization where there's a
`p2` module and a `

Refactor WASIp2 `wasi:http` implementation (#12748)

* Sequester WASIp2 in `wasmtime-wasi-http` to a module

This mirrors the `wasmtime-wasi` crate's organization where there's a
`p2` module and a `p3` module at the top level.

* Refactor WASIp2 `wasi:http` implementation

This commit reorganizes and refactors the WASIp2 implementation of
`wasi:http` to look more like other `wasmtime-wasi`-style interfaces.
Specifically the old `WasiHttpImpl<T>` structure is removed in favor of
as `WasiHttpCtxView<'_>` type that is used to implement
bindgen-generated `Host` traits. This necessitated reorganizing the
methods of the previous `WasiHttpView` trait like so:

* The `WasiHttpView` trait is renamed to `WasiHttpHooks` to make space
for a new `WasiHttpView` which behaves like `WasiView`, for example.

* The `ctx` and `table` methods of `WasiHttpHooks` were removed since
they'll be fields in `WasiHttpCtxView`.

* Helper methods for WASIp2 were moved to methods on `WasiHttpCtxView`
instead of default methods on `WasiHttpHooks`.

With these changes in place the WASIp3 organization was also updated
slightly as well. Notably WASIp3 now contains a reference to the crate's
`WasiHttpCtx` structure (which has field limits for example). WASIp3's
previous `WasiHttpCtx` trait is now renamed to `WasiHttpHooks` as well.
This means that there are two `WasiHttpHooks` traits right now, one for
WASIp2 and one for WASIp3. In the future I would like to unify these two
but that will require some more work around the default `send_request`.

A final note here is that the `WasiHttpHooks` trait previously, and
continues to be, optional for embedders to implement. Default functions
are provided as `wasmtime_wasi_http::{p2, p3}::default_hooks`.
Additionally there's a `Default for &mut dyn WasiHttpHooks`
implementation, too.

With all that outlined: the motivation for this change is to bring the
WASIp2 and WASIp3 implementations of `wasi:http` closer together. This
is inspired by refactorings I was doing for #12674 to apply the same
header limitations for WASIp3 as is done for WASIp2. Prior to this
change there were a number of differences such as WASIp3 not having
`crate::WasiHttpCtx` around, WASIp2 having a different organization of
structures/borrows, etc. The goal is to bring the two implementations
closer in line with each other to make refactoring across them more
consistent and easier.

* Make `WasiHttp` in WASIp2 public

* Fix some conditional build

* Fix some doctests

* Fix configured build

* Fixup documentation

show more ...

Forbid rustdoc warnings in CI (#12420)

* Forbid rustdoc warnings in CI

This commit corrects our handling of rustdoc flags in CI to ensure that
warnings indeed fire. Additionally this changes our fl

Forbid rustdoc warnings in CI (#12420)

* Forbid rustdoc warnings in CI

This commit corrects our handling of rustdoc flags in CI to ensure that
warnings indeed fire. Additionally this changes our flags to pass
`-Dwarnings` to ensure that we have warning-free doc builds when all
features are enabled at least.

There were quite a lot of preexisting issues to fix, so this
additionally goes through and fixes all the warnings that cropped up.

* Update nightly toolchain again

prtest:full

* Update another nightly

* Fix a warning in generated code

show more ...

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Migrate wasi-http to `wasmtime::error` (#12291)

* Migrate wiggle and wiggle generate to `wasmtime::error`

* Fix testing `wiggle` and `wiggle-test` in isolation

When doing plain old `cargo test -p

Migrate wasi-http to `wasmtime::error` (#12291)

* Migrate wiggle and wiggle generate to `wasmtime::error`

* Fix testing `wiggle` and `wiggle-test` in isolation

When doing plain old `cargo test -p wiggle` (or `wiggle-test`) the test would
fail due to linking errors because Wasmtime's `std` cargo feature (and maybe
others) wasn't being enabled. The crate's tests would pass when run as part of
the whole workspace, however, because of cargo feature resolution and other
crates that enabled the necessary features, which is why CI is green.

* Remove direct anyhow dependency in wiggle-test

* Migrate wasi-nn to `wasmtime::error`

* Migrate wasi-keyvalue to `wasmtime::error`

* Migrate wasi-io to `wasmtime::error`

* Migrate wasi-http to `wasmtime::error`

show more ...

support WASIp3 instance reuse in `wasmtime serve` (#11807)

* support WASIp3 instance reuse in `wasmtime serve`

This is a draft implementation of WASIp3 instance reuse. Previously, we used
one stor

support WASIp3 instance reuse in `wasmtime serve` (#11807)

* support WASIp3 instance reuse in `wasmtime serve`

This is a draft implementation of WASIp3 instance reuse. Previously, we used
one store and one instance per request for both p2 and p3 handlers, discarding
the store and instance immediately after the request was handled. Now we
attempt to reuse p3 instances and their stores instead, both serially and
concurrently.

Advantages of reuse:

- Higher throughput: We can amortize the time spent creating and disposing of instances and stores over a number of requests.
- Lower memory and address space usage: Concurrent instance reuse allows us to handle more requests with fewer instances, especially when the handler is I/O bound.
- Developers will presumably need to make changes to their applications anyway when migrating from p2 to p3, so this is an ideal time to change the default from no reuse to (some) reuse, assuming we want to do it at all.

Disadvantages of reuse:

- Reduced isolation: When the same instance is used to handle multiple requests, the blast radius of bugs in the guest is larger; a trap can affect unrelated requests, and a security flaw could leak state across requests.
- Host implementation complexity: There's more code to manage and more configuration knobs to tune.
- Guest implementation complexity: You can't just stash state in a global variable and call it a day. Similarly, shortcuts like using a leaking GC require additional thought.

Given the tradeoffs, we'll definitely need to provide components a way to opt
out of reuse if desired. See
https://github.com/WebAssembly/wasi-http/issues/190 for related discussion.

Implementation details:

I've moved `ProxyHandler` from `wasmtime_cli::commands::serve` to
`wasmtime_wasi_http::handler` (so it can be reused by custom embedders),
abstracted away the `serve`-specific parts, and added support for instance reuse
via a new `ProxyHandler::push` function. The `push` function takes a work item
representing an incoming request and dispatches it to a dynamically sized pool
of worker tasks, each with its own store and `wasi:http/[email protected]` instance.
Worker tasks accept work items as capacity allows until they either reach a
maximum total reuse count or hit an idle timeout, at which point they exit.

Performance:

I've run a few benchmarks using `wasmtime-serve-rps.sh` and
[hello-wasip3-http](https://github.com/dicej/hello-wasip3-http/blob/main/src/lib.rs)
(a simple "hello, world" request handler). Highlights:

- 45% more requests per second with instance reuse enabled vs. disabled
- 64% fewer concurrent instances required for an I/O-bound handler
- Here I added a 40ms delay to the "hello, world" handler to simulate I/O

TODOs:

- Support host-enforced request timeouts. Note that we can't simply use the epoch-based trap-on-timeout approach we've traditionally used since a given instance may be responsible for a number of concurrent requests. This will require adding a public API to the `wasmtime` crate for cleanly cancelling a guest task without affecting other ones.
- Add CLI options for the configuration knobs (e.g. max request count per instance, idle timeout, etc.)
- Use guest signals like `backpressure`, explicit `wasi:cli/exit/exit`, https://github.com/WebAssembly/wasi-http/issues/190, etc. to drive reuse decisions

Signed-off-by: Joel Dice <[email protected]>

* additional instance reuse support

This addresses a couple of TODO items from my previous instance reuse PR:

- Support request timeouts by gracefully shutting down the worker as soon as any request hits a timeout, letting any other pending tasks finish or time out before actually dropping the instance.
- Convert the previously hard-coded config options to CLI options.
- Enable profiling when instance reuse is enabled.

Signed-off-by: Joel Dice <[email protected]>

* start worker tasks more aggressively

This improves `wasmtime serve` performance when instance reuse is enabled by:

- spawning a worker task any time there are no idle workers available
- considering a worker to be unavailable until it has finished initializing itself

Signed-off-by: Joel Dice <[email protected]>

* revert unecessary wasmtime-serve-rps.sh changes

Signed-off-by: Joel Dice <[email protected]>

* enable WASIp2 instance reuse

Signed-off-by: Joel Dice <[email protected]>

* add clarifying comments about when and why we start worker tasks

Signed-off-by: Joel Dice <[email protected]>

* remove duplicated code in `ProxyHandler::spawn`

By bypassing the task queue when there are no already-available workers, I've
narrowed the performance gap between the "fast path" code and the normal path to
about 2%, which is close enough that we can remove the duplicated code.

Signed-off-by: Joel Dice <[email protected]>

* ensure timeouts enforced if event loop stalls

This fixes an issue such that timeouts weren't being reliably enforced when the
guest either busy loops for an extended time or makes a sync call to a host
function that takes exclusive access to the `Store` and blocks for a while. In
either of those cases, the `StoreContextMut::run_concurrent` event loop will
stall, being unable to make progress while the task fiber monopolizes the
`Store`. In this case, we must enforce timeouts _outside_ the event loop. See
the new code comments for details on how that is done.

This also tweaks `wasmtime serve` stderr/stdout output a bit to address
`cli_tests::cli_serve_*` test regressions.

Signed-off-by: Joel Dice <[email protected]>

* add reuse and timeout tests to `cli_tests`

This adds several new integration tests which exercise various aspects of
`wasmtime serve`'s instance reuse feature, including timeouts for p2-style
(synchronous) sleeps and p3-style (asynchronous) sleeps.

Signed-off-by: Joel Dice <[email protected]>

* fix foreach_api breakage

Signed-off-by: Joel Dice <[email protected]>

* Minor cleanups to `handler.rs`

* Don't use `FutureExt`, use `async { .. }` instead.
* Don't use `T`, instead use `S::StoreData`.

* Shift an `unreachable!()` to be closer to the "source"

* add more comments to `handler.rs`

These comments draw attention to a few of the more subtle aspects of the code
and the invariants it is upholding.

Signed-off-by: Joel Dice <[email protected]>

* add docs regarding the `req_id` parameter

Signed-off-by: Joel Dice <[email protected]>

* add comments regarding atomic load/store orderings

This also changes one of the `SeqCst` orderings to `Relaxed`.

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

show more ...

Update nightly Rust used in CI (#11755)

* Update nightly Rust used in CI

Keeping it up-to-date

prtest:full

* Fix unused warnings on nightly

* Rename rustdoc feature

* Adjust some removals

Document all wasmtime-wasi-http features on docs.rs (#11754)

Similar to other crates, copy the configuration for this crate.

p3-http: rework `content-length` handling (#11658)

* p3-http: correctly handle `result` future cancellation

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-http: restructure the `con

p3-http: rework `content-length` handling (#11658)

* p3-http: correctly handle `result` future cancellation

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-http: restructure the `content-length` test a bit

Signed-off-by: Roman Volosatovs <[email protected]>

* test(http): keep accepting connections after errors

prtest:full

Signed-off-by: Roman Volosatovs <[email protected]>

* test(p3-http): assert `handle` error on exceeding `content-length`

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-http: perform `content-length` check early

Signed-off-by: Roman Volosatovs <[email protected]>

* test(p3-http): account for `handle` race condition

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor(http): reuse `get_content_length`

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-http: check `content-length` for host bodies

Signed-off-by: Roman Volosatovs <[email protected]>

* doc(p3-http): call out that host bodies are not validated

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-http: refactor body size error send

Signed-off-by: Roman Volosatovs <[email protected]>

* fix(p3-http): do not rely on `Drop` for host body check

Signed-off-by: Roman Volosatovs <[email protected]>

* doc(p3-http): ensure non-default send request is documented

Signed-off-by: Roman Volosatovs <[email protected]>

* doc(p3-http): correct `send_request` doc

Signed-off-by: Roman Volosatovs <[email protected]>

---------

Signed-off-by: Roman Volosatovs <[email protected]>

show more ...

p3-http: finish `wasi:[email protected]` implementation (#11636)

* refactor(p3-http): use trappable errors

Signed-off-by: Roman Volosatovs <[email protected]>

* feat(p3-http): implement `content-lengt

p3-http: finish `wasi:[email protected]` implementation (#11636)

* refactor(p3-http): use trappable errors

Signed-off-by: Roman Volosatovs <[email protected]>

* feat(p3-http): implement `content-length` handling

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor(p3-http): remove a few resource utilities

Signed-off-by: Roman Volosatovs <[email protected]>

* remove unused test import

Signed-off-by: Roman Volosatovs <[email protected]>

* fix(p3-http): close stream handles on drop

Signed-off-by: Roman Volosatovs <[email protected]>

* test(p3-http): stream responses back

This is something we've been doing in wasip3, but I forgot to port this
over

Signed-off-by: Roman Volosatovs <[email protected]>

* doc(p3-http): add missing docs, internalize more, simplify

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor(p3-http): extract `Body::consume`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor(p3-http): clean-up `content-length` error reporting

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor(p3-http): drop elided lifetime

Signed-off-by: Roman Volosatovs <[email protected]>

* fix(p3-http): avoid guest body deadlock hazard

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor(p3-http): add more docs, clean-up

Signed-off-by: Roman Volosatovs <[email protected]>

* doc(p3-http): add more docs

Signed-off-by: Roman Volosatovs <[email protected]>

* fix(p3-http): rework result future handling

Most importantly this avoids a race condition between `content-length` error observed by `GuestBody`
and hyper I/O driver

Signed-off-by: Roman Volosatovs <[email protected]>

* add new imports after rebase

Signed-off-by: Roman Volosatovs <[email protected]>

* clean-up `poll_consume`

Signed-off-by: Roman Volosatovs <[email protected]>

* assert content-length `handle` results

Signed-off-by: Roman Volosatovs <[email protected]>

* relax `content_length` test `handle` assert

Signed-off-by: Roman Volosatovs <[email protected]>

---------

Signed-off-by: Roman Volosatovs <[email protected]>

show more ...

feat(p3): begin `wasi:http` implementation (#11439)

* doc: fix typo in p3 filesystem ref

Signed-off-by: Roman Volosatovs <[email protected]>

* build: vendor `wasi:[email protected]` WIT

Signed-off-

feat(p3): begin `wasi:http` implementation (#11439)

* doc: fix typo in p3 filesystem ref

Signed-off-by: Roman Volosatovs <[email protected]>

* build: vendor `wasi:[email protected]` WIT

Signed-off-by: Roman Volosatovs <[email protected]>

* feat(p3-http): scaffold p3 implementation

Signed-off-by: Roman Volosatovs <[email protected]>

* test(p3-http): add tests

Signed-off-by: Roman Volosatovs <[email protected]>

* chore(p3-http): add utils used by p3 tests

Signed-off-by: Roman Volosatovs <[email protected]>

* chore: add `expect` reasons

Signed-off-by: Roman Volosatovs <[email protected]>

* address review comments

Signed-off-by: Roman Volosatovs <[email protected]>

* establish `conn` connections on drop

Signed-off-by: Roman Volosatovs <[email protected]>

---------

Signed-off-by: Roman Volosatovs <[email protected]>

show more ...

Unify WASIp{2,3} context structures (#11370)

This removes `wasmtime_wasi::p{2,3}::{WasiCtx, WasiCtxBuilder,
WasiView}` in favor of only having `wasmtime_wasi::{WasiCtx,
WasiCtxBuilder, WasiView}` in

Unify WASIp{2,3} context structures (#11370)

This removes `wasmtime_wasi::p{2,3}::{WasiCtx, WasiCtxBuilder,
WasiView}` in favor of only having `wasmtime_wasi::{WasiCtx,
WasiCtxBuilder, WasiView}` instead. Conceptually these revisions of WASI
all provide the same functionality just with a different veneer that the
component model offers, so having only one way to configure host-side
behavior will make it easier to both organize implementations internally
(e.g. more sharing of code) as well as for embedders to configure (only
one context to create/manage).

show more ...

Simplify WASI internal implementations (#11365)

* Simplify WASI internal implementations

This commit migrates the WASIp2 implementation to be closer to the
upcoming WASIp3 implementation in terms o

Simplify WASI internal implementations (#11365)

* Simplify WASI internal implementations

This commit migrates the WASIp2 implementation to be closer to the
upcoming WASIp3 implementation in terms of how things are implemented
internally. Previously the way things worked with WASIp2 is:

* Embedders call `add_to_linker` with `T: WasiView`
* Internally `add_to_linker` is called which creates `WasiImpl<&mut T>`
* All internal implementations were `impl<T> Host for WasiImpl<T> where T: WasiView`
* A forwarding impl of `impl<T: WasiView> WasiView for &mut T` was
required

While this all worked it's a bit complicated for a few reasons:

1. Dealing with generically named structures like `WasiImpl` (or
`IoImpl` or `WasiHttpImpl`) is a bit baroque and not always obvious
as to what's going on.
2. The extra layer of generics in `impl<T> Host for WasiImpl<T>` adds a
layer of conceptual indirection which is non-obvious.
3. Other WASI proposal implementations do not use this strategy and
instead use "view" types or `impl Host for TheType` for example.
4. Internal incantations of `add_to_linker` had to deal with mixtures of
`IoImpl` and `WasiImpl` and aligning everything just right.
5. An extra layer of generics on all impls meant that everything was
generic meaning that `wasmtime-wasi`-the-crate didn't generate much
code, causing longer codegen times for consumers.

The goal of this commit is to migrate towards the style of what WASIp3
is prototyping for how impls are modeled. This is done to increase the
amount of code that can be shared between WASIp2 and WASIp3. This has a
number of benefits such as being easier to understand and also being
more modular where `wasi:clocks` implementations of traits don't require
filesystem context to be present (as is the case today). This in theory
helps a more mix-and-match paradigm of blending together various bits
and pieces of `wasmtime-wasi` implementations.

Concretely the changes made here are:

* `WasiView` no longer inherits from `IoView`, they're unrelated traits
now.
* `WasiView` now returns `WasiViewCtx<'a>` which has `ctx: &'a mut WasiCtx`
and `table: &'a mut ResourceTable`. That means it basically does the
same thing before but in a slightly different fashion.
* Implementations of `Host` traits are now directly for
`WasiCtxView<'_>` and don't involve any generics at all. These are
hopefully easier to understand and also better from a
codegen/compile-time perspective.
* Embedders no longer need to implement `IoView` directly and instead
fold that functionality into `WasiView`.
* `WasiHttpView` no longer inherits from `IoView` and instead has a
direct `fn table` method. Additionally `WasiHttpImpl` no longer embeds
`IoImpl` inside of it.
* Host traits for `wasi:io` are now implemented directly for
`ResourceTable` instead of `IoImpl<T>`.

The immediate goal of this refactoring is to enable more sharing along
the lines of #11362. This was not possible prior because WASIp3 requires
a simultaneous borrow on the table/ctx while the trait hierarchy
previously gave you one-or-the-other. With this new organization it will
be possible to get both at the same time meaning more
structure/contexts/etc can be shared between implementations.

prtest:full

* CI fixes

* More CI fixes

* More CI fixes

show more ...

Cargo doc fixes (#11171)

* wasmtime-wasi-http: fix docs missing wasmtime_wasi::p2::add_to_linker

reported in https://bytecodealliance.zulipchat.com/#narrow/channel/219900-wasi/topic/Help.3A.20Wasmt

Cargo doc fixes (#11171)

* wasmtime-wasi-http: fix docs missing wasmtime_wasi::p2::add_to_linker

reported in https://bytecodealliance.zulipchat.com/#narrow/channel/219900-wasi/topic/Help.3A.20Wasmtime's.20runner.20for.20WASI-HTTP.20p2.20guest/with/526609626

Unfortunately, this doc is no_run because it acts as a server daemon,
and loads a component which doesnt exist. If the doc had been executed
this would have been caught.

* wasmtime-wasi-http: other doc warning fixes

* wasmtime-wasi: cargo doc fixes

* Update crates/wasi-http/src/lib.rs

Co-authored-by: Lann <[email protected]>

---------

Co-authored-by: Lann <[email protected]>

show more ...

Update to WASI 0.2.6. (#11049)

* Update to WASI 0.2.6.

This includes two changes:

# https://github.com/WebAssembly/wasi-filesystem/pull/165
Documentation update

# https://github.com/WebAssembly/w

Update to WASI 0.2.6. (#11049)

* Update to WASI 0.2.6.

This includes two changes:

# https://github.com/WebAssembly/wasi-filesystem/pull/165
Documentation update

# https://github.com/WebAssembly/wasi-http/pull/139
Adds a new unstable `send-informational` function to the `response-outparam` resource. I've left this `unimplemented!()`.

* Trap instead of panic

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Replace `GetHost` with a function pointer, add `HasData` (#10770)

* Replace `GetHost` with a function pointer, add `HasData`

This commit is a refactoring to the fundamentals of the `bindgen!` macro

Replace `GetHost` with a function pointer, add `HasData` (#10770)

* Replace `GetHost` with a function pointer, add `HasData`

This commit is a refactoring to the fundamentals of the `bindgen!` macro
and the functions that it generates. Prior to this change the
fundamental entrypoint generated by `bindgen!` was a function
`add_to_linker_get_host` which takes a value of type `G: GetHost`. This
`GetHost` implementation is effectively an alias for a closure whose
return value is able to close over the parameter given lfietime-wise.

The `GetHost` abstraction was added to Wasmtime originally to enable
using any type that implements `Host` traits, not just `&mut U` as was
originally supported. The definition of `GetHost` was _just_ right to
enable a type such as `MyThing<&mut T>` to implement `Host` and a
closure could be provided that could return it. At the time that
`GetHost` was added it was known to be problematic from an
understandability point of view, namely:

* It has a non-obvious definition.
* It's pretty advanced Rust voodoo to understand what it's actually
doing
* Using `GetHost` required lots of `for<'a> ...` in places which is
unfamiliar syntax for many.
* `GetHost` values couldn't be type-erased (e.g. put in a trait object)
as we couldn't figure out the lifetime syntax to do so.

Despite these issues it was the only known solution at hand so we landed
it and kept the previous `add_to_linker` style (`&mut T -> &mut U`) as a
convenience. While this has worked reasonable well (most folks just try
to not look at `GetHost`) it has reached a breaking point in the WASIp3
work.

In the WASIp3 work it's effectively now going to be required that the
`G: GetHost` value is packaged up and actually stored inside of
accessors provided to host functions. This means that `GetHost` values
now need to not only be taken in `add_to_linker` but additionally
provided to the rest of the system through an "accessor". This was made
possible in #10746 by moving the `GetHost` type into Wasmtime itself (as
opposed to generated code where it lived prior).

While this worked with WASIp3 and it was possible to plumb `G: GetHost`
safely around, this ended up surfacing more issues. Namely all
"concurrent" host functions started getting significantly more
complicated `where` clauses and type signatures. At the end of the day I
felt that we had reached the end of the road to `GetHost` and wanted to
search for alternatives, hence this change.

The fundamental purpose of `GetHost` was to be able to express, in a
generic fashion:

* Give me a closure that takes `&mut T` and returns `D`.
* The `D` type can close over the lifetime in `&mut T`.
* The `D` type must implement `bindgen!`-generated traits.

A realization I had was that we could model this with a generic
associated type in Rust. Rust support for generic associated types is
relatively new and not something I've used much before, but it ended up
being a perfect model for this. The definition of the new `HasData`
trait is deceptively simple:

trait HasData {
type Data<'a>;
}

What this enables us to do though is to generate `add_to_linker`
functions that look like this:

fn add_to_linker<T, D>(
linker: &mut Linker<T>,
getter: fn(&mut T) -> D::Data<'_>,
) -> Result<()>
where
D: HasData,
for<'a> D::Data<'a>: Host;

This definition here models `G: GetHost` as a literal function pointer,
and the ability to close over the `&mut T` lifetime with type (not just
`&mut U`) is expressed through the type constructor `type Data<'a>`).
Ideally we could take a generic generic associated type (I'm not even
sure what to call that), but that's not something Rust has today.

Overall this felt like a much simpler way of modeling `GetHost` and its
requirements. This plumbed well throughout the WASIp3 work and the
signatures for concurrent functions felt much more appropriate in terms
of complexity after this change. Taking this change to the limit means
that `GetHost` in its entirety could be purged since all usages of it
could be replaced with `fn(&mut T) -> D::Data<'a>`, a hopefully much
more understandable type.

This change is not all rainbows however, there are some gotchas that
remain:

* One is that all `add_to_linker` generated functions have a `D:
HasData` type parameter. This type parameter cannot be inferred and
must always be explicitly specified, and it's not easy to know what to
supply here without reading documentation. Actually supplying the type
parameter is quite easy once you know what to do (and what to fill
in), but it may involve defining a small struct with a custom
`HasData` implementation which can be non-obvious.

* Another is that the `G: GetHost` value was previously a full Rust
closure, but now it's transitioning to a function pointer. This is
done in preparation for WASIp3 work where the function needs to be
passed around, and doing that behind a generic parameter is more
effort than it's worth. This means that embedders relying on the true
closure-like nature here will have to update to using a function
pointer instead.

* The function pointer is stored in locations that require `'static`,
and while `fn(T)` might be expected to be `'static` regardless of `T`
is is, in fact, not. This means that practically `add_to_linker`
requires `T: 'static`. Relative to just before this change this is a
possible regression in functionality, but there orthogonal reasons
beyond just this that we want to start requiring `T: 'static` anyway.
That means that this isn't actually a regression relative to #10760, a
related change.

The first point is partially ameliorated with WASIp3 work insofar that
the `D` type parameter will start serving as a location to specify where
concurrent implementations are found. These concurrent methods don't
take `&mut self` but instead are implemented for `T: HasData` types. In
that sense it's more justified to have this weird type parameter, but in
the meantime without this support it'll feel a bit odd to have this
little type parameter hanging off the side.

This change has been integrated into the WASIp3 prototyping repository
with success. This has additionally been integrated into the Spin
embedding which has one of the more complicated reliances on
`*_get_host` functions known. Given that it's expected that while this
is not necessarily a trivial change to rebase over it should at least be
possible.

Finally the `HasData` trait here has been included with what I'm hoping
is a sufficient amount of documentation to at least give folks a spring
board to understand it. If folks have confusion about this `D` type
parameter my hope is they'll make their way to `HasData` which showcases
various patterns for "librarifying" host implementations of WIT
interfaces. These patterns are all used throughout Wasmtime and WASI
currently in crates and tests and such.

* Update expanded test expectations

show more ...

Remove some `#![expect(clippy::allow_attributes_without_reason)]` (#10661)

Clean up some crates by migrating from `#[allow]` to `#[expect]`
(ideally) or `#[allow]`-with-reason

refactor: move wasip2 implementation to `wasmtime_wasi::p2` (#10073)

* refactor: move `wasmtime-wasi` p2-specific functionality to `p2`

Signed-off-by: Roman Volosatovs <[email protected]>

* d

refactor: move wasip2 implementation to `wasmtime_wasi::p2` (#10073)

* refactor: move `wasmtime-wasi` p2-specific functionality to `p2`

Signed-off-by: Roman Volosatovs <[email protected]>

* doc: move some of the p2 comments to top-level

Signed-off-by: Roman Volosatovs <[email protected]>

* chore: shorten `network` and `filesystem` module names

Signed-off-by: Roman Volosatovs <[email protected]>

---------

Signed-off-by: Roman Volosatovs <[email protected]>

show more ...

Don’t panic when http `handle` function returns before invoking `set` (#10387)

* handle return before `set` call

* move wasm32 gate

* unify error messages

Split off wasmtime-wasi-io crate from wasmtime-wasi (#10036)

* stub: wasmtime-wasi-io crate

* wasmtime: component::ResourceTableError now impls core::error::Error

for compatibility without std

*

Split off wasmtime-wasi-io crate from wasmtime-wasi (#10036)

* stub: wasmtime-wasi-io crate

* wasmtime: component::ResourceTableError now impls core::error::Error

for compatibility without std

* relocate much of the wasi-io impl into wasmtime-wasi-io

* stump of poll that uses in_tokio

* finish moving instances over to wasmtime_wasi_io

* redirect wasmtime_wasi's bindgen properly over to wasmtime_wasi_io

* wasmtime-wasi-http: point directly at wasmtime_wasi_io in sources

it worked without these changes because all the re-exports are in the
right places, but this is nice to do

* comment work

* fix streams rename, migrate bindings to its own file

* move wasi-io impls into their own mod with appropriate name. check in CI.

* change ResourceTable::iter_entries from taking a HashMap to BTreeMap so it works without std

* crate-level docs for wasmtime-wasi-io

* more docs

* more docs, wasi-io gives an add_to_linker function for async only

* wasi-io: inline view into lib.rs. improve docs.

* more streams vs stream fixes...

* wasi-http stream->streams fixes

* fix adding wasmtime-wasi-io to public crates

* wasmtime-cli: drop overzealous `=` version constraint on wasmtime-wasi-http

wasmtime-wasi-http is part of the public API where we guarantee semver
is obeyed

* fix doctest

* mechanically rename the wasi-io pub traits, and resource types

resource type Pollable -> DynPollable
resource type InputStream -> DynInputStream
resource type OutputStream -> DynOutputStream

trait Subscribe -> Pollable
trait HostInputStream -> InputStream
trait HostOutputStream -> OutputStream

type alias PollableFuture -> DynFuture (little-used)

* delete unused ClosureFuture alias

* doc fixes

* wasmtime-wasi-http: use all of wasmtime-wasi-io through wasmtime-wasi re-exports

* fix nostd build

* missing separator. i love yml

* make wasmtime-wasi-io #![no_std]

show more ...

wasmtime-wasi: Split a new `IoView` trait off of `WasiView` (#10016)

* split IoView trait off of WasiView

* move wasi-http over to split views

* config and keyvalue: no changes to libraries, just

wasmtime-wasi: Split a new `IoView` trait off of `WasiView` (#10016)

* split IoView trait off of WasiView

* move wasi-http over to split views

* config and keyvalue: no changes to libraries, just tests where WasiView used

* wasmtime-cli: fixes for IoView/WasiView split

* move rest of wasi-io impl to be on IoImpl

* wasi-http: linker with wasi IoImpl

* wasi-nn tests: IoView impl

show more ...

cli: add http outgoing body options (#9800)

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint levels. This new lint annotation will
silence a lint but will itself cause a lint if it doesn't actually
silence anything. This is quite useful to ensure that annotations don't
get stale over time.

Another feature is the ability to use a `reason` directive on the
attribute with a string explaining why the attribute is there. This
string is then rendered in compiler messages if a warning or error
happens.

This commit migrates applies a few changes across the workspace:

* Some `#[allow]` are changed to `#[expect]` with a `reason`.
* Some `#[allow]` have a `reason` added if the lint conditionally fires
(mostly related to macros).
* Some `#[allow]` are removed since the lint doesn't actually fire.
* The workspace configures `clippy::allow_attributes_without_reason = 'warn'`
as a "ratchet" to prevent future regressions.
* Many crates are annotated to allow `allow_attributes_without_reason`
during this transitionary period.

The end-state is that all crates should use
`#[expect(..., reason = "...")]` for any lint that unconditionally fires
but is expected. The `#[allow(..., reason = "...")]` lint should be used
for conditionally firing lints, primarily in macro-related code.
The `allow_attributes_without_reason = 'warn'` level is intended to be
permanent but the transitionary
`#[expect(clippy::allow_attributes_without_reason)]` crate annotations
to go away over time.

* Fix adapter build

prtest:full

* Fix one-core build of icache coherence

* Use `allow` for missing_docs

Work around rust-lang/rust#130021 which was fixed in Rust 1.83 and isn't
fixed for our MSRV at this time.

* More MSRV compat

show more ...

Fix handling of authority/scheme in `wasmtime serve` (#8923)

This commit is aimed at fixing an accidental regression from #8861 where
the `wasmtime serve` subcommand stopped reporting some instances

Fix handling of authority/scheme in `wasmtime serve` (#8923)

This commit is aimed at fixing an accidental regression from #8861 where
the `wasmtime serve` subcommand stopped reporting some instances of
authority and scheme. After discussion in #8878 the new logic
implemented is:

* Creation of an incoming request now explicitly requires specifying a
scheme which is out-of-band information about how the surrounding
server received the request.

* The authority is stored separately within a request and is inferred
from the URI's authority or the `Host` header if present. If neither
are present then an error is returned.

Closes #8878

show more ...