Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Migrate wasi-threads to `wasmtime::error` (#12275)

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as wel

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as well.

* Update dependency directive in `Cargo.toml`

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Make the Wasmtime CLI use async (#9184)

* Make the Wasmtime CLI use async

This means that interrupting a running Wasm program will now work
correctly, even when the program is blocked on I/O or wai

Make the Wasmtime CLI use async (#9184)

* Make the Wasmtime CLI use async

This means that interrupting a running Wasm program will now work
correctly, even when the program is blocked on I/O or waiting on a timeout or
some such.

This also involved making `wasi-threads` async-compatible.

Co-Authored-By: Alex Crichton <[email protected]>

* rustfmt

* Make `run` command enable the `tokio` feature

* Add a test for CLI, timeouts, and sleeping forever

* Fix warning

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

Wasmtime: Finish support for the typed function references proposal (#7943)

* Wasmtime: Finish support for the typed function references proposal

While we supported the function references proposal

Wasmtime: Finish support for the typed function references proposal (#7943)

* Wasmtime: Finish support for the typed function references proposal

While we supported the function references proposal inside Wasm, we didn't
support it on the "outside" in the Wasmtime embedder APIs. So much of the work
here is exposing typed function references, and their type system updates, in
the embedder API. These changes include:

* `ValType::FuncRef` and `ValType::ExternRef` are gone, replaced with the
introduction of the `RefType` and `HeapType` types and a
`ValType::Ref(RefType)` variant.

* `ValType` and `FuncType` no longer implement `Eq` and `PartialEq`. Instead
there are `ValType::matches` and `FuncType::matches` methods which check
directional subtyping. I also added `ValType::eq` and `FuncType::eq` static
methods for the rare case where someone needs to check precise equality, but
that is almost never actually the case, 99.99% of the time you want to check
subtyping.

* There are also public `Val::matches_ty` predicates for checking if a value is
an instance of a type, as well as internal helpers like
`Val::ensure_matches_ty` that return a formatted error if the value does not
match the given type. These helpers are used throughout Wasmtime internals
now.

* There is now a dedicated `wasmtime::Ref` type that represents reference
values. Table operations have been updated to take and return `Ref`s rather
than `Val`s.

Furthermore, this commit also includes type registry changes to correctly manage
lifetimes of types that reference other types. This wasn't previously an issue
because the only thing that could reference types that reference other types was
a Wasm module that added all the types that could reference each other at the
same time and removed them all at the same time. But now that the previously
discussed work to expose these things in the embedder API is done, type lifetime
management in the registry becomes a little trickier because the embedder might
grab a reference to a type that references another type, and then unload the
Wasm module that originally defined that type, but then the user should still be
able use that type and the other types it transtively references. Before, we
were refcounting individual registry entries. Now, we still are refcounting
individual entries, but now we are also accounting for type-to-type references
and adding a new type to the registry will increment the refcounts of each of
the types that it references, and removing a type from the registry will
decrement the refcounts of each of the types it references, and then recursively
(logically, not literally) remove any types whose refcount has now reached zero.

Additionally, this PR adds support for subtyping to `Func::typed`- and
`Func::wrap`-style APIs. For result types, you can always use a supertype of the
WebAssembly function's actual declared return type in `Func::typed`. And for
param types, you can always use a subtype of the Wasm function's actual declared
param type. Doing these things essentially erases information but is always
correct. But additionally, for functions which take a reference to a concrete
type as a parameter, you can also use the concrete type's supertype. Consider a
WebAssembly function that takes a reference to a function with a concrete type:
`(ref null <func type index>)`. In this scenario, there is no static
`wasmtime::Foo` Rust type that corresponds to that particular Wasm-defined
concrete reference type because Wasm modules are loaded dynamically at
runtime. You *could* do `f.typed::<Option<NoFunc>, ()>()`, and while that is
correctly typed and valid, it is often overly restrictive. The only value you
could call the resulting typed function with is the null function reference, but
we'd like to call it with non-null function references that happen to be of the
correct type. Therefore, `f.typed<Option<Func>, ()>()` is also allowed in this
case, even though `Option<Func>` represents `(ref null func)` which is the
supertype, not subtype, of `(ref null <func type index>)`. This does imply some
minimal dynamic type checks in this case, but it is supported for better
ergonomics, to enable passing non-null references into the function.

We can investigate whether it is possible to use generic type parameters and
combinators to define Rust types that precisely match concrete reference types
in future, follow-up pull requests. But for now, we've made things usable, at
least.

Finally, this also takes the first baby step towards adding support for the Wasm
GC proposal. Right now the only thing that is supported is `nofunc` references,
and this was mainly to make testing function reference subtyping easier. But
that does mean that supporting `nofunc` references entailed also adding a
`wasmtime::NoFunc` type as well as the `Config::wasm_gc(enabled)` knob. So we
officially have an in-progress implementation of Wasm GC in Wasmtime after this
PR lands!

Fixes https://github.com/bytecodealliance/wasmtime/issues/6455

* Fix WAT in test to be valid

* Check that dependent features are enabled for function-references and GC

* Remove unnecessary engine parameters from a few methods

Ever since `FuncType`'s internal `RegisteredType` holds onto its own `Engine`,
we don't need these anymore.

Still useful to keep the `Engine` parameter around for the `ensure_matches`
methods because that can be used to check correct store/engine usage for
embedders.

* Add missing dependent feature enabling for some tests

* Remove copy-paste bit from test

* match self to show it is uninhabited

* Add a missing `is_v128` method

* Short circuit a few func type comparisons

* Turn comment into part of doc comment

* Add test for `Global::new` and subtyping

* Add tests for embedder API, tables, and subtyping

* Add an embedder API test for setting globals and subtyping

* Construct realloc's type from its index, rather than from scratch

* Help LLVM better optimize our dynamic type checks in `TypedFunc::call_raw`

* Fix call benchmark compilation

* Change `WasmParams::into_abi` to take the whole func type instead of iter of params

* Fix doc links

prtest:full

* Fix size assertion on s390x

show more ...

Make wasi-common self-contained, deprecate exports from wasmtime-wasi (#7881)

* WIP: try to make wasi-common self contained.

* rebase: cargo.lock

* remove all dependencies between wasi-common and

Make wasi-common self-contained, deprecate exports from wasmtime-wasi (#7881)

* WIP: try to make wasi-common self contained.

* rebase: cargo.lock

* remove all dependencies between wasi-common and wasmtime-wasi

* use wasi-common directly throughout tests, benches, examples, cli run

* wasi-threads: use wasi-common's maybe_exit_on_error in spawned thread

not a very modular design, but at this point wasi-common and
wasi-threads are forever wed

* fix wasmtime's docs

* re-introduce wasmtime-wasi's exports of wasi-common definitions behind deprecated

* factor out determining i32 process exit code

and remove libc dep because rustix provides the same constant

* commands/run: inline the logic about aborting on trap

since this is the sole place in the codebase its used

* Add high-level summary to wasi-common's top-level doc comment.

* c-api: fix use of wasi_cap_std_sync => wasi_common::sync, wasmtime_wasi => wasi_common

* fix tokio example

* think better of combining downcast and masking into one method

* fix references to wasmtime_wasi in docs

prtest:full

* benches: use wasi-common

* cfg-if around use of rustix::process because that doesnt exist on windows

* wasi-common: include tests, caught by verify-publish

* fix another bench

* exit requires wasmtime dep. caught by verify-publish.

show more ...

Allocate thread IDs for wasi_thread_spawn sequentially (#7884)

wasi-threads: check module shape at spawn time (#6492)

This change relaxes what kinds of modules can be run when wasi-threads
is enabled via `--wasi-modules experimental-wasi-threads`. Previously,
a

wasi-threads: check module shape at spawn time (#6492)

This change relaxes what kinds of modules can be run when wasi-threads
is enabled via `--wasi-modules experimental-wasi-threads`. Previously,
as reported in #6153, simple modules that made no use of thread spawning
or shared memories were preemptively rejected when the wasi-threads
context was created. This is too restrictive.

Instead, this change does the following:
- it moves the check for whether a module is valid according to the
wasi-threads specification to the point a new thread is spawned; this
resolves #6153
- as noted in #6153, this change also adds a better error message
indicating that wasi-threads expects a shared memory import
- the way this is implemented also improves the module instantiation: by
constructing an `InstancePre` once when the `WasiThreadsCtx` is built,
we might shave off a bit of time from the "spawn a thread" call;
this supercedes a similar effort in #5741

show more ...

wasi-threads: fix import name (#5748)

* wasi-threads: fix import name

As @TerrorJack pointed out in #5484, that PR implements an older
name--`thread_spawn`. This change uses the now-official nam

wasi-threads: fix import name (#5748)

* wasi-threads: fix import name

As @TerrorJack pointed out in #5484, that PR implements an older
name--`thread_spawn`. This change uses the now-official name from the
specification--`thread-spawn`.

* fix: update name in test

show more ...

wasi-threads: an initial implementation (#5484)

This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime:

* feat: remove mutability from the WasiCtx Table

wasi-threads: an initial implementation (#5484)

This commit includes a set of changes that add initial support for `wasi-threads` to Wasmtime:

* feat: remove mutability from the WasiCtx Table

This patch adds interior mutability to the WasiCtx Table and the Table elements.

Major pain points:
* `File` only needs `RwLock<cap_std::fs::File>` to implement
`File::set_fdflags()` on Windows, because of [1]
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be hold across an `.await`, The `async` from
`async fn num_ready_bytes(&self)` had to be removed
* Because `File` needs a `RwLock` and `RwLock*Guard` cannot
be dereferenced in `pollable`, the signature of
`fn pollable(&self) -> Option<rustix::fd::BorrowedFd>`
changed to `fn pollable(&self) -> Option<Arc<dyn AsFd + '_>>`

[1] https://github.com/bytecodealliance/system-interface/blob/da238e324e752033f315f09c082ad9ce35d42696/src/fs/fd_flags.rs#L210-L217

* wasi-threads: add an initial implementation

This change is a first step toward implementing `wasi-threads` in
Wasmtime. We may find that it has some missing pieces, but the core
functionality is there: when `wasi::thread_spawn` is called by a running
WebAssembly module, a function named `wasi_thread_start` is found in the
module's exports and called in a new instance. The shared memory of the
original instance is reused in the new instance.

This new WASI proposal is in its early stages and details are still
being hashed out in the [spec] and [wasi-libc] repositories. Due to its
experimental state, the `wasi-threads` functionality is hidden behind
both a compile-time and runtime flag: one must build with `--features
wasi-threads` but also run the Wasmtime CLI with `--wasm-features
threads` and `--wasi-modules experimental-wasi-threads`. One can
experiment with `wasi-threads` by running:

```console
$ cargo run --features wasi-threads -- \
--wasm-features threads --wasi-modules experimental-wasi-threads \
<a threads-enabled module>
```

Threads-enabled Wasm modules are not yet easy to build. Hopefully this
is resolved soon, but in the meantime see the use of
`THREAD_MODEL=posix` in the [wasi-libc] repository for some clues on
what is necessary. Wiggle complicates things by requiring the Wasm
memory to be exported with a certain name and `wasi-threads` also
expects that memory to be imported; this build-time obstacle can be
overcome with the `--import-memory --export-memory` flags only available
in the latest Clang tree. Due to all of this, the included tests are
written directly in WAT--run these with:

```console
$ cargo test --features wasi-threads -p wasmtime-cli -- cli_tests
```

[spec]: https://github.com/WebAssembly/wasi-threads
[wasi-libc]: https://github.com/WebAssembly/wasi-libc

This change does not protect the WASI implementations themselves from
concurrent access. This is already complete in previous commits or left
for future commits in certain cases (e.g., wasi-nn).

* wasi-threads: factor out process exit logic

As is being discussed [elsewhere], either calling `proc_exit` or
trapping in any thread should halt execution of all threads. The
Wasmtime CLI already has logic for adapting a WebAssembly error code to
a code expected in each OS. This change factors out this logic to a new
function, `maybe_exit_on_error`, for use within the `wasi-threads`
implementation.

This will work reasonably well for CLI users of Wasmtime +
`wasi-threads`, but embedders will want something better in the future:
when a `wasi-threads` threads fails, they may not want their application
to exit. Handling this is tricky, because it will require cancelling the
threads spawned by the `wasi-threads` implementation, something that is
not trivial to do in Rust. With this change, we defer that work until
later in order to provide a working implementation of `wasi-threads` for
experimentation.

[elsewhere]: https://github.com/WebAssembly/wasi-threads/pull/17

* review: work around `fd_fdstat_set_flags`

In order to make progress with wasi-threads, this change temporarily
works around limitations induced by `wasi-common`'s
`fd_fdstat_set_flags` to allow `&mut self` use in the implementation.
Eventual resolution is tracked in
https://github.com/bytecodealliance/wasmtime/issues/5643. This change
makes several related helper functions (e.g., `set_fdflags`) take `&mut
self` as well.

* test: use `wait`/`notify` to improve `threads.wat` test

Previously, the test simply executed in a loop for some hardcoded number
of iterations. This changes uses `wait` and `notify` and atomic
operations to keep track of when the spawned threads are done and join
on the main thread appropriately.

* various fixes and tweaks due to the PR review

---------

Signed-off-by: Harald Hoyer <[email protected]>
Co-authored-by: Harald Hoyer <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

show more ...