Refactor borrow state tracking for async tasks (#12550)

* Refactor borrow state tracking for async tasks

This commit is a somewhat deep refactoring of how the state of
`borrow<T>` is managed for b

Refactor borrow state tracking for async tasks (#12550)

* Refactor borrow state tracking for async tasks

This commit is a somewhat deep refactoring of how the state of
`borrow<T>` is managed for both the host and the guest with respect to
async tasks. This additionally refactors how some async task management
is done for host-called functions.

The fundamental problem being tackled here is #12510. In that issue it
was discovered that the way `CallContext`, the borrow tracking mechanism
in Wasmtime, is managed is incompatible with async tasks. Specifically
the previous assumption of the scope being mutated for a borrow is
somewhere on the call stack is no longer true. It's possible for an
async task to be suspended, for example, and then a sibling task drops a
borrow which should update the scope of the suspended task. There were a
number of other small issues I noticed here and there which this PR
additionally has tests for, all of which failed before this change and
pass afterwards.

The manner in which borrow state is manipulated is a pretty old part of
the component model implementation dating back to the original
implementation of resources. I decided to forgo any possible quick fix
and have attempted to more deeply refactor and integrate async tasks
into all of this infrastructure. A list of the changes made here are:

* The `CallContexts` structure, a stack of `CallContext`, was removed.
Tasks now directly store a `CallContext` which is the source of truth
for borrow tracking for that call, and it does not move from this
location. The store `CallContexts` is now deleted in favor of updating
the `Option<ConcurrentState>` in the store to be an `enum` of either
concurrent state or a stack. In this manner the old stack-based
structure is still used sometimes, but it's impossible to reach when
concurrency is enabled.

* Entry to the host from guests now reliably pushes a `HostTask` into
the store. Previously where a frame were always pushed into a
`CallContext` a `HostTask` is pushed into the store. This is still
expected to be a bit too expensive for cheap host calls, but it
doesn't meaningfully change the performance profile of before.

* The `resource_enter_call` and `resource_exit_call` libcalls have been
removed. These are now folded into the `enter_sync_call` and
`exit_sync_call` libcalls. Emission of these hooks has been updated
accordingly. The concept of entering a call more generally has been
removed. This is more formally known in the async world as a task
starting, so the task creation is now responsible for the demarcation
of entering a call. Additionally this means that the concept of
exiting a call has somewhat gone away. Instead this method was renamed
to `validate_scope_exit` which double-checks that a borrow-scope can
be exited but doesn't actually remove the task. Task removal is
deferred to preexisting mechanisms.

* Management of a `GuestTask`'s previous `Option<CallContext>` field,
for example taking/restoring and pushing/popping onto `CallContexts`
is now all gone. All related code is outright deleted as the
`GuestTask`'s now non-optional `CallContext` field is the source of truth.

* The `ConcurrentState` structure now stores a `CurrentThread` enum
instead of `Option<QualifiedThreadId>`. This represents how the
currently executing thread could be a host thread, not just a guest
thread, which is required for borrow-tracking.

* `HostTask` creation in `poll_and_block` and `first_poll`, the two main
entrypoints of async host tasks when called by the guest, is now
externalized from these functions. Instead these functions assume that
the currently running thread is already a `HostTask` of some kind.

* In `poll_and_block` the host's result is no longer stored in the guest
task but in the host task instead.

Overall this enables the `*.wast` test for #12510 to fix the original
issue. This then adds new tests to ensure that cleanup of various
constructs happens appropriately, such as cancelling a host task should
clean up its associated resources. Additionally synchronously calling an
async host task no longer leaks resources in a `Store` and should
properly clean up everything.

There is still more work to do in this area (e.g. #12544) but that's
going to be deferred to a future PR at this point.

Closes #12510

prtest:full

* Review comments/CI fixes

show more ...

Do not allocate when cloning `ModuleRuntimeInfo` (#12413)

We need to clone the info from the allocation request, into the allocated
instance, so we should keep it a cheap operation.

Share empty `ModuleRuntimeInfo`s across all stores in an engine (#12409)

* Share empty `ModuleRuntimeInfo`s across all stores in an engine

This avoids an `Arc`- and `Box`-allocation during `Store`

Share empty `ModuleRuntimeInfo`s across all stores in an engine (#12409)

* Share empty `ModuleRuntimeInfo`s across all stores in an engine

This avoids an `Arc`- and `Box`-allocation during `Store` creation.

* Fix no-runtime build

* Pin x86-64_macos build to nightly to avoid a rustc bug

show more ...

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtim

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtime_environ::error` as the `wasmtime::error`
module, updates the prelude to include these new error-handling types, redirects
our top-level `wasmtime::{Error, Result}` re-exports to re-export
`wasmtime::error::{Error, Result}`, and updates various use sites that were
directly using `anyhow` to use the new `wasmtime` versions.

This process also required updating the component macro and wit-bindgen macro to
use the new error types instead of `anyhow`.

Part of https://github.com/bytecodealliance/wasmtime/issues/12069

* Replace wasmtime::error::Thing with wasmtime::Thing where it makes sense

* cargo fmt

* Move `crate::error::Thing` to `crate::Thing` where it makes sense

show more ...

Use `core::convert::Infallible` instead of our own `Uninhabited` type (#12115)

* Use `core::convert::Infallible` instead of our own `Uninhabited` type

I didn't realize that the standard library alr

Use `core::convert::Infallible` instead of our own `Uninhabited` type (#12115)

* Use `core::convert::Infallible` instead of our own `Uninhabited` type

I didn't realize that the standard library already had an uninhabited type
available for us to reuse.

* Actually remove the uninhabited module and its re-exports

show more ...

Debug: create private code memories per store when debugging is enabled. (#12051)

* Debug: create private code memories per store when debugging is enabled.

This will allow patching code to implem

Debug: create private code memories per store when debugging is enabled. (#12051)

* Debug: create private code memories per store when debugging is enabled.

This will allow patching code to implement e.g. breakpoints. (That is,
for now the copies are redundant, but soon they will not be.)

This change follows the discussion [here] and offline to define a few
types that better encapsulate the distinction we want to enforce.
Basically, there is almost never a bare `CodeMemory`; they are always
wrapped in an `EngineCode` or `StoreCode`, the latter being a per-store
instance of the former. Accessors are moved to the relevant place so
that, for example, one cannot get a pointer to a Wasm function's body
without being in the context of a `Store` where the containing module
has been registered. The registry then returns a `ModuleWithCode` that
boxes up a `Module` reference and `StoreCode` together for cases where
we need both the metadata from the module and the raw code to derive
something.

The only case where we return raw code pointers to the `EngineCode`
directly have to do with Wasm-to-array trampolines: in some cases, e.g.
`InstancePre` pre-creating data structures with references to host
functions, it breaks our expected performance characteristics to make
the function pointers store-specific. This is fine as long as the
Wasm-to-array trampolines never bake in direct calls to Wasm functions;
the strong invariant is that Wasm functions never execute from
`EngineCode` directly. Some parts of the component runtime would also
have to be substantially refactored if we wanted to do away with this
exception.

The per-`Store` module registry is substantially refactored in this PR.
I got rid of the modules-without-code distinction (the case where a
module only has trampolines and no defined functions still works fine),
and organized the BTreeMaps to key on start address rather than end
address, which I find a little more intuitive (one then queries with the
dual to the range -- 0-up-to-PC and last entry found).

[here]: https://github.com/bytecodealliance/wasmtime/pull/12051#pullrequestreview-3493711812

* Review feedback: do not assume a reasonable code alignment; error when it cannot be known

* Review feedback: fail properly in profiler when we are cloning code

* Fix guest-profiler C API.

* Review feedback: make private-code representation impossible in non-debugging-support builds.

* Add TODO comment referencing issue for cloning only .text.

* clang-format

* Review feedback: add back Component::image_range.

* Review feedback: error on registering profiling metadata when debug is enabled.

* rustfmt

* Remove early bail on profiling-data registration when debugging is enabled: this always happens so we cannot error out.

show more ...

Debugging: add a debugger callback mechanism to handle debug events. (#11895)

* Debugging: add a debugger callback mechanism to handle debug events.

This PR adds a notion of "debug events", and a m

Debugging: add a debugger callback mechanism to handle debug events. (#11895)

* Debugging: add a debugger callback mechanism to handle debug events.

This PR adds a notion of "debug events", and a mechanism in Wasmtime to
associate a "debug handler" with a store such that the handler is
invoked as-if it were an async hostcall on each event. The async
handler owns the store while its future exists, so the whole "world"
(within the store) is frozen and the handler can examine any state it
likes with a `StoreContextMut`.

Note that this callback-based scheme is a compromise: eventually, we
would like to have a native async API that produces a stream of events,
as sketched in #11826 and in [this branch]. However, the async approach
implemented naively (that is, with manual fiber suspends and with state
passed on the store) suffers from unsoundness in the presence of dropped
futures. Alex, Nick and I discussed this extensively and agreed that the
`Accessor` mechanism is the right way to allow for a debugger to have
"timesliced"/"shared" access to a store (only when polled/when an event
is delivered), but we will defer that for now, because it requires
additional work (mainly, converting existing async yield points in the
runtime to "give up" the store with the `run_concurrent` mechanism).
I'll file a followup issue to track that. The idea is that we can
eventually build that when ready, but the API we provide to a debugger
component can remain unchanged; only this plumbing and the glue to the
debugger component will be reworked.

With this scheme based on callbacks, we expect that one should be able
to implement a debugger using async channels to communicate with the
callback. The idea is that there would be a protocol where the callback
sends a debug event to the debugger main loop elsewhere in the executor
(e.g., over a Tokio channel or other async channel mechanism), and when
the debugger wants to allow execution to continue, it sends a "continue"
message back. In the meantime, while the world is paused, the debugger
can send messages to the callback to query the `StoreContextMut` it has
and read out state. This indirection/proxying of Store access is
necessary for soundness: again, teleporting the Store out may look like
it almost works ("it is like a mutable reborrow on a hostcall") except
in the presence of dropped futures with sandwiched Wasm->host->Wasm
situations.

This PR implements debug events for a few cases that can be caught
directly in the runtime, e.g., exceptions and traps raised just before
re-entry to Wasm. Other kinds of traps, such as those normally
implemented by host signals, require additional work (as in #11826) to
implement "hostcall injection" on signal reception; and breakpoints will
be built on top of that. The point of this PR is only to get the initial
plumbing in place for events.

[this branch]: https://github.com/cfallin/wasmtime/tree/wasmtime-debug-async

* Add some more tests.

* Review feedback: comment updates, and make `debug` feature depend on `async`.

* Review feedback: debug-hook setter requires guest debugging to be enabled.

* Review feedback: ThrownException event; handle block_on errors; explicitly list UnwindState cases.

* Add comment about load-bearing Send requirement.

* Fix no-unwind build.

* Review feedback: pass in hostcall error messages while keeping the trait object-safe.

Co-authored-by: Alex Crichton <[email protected]>

* Ignore divide-trapping test on Pulley for now.

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

feat(no_std): Add custom sync primitives support via C API (#11836)

* Add custom-sync-primitives feature for no_std environments

This adds a new `custom-sync-primitives` feature that allows embedde

feat(no_std): Add custom sync primitives support via C API (#11836)

* Add custom-sync-primitives feature for no_std environments

This adds a new `custom-sync-primitives` feature that allows embedders
to provide their own synchronization primitives through a C API. This
is particularly useful for kernel and embedded environments that need
custom lock implementations.

The implementation follows the existing pattern of `custom-virtual-memory`
and `custom-native-signals`, providing host-provided implementations of
OnceLock and RwLock through callback functions defined in the C API.

* assume no_std in has_custom_sync flag

* updating panic on contention error messages in sync_nostd.rs

* inlining free in drop in custom/sync.rs

* simplifying custom/sync.rs to delegate lazy initialization to the embedder

* Consolidate no_std sync implementations and add RwLock-specific C API

* Minor cleanup

* adding custom_sync support to min-platform example

* adding multithreaded TLS using pthread to min-platform example

* addressing PR comments:
Remove *_new files from capi sync api.
Implement Drop-based lock releasing
Update documentation + minor changes

* fixes min-platform example for sync primitives
Use heap allocation to avoid deadlock in the implementation with no heap allocation.
Use compare-and-swap for lazy initialization of rw lock for thread safety.

* upgrade cbindgen in CI to 0.29

* undo accidental edits to cbindgen generated header file

show more ...

Add PoolingAllocatorMetrics (#11490)

This exposes some basic runtime metrics derived from the internal state
of a `PoolingInstanceAllocator`.

Two new atomics were added to PoolingInstanceAllocator:

Add PoolingAllocatorMetrics (#11490)

This exposes some basic runtime metrics derived from the internal state
of a `PoolingInstanceAllocator`.

Two new atomics were added to PoolingInstanceAllocator: `live_memories`
and `live_tables`. While these counts could be derived from existing
state it would require acquiring mutexes on some inner state.

show more ...

WebAssembly exception-handling support. (#11326)

* WebAssembly exception-handling support.

This PR introduces support for the [Wasm exception-handling proposal],
which introduces a conventional try

WebAssembly exception-handling support. (#11326)

* WebAssembly exception-handling support.

This PR introduces support for the [Wasm exception-handling proposal],
which introduces a conventional try/catch mechanism to WebAssembly. The
PR supports modules that use `try_table` to register handlers for a
lexical scope; and provides `throw` and `throw_ref` that allocate (in
the first case) and throw exception objects.

This PR builds on top of the work in #10510 for Cranelift-level
exception support, #10919 for an unwinder, and #11230 for exception
objects built on top of GC, in addition a bunch of smaller fix and
enabling PRs around those.

[Wasm exception-handling proposal]: https://github.com/WebAssembly/exception-handling/

prtest:full

* Permit UnwindToWasm to have unused fields in Pulley builds (for now).

* Resolve miri-caught reborrowing issue.

* Ignore exceptions tests in miri for now (Pulley not supported).

* Use wasmtime_test on exceptions tests.

* Get tests passing on pulley platforms

* Add a check to `supports_host` for the generated test and assert
failure also when that is false.
* Remove `pulley_unsupported` test as it falls out of `#[wasmtime_test]`
* Remove `exceptions_store` helper as it falls out of `#[wasmtime_test]`
* Remove miri annotations as they fall out of `#[wasmtime_test]`

* Remove dead import

* Skip some unsupported tests entirely in `#[wasmtime_test]`

If the selected compiler doesn't support the host at all then there's no
need to run it. Actually running it could misinterpret `CraneliftNative`
as "run with pulley" otherwise, so avoid such false negatives.

* Cranelift: dynamic contexts: account for outgoing-args area.

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Remove `StoreOpaque::async_yield_impl` (#11482)

* Remove `StoreOpaque::async_yield_impl`

Now that we have all this fancy support for natively running `async`
things this commit refactors fuel/epoch

Remove `StoreOpaque::async_yield_impl` (#11482)

* Remove `StoreOpaque::async_yield_impl`

Now that we have all this fancy support for natively running `async`
things this commit refactors fuel/epochs to use it. This simplifies the
`VMStore` trait, removes a usage of `block_on`, and helps keep the
`async` boundary close to the libcall entrypoint rather than further
down the stack. This all in turn enables using rustc to check our
stack-locals for non-`Send` values instead of pinky promising that we're
doing the right thing everywhere.

* Clean up some code movement and comments

* Fix a merge conflict

show more ...

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `Store

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `StoreOpaque::allocate_instance` is now an `async fn`.
This function does not actually use the `async`-ness just yet so it's a
noop from that point of view, but this propagates outwards to enough
locations that I wanted to split this off to make future changes more
digestable.

Notably some creation functions here such as making an `Instance`,
`Table`, or `Memory` are refactored internally to use this new `async`
function. Annotations of `assert_ready` or `one_poll` are used as
appropriate as well.

For reference this commit was benchmarked with our `instantiation.rs`
benchmark in the pooling allocator and shows no changes relative to the
original baseline from before-`async`-PRs.

* Make table/memory creation `async` functions

This commit is a large-ish refactor which is made possible by the many
previous refactorings to internals w.r.t. async-in-Wasmtime. The end
goal of this change is that table and memory allocation are both `async`
functions. Achieving this, however, required some refactoring to enable
it to work:

* To work with `Send` neither function can close over `dyn VMStore`.
This required changing their `Option<&mut dyn VMStore>` arugment to
`Option<&mut StoreResourceLimiter<'_>>`
* Somehow a `StoreResourceLimiter` needed to be acquired from an
`InstanceAllocationRequest`. Previously the store was stored here as
an unsafe raw pointer, but I've refactored this now so
`InstanceAllocationRequest` directly stores `&StoreOpaque` and
`Option<&mut StoreResourceLimiter>` meaning it's trivial to acquire
them. This additionally means no more `unsafe` access of the store
during instance allocation (yay!).
* Now-redundant fields of `InstanceAllocationRequest` were removed since
they can be safely inferred from `&StoreOpaque`. For example passing
around `&Tunables` is now all gone.
* Methods upwards from table/memory allocation to the
`InstanceAllocator` trait needed to be made `async`. This includes new
`#[async_trait]` methods for example.
* `StoreOpaque::ensure_gc_store` is now an `async` function. This
internally carries a new `unsafe` block carried over from before with
the raw point passed around in `InstanceAllocationRequest`. A future
PR will delete this `unsafe` block, it's just temporary.

I attempted a few times to split this PR up into separate commits but
everything is relatively intertwined here so this is the smallest
"atomic" unit I could manage to land these changes and refactorings.

* Shuffle `async-trait` dep

* Fix configured build

show more ...

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
wi

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
with a number of provided impls. The `InstanceAllocator` trait is
implemented for everything implementing `InstanceAllocatorImpl` to force
users to be unable to override the default methods. When adding `async`
support internally to Wasmtime these are going to need to be
`#[async_trait]`-annotated-traits which adds a cost to `async` functions
as a future needs to be heap-allocated.

The goal of this commit is to make this future `async`-ification a bit
more optimal. Notably the `InstanceAllocator` trait is removed and
replaced with inherent methods on `impl dyn InstanceAllocatorImpl`.
After that the previous `InstanceAllocatorImpl` trait was renamed to
`InstanceAllocator` meaning that there's just one `InstanceAllocator`
trait which has inherent methods which cannot be overridden. A
consequence of this is that the inherent methods are also forced to do
virtual dispatch unlike before where they would internally use
monomorphization to have static dispatch. Given the complexity of
instance allocation this is expected to be a negligible cost, however.

The main benefit is that `allocate_module`, `allocate_tables`, and
`allocate_memories` all get to be native `async` functions without the
cost of `#[async_trait]`. Only allocation of a single table/memory will
require an allocation of a future which in profiling helps reduce the
cost of instantiation slightly.

Note that `#[async_trait]` is not currently used, this commit is just
preparation for its eventual use.

show more ...

Add a helper always-`Sync` utility to Wasmtime (#11453)

* Add a helper always-`Sync` utility to Wasmtime

This commit adds a newtype wrapper to Wasmtime, `AlwaysMut`, which is
unconditionally `Sync`

Add a helper always-`Sync` utility to Wasmtime (#11453)

* Add a helper always-`Sync` utility to Wasmtime

This commit adds a newtype wrapper to Wasmtime, `AlwaysMut`, which is
unconditionally `Sync` if the stored type is `Send`. This is similar to
a `Mutex<T>` where it promotes a `Send` bound to a `Sync` bound, but
it's unlike `Mutex<T>` in that `AlwaysMut<T>` has no synchronization.
The reason that this is safe is that `AlwaysMut<T>` completely disallows
access to the underlying data through `&self` and requires `&mut self`.
This is similar to how `Mutex::get_mut` is safe, for example.

This type cleans up a preexisting `unsafe impl Sync` block in funcref
management around `SendSyncBump` (`bumpalo::Bump` is `Send`, not `Sync`,
but we only access it through `&mut self`). This then additionally
removes `unsafe impl Sync for StoreFiber` which, upon reflection, is not
sound because we don't ever constraint the store's `T` type to `Sync`,
only `Send`. This is effectively no change throughout Wasmtime, however,
as fibers are only accessed with `&mut`.

Overall this is mostly just internal refactoring to reduce the amount of
`unsafe` inside of Wasmtime and to add a new utility to use in the
future too.

* Fix configured build

* Review comments

* Fix configured build

show more ...

Require a store in `catch_unwind_and_record_trap` (#11441)

* Require a store in `catch_unwind_and_record_trap`

This commit does some preparatory refactoring for #11326 to ensure that
a store is ava

Require a store in `catch_unwind_and_record_trap` (#11441)

* Require a store in `catch_unwind_and_record_trap`

This commit does some preparatory refactoring for #11326 to ensure that
a store is available when trap information is being processed. Currently
this doesn't leverage the new parameter but it should be leverage-able
in #11326.

* Review comments

show more ...

Make table growth a true `async fn` (#11442)

* Make table growth a true `async fn`

Upon further refactoring and thinking about #11430 I've realized that we
might be able to sidestep `T: Send` on th

Make table growth a true `async fn` (#11442)

* Make table growth a true `async fn`

Upon further refactoring and thinking about #11430 I've realized that we
might be able to sidestep `T: Send` on the store entirely which would be
quite the boon if it can be pulled off. The realization I had is that
the main reason for this was `&mut dyn VMStore` on the stack, but that
itself is actually a bug in Wasmtime (#11178) and shouldn't be done.
The functions which have this on the stack should actually ONLY have the
resource limiter, if configured. This means that while the
`ResourceLimiter{,Async}` traits need a `Send` supertrait that's
relatively easy to add without much impact. My hunch is that plumbing
this through to the end will enable all the benefits of #11430 without
requiring adding `T: Send` to the store.

This commit starts out on this journey by making table growth a true
`async fn`. A new internal type is added to represent a store's limiter
which is plumbed to growth functions. This represents a hierarchy of
borrows that look like:

* `StoreInner<T>`
* `StoreResourceLimiter<'_>`
* `StoreOpaque`
* `Pin<&mut Instance>`
* `&mut vm::Table`

This notably, safely, allows operating on `vm::Table` with a
`StoreResourceLimiter` at the same time. This is exactly what's needed
and prevents needing to have `&mut dyn VMStore`, the previous argument,
on the stack.

This refactoring cleans up `unsafe` blocks in table growth right
now which manually uses raw pointers to work around the borrow checker.
No more now!

I'll note as well that this is just an incremental step. What I plan on
doing next is handling other locations like memory growth, memory
allocation, and table allocation. Each of those will require further
refactorings to ensure that things like GC are correctly accounted for
so they're going to be split into separate PRs. Functionally though this
PR should have no impact other than a fiber is no longer required for
`Table::grow_async`.

* Remove #[cfg] gate

show more ...

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation for example to use `Val` instead of `ValRaw`. Previously
the usage of `ValRaw` meant that wasm was disallowed from performing a
GC during const evaluation, but currently constant expressions can
indeed perform a GC. The goal of this commit is to lift this limitation.
This is expected to be a minor slowdown for modules that hit this path,
but most modules shouldn't hit this in a hot loop since LLVM doesn't
generate modules that use this branch of const eval.

The usage of `Val` brings a number of benefits and refactorings
associated with it:

* Const-evaluation is generally safer than before since everything is
higher-level.
* GC types in const-eval were almost already using `Val` meaning that
there's actually fewer conversions now.
* Instantiation code was refactored to use `wasmtime::*`-API types
instead of low-level VM types. This deduplicates a good deal and lifts
complexity out of the raw VM bits.

Another issue that this commit fixes is to change how table
initialization is modeled internally in
`vm::Instance::table_init_segment`. Previously this was done by removing
the tables from an instance to get a split borrow into the store and the
table. This is not valid though because if, during initialization, a GC
is performed then the table is not present to find roots through the
table. This function is refactored to scope borrows to within a loop
instead of over a loop via various refactorings and such and usage of
higher level APIs. This is again, like above, expected to pessimize
performance but this is also not known to be a hot path for modules at
this time.

* Remove the `TableElement` type

This commit is a refactoring of how tables work within Wasmtime to avoid
funneling table elements through a `TableElement` enum internally.
Instead methods are "exploded" to `grow_{gc_ref,func,cont}` which means,
for example, funcrefs don't need a GcStore. The main motivation for this
change was to avoid the idiom where `TableElement` represents a cloned,
but unrooted, GC reference.

Prior to this commit there were a number of subtle bugs in the table
code for Wasmtime where write barriers were forgotten on `table.init`,
`table.set` (via the embedder API), and `table.grow`. While `table.fill`
correctly handled the GC references it was awkward to get everything
else working consistently so I opted to remove `TableElement` entirely
to make it more clear that `&VMGcRef` is ubiquitously used meaning that
the write barriers, for example, are the same as other parts of the
Wasmtime codebase.

This has a few extra tests for "make sure this doesn't leak" to ensure
that GC works correctly with new barriers in place.

* Fix some lints and warnings

* Fix wmemcheck build

* Review comments

* Optimize const eval and global initialization

* Fix compile

* Fix lints

show more ...

Remove a dead trait method (#11407)

* Remove a dead trait method

Made obsolete from previous refactorings, so no need to keep this any
more.

* Fix unused import

* Fix missing import

Add support for the Linux PAGEMAP_SCAN ioctl (#11372)

* WIP: use the pagemap_scan ioctl to selectively reset an instance's dirty pages

* Less hacky, and supporting tables, too

* Bugfixes

* WIP: m

Add support for the Linux PAGEMAP_SCAN ioctl (#11372)

* WIP: use the pagemap_scan ioctl to selectively reset an instance's dirty pages

* Less hacky, and supporting tables, too

* Bugfixes

* WIP: memcpy instead of pread

* Refactor support for pagemap

* Don't hold a raw pointer to the original data, plumb through an `Arc`
to have a safe reference instead.
* Update pagemap bindings to latest version of abstraction written.
* Include pagemap-specific tests.
* Use a `PageMap` structure created once-per-pool instead of a
static-with-a-file.
* Refactor to use the "pagemap path" unconditionally which blends in the
keep_resident bits.

* Improve safety documentation

prtest:full

* Fix some lints

* Skip ioctl tests when it's not supported

* Fix a memory leak by moving impls around

* Fix no vm build

* Review comments

* Add more pagemap-specific documentation

* Add more docs, refactor implementation slightly

* Improve `category_*` docs

Basically forward to the Linux kernel source itself.

* Fix compile

* Make pagemap integration resilient across forks

* Fix non-pooling-allocator-build

* Fix portability issues of new test

* Actually use config on macos

---------

Co-authored-by: Till Schneidereit <[email protected]>

show more ...

Raise `unsafe_op_in_unsafe_fn` further in Wasmtime (#11322)

* Raise `unsafe_op_in_unsafe_fn` further in Wasmtime

Now it's at `wasmtime::runtime`, not just `wasmtime::runtime::vm`.

* Review comments

Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm` (#11312)

* Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm`

Slowly expanding this lint to more of the crate.

prtest:full

* Fix lin

Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm` (#11312)

* Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm`

Slowly expanding this lint to more of the crate.

prtest:full

* Fix lints in custom module

* Fix some lints with miri

* Fix non-VM build

* Fix arm windows

show more ...

Gut `vm::Export` to mostly be `crate::Extern` (#11229)

* Remove `Table::from_wasmtime_table`

This commit removes the unsafe function `Table::from_wasmtime_table`.
This goes a bit further and remove

Gut `vm::Export` to mostly be `crate::Extern` (#11229)

* Remove `Table::from_wasmtime_table`

This commit removes the unsafe function `Table::from_wasmtime_table`.
This goes a bit further and removes `ExportTable` entirely as well which
means that table lookup on a `vm::Instance` directly returns a
`wasmtime::Table` without any need to translate back-and-forth.

* Remove `Tag::from_wasmtime_tag`

Like the previous commit, but for tags.

* Remove `Global::from_wasmtime_global`

Like the previous commit, but for globals.

* Remove `Memory::from_wasmtime_memory`

Like the previous commit, but for memories.

* Remove `Func::from_wasmtime_function`

Like previous commits, but for functions.

* Fix lints

* Fill out missing safety comment

* Review comments

show more ...

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previou

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previously forcibly allowed it. The reason
this was allowed was that when the workspace was first migrated the
Wasmtime crate had too many instances that I was willing to fix. I've
now come back around and tried to fix everything.

In short: ideally delete `#[allow]`, otherwise use `#[expect]`,
otherwise use `#[allow]`.

prtest:full

* Adjust some directives

* Fix some warnings

* Fix stack switching size tests on unix

* Don't have a conditional `Drop` impl

* Force `testing_freelist` method to be used

Too lazy to write `#[cfg]`, but not too lazy to write a test.

show more ...

generalize async fiber abstraction (#11114)

* generalize async fiber abstraction

As part of the work implementing the new Component Model async ABI in the
`wasip3-prototyping` repo, I've generalize

generalize async fiber abstraction (#11114)

* generalize async fiber abstraction

As part of the work implementing the new Component Model async ABI in the
`wasip3-prototyping` repo, I've generalized the `FiberFuture` abstraction in
`wasmtime::runtime::store::async_` to support fibers which can either retain
exclusive access to the store across suspend points or release it. The latter
allows the store to be used by the `component-model-async` event loop and/or
other fibers to run before the original fiber resumes, which is the key to
allowing multiple fibers to run concurrently, passing control of the store back
and forth.

In the case of Pulley, the above generalization means we also need to give each
fiber its own `Interpreter` so that multiple concurrent fibers don't clobber
each other's state.

Concretely, this moves a lot of the code out of `async_.rs` and into a new
`fiber.rs` submodule which will be shared with the `component-model-async`
implementation.

This also pulls in a new `StoreToken<T>` utility which has been useful in
`wasip3-prototyping` to safely convert from a `&mut dyn VMStore` to a
`StoreContextMut<'a, T>` when we previously witnessed a conversion in the other
direction.

Note that I've added a `'static` bound to the `VMStore` trait, which simplifies
use of `&mut dyn VMStore`, avoiding thorny lifetime issues.

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

Signed-off-by: Joel Dice <[email protected]>

* fix miri-flagged stacked borrow violation

As part of my earlier effort to unify the fiber abstractions in the `wasmtime`
crate, I changed a `*mut StoreFiber` field to a `&mut StoreFiber`, not realizing
that it resulted in a mutable alias at runtime and thus undefined behavior.
Miri caught it, fortunately.

Signed-off-by: Joel Dice <[email protected]>

* remove unneeded `Send` bounds

Signed-off-by: Joel Dice <[email protected]>

* address more review feedback

Main changes:

- Make `resume_fiber[_raw]` take a `&mut StoreOpaque` parameter to make its unsafe internals easier to reason about, safety-wise.

- Panic if `StoreFiber::drop` is called on an in-progress fiber without having called `StoreFiber::dispose` to gracefully end it first.

- (Re)introduce `FiberFuture`, which closes over a `&mut StoreOpaque` and uses it to call `StoreFiber::dispose` on drop.

This will require a few more changes to make it usable by `concurrent.rs`, but
I'll save those changes for a later PR.

Signed-off-by: Joel Dice <[email protected]>

* address more review feedback

Signed-off-by: Joel Dice <[email protected]>

* update `impl Send For StoreFiber` comment

Signed-off-by: Joel Dice <[email protected]>

* Remove currently-extraneous `Result<()>` from fibers

May be needed for concurrent bits, but for now not necessary.

* Use safe pointers instead of raw pointers

It's predicted Miri won't like this, but for now in-repo it's ok with it.

* Fold more responsibility into `resume_fiber_raw`

Remove the need for the function entirely and replace it with
`resume_fiber`.

* Remove channels from async fibers

Can use stack-based closures/results to transmit the result instead of
needing a channel.

* Fold `on_fiber_raw` directly into `on_fiber`

The `on_fiber` function is small enough it should be possible to do so.

* Don't use `Option` in `FiberFuture`

Leave the fiber non-optional at-rest so it's always available for the
destructor.

* Fold `suspend` functions together

Small shims, not otherwise public at this time, so remove a layer of
indirection.

* Move stack limit management to `FiberResumeState`

Helps remove some raw pointers that are held for a long time within
`AsyncCx`

* add some doc comments to `fiber.rs`

Signed-off-by: Joel Dice <[email protected]>

* update `fiber.rs` and friends to match CM async requirements

This adds a `resolve_or_release` function, which `Instance::resume_fiber` will
use when current `concurrent.rs` stub is replaced by a real implementation.

Signed-off-by: Joel Dice <[email protected]>

* fix non-component-model-async build warnings

Signed-off-by: Joel Dice <[email protected]>

* make `resume_fiber` private in `fiber.rs`

Signed-off-by: Joel Dice <[email protected]>

* Shrink `PollContext` state

Move management of the async guard range elsewhere to the normal
save/restore area.

* Refactor `AsyncCx`, reduce `unsafe`

* Remove the `AsyncCx` type from Wasmtime as it's inherently `unsafe` to
use, instead bundle operations directly on a `Store*` reference.

* Don't retain pointers-to-pointers within the roughly-equivalent
`BlockingContext` created in this PR. Instead when a blocking context
is created "take" the metadata from the store to assert exclusive
ownership of the pointers.

* Refactor how `&mut Context<'_>` is passed around, namely thread it
through fiber parameters to model resumption as registering a new
context to poll with.

* Remove `PollContext` in favor of directly storing a pointer as it's
now mostly an empty structure.

* Minor refactorings to make things more future-refactorable and/or
clear in a few places.

* Refactor management of the "current suspend" and "current future
context" pointers. These are now null'd out on resumption and asserted
null on suspension.

* Remove the need for a generic `Reset` structure in the fiber bits as
it's a pretty dangerous structure to have in general.

The end result of this refactoring is that all usage of `block_on` is
now safe and additionally many of the internals of the implementation
are safer than they were before

* Adjust some lint attributes

* Make manipulation of `AsyncState` safe

No need for raw pointers with recent refactorings.

* Fix dead code warning

* More dead code warnings

* Cut down on raw pointers in fiber.rs

* Move executor save/restore to normal fiber state save/restore

* Bikeshed a method name

* update comment in make_fiber

Signed-off-by: Joel Dice <[email protected]>

* fix machports build

Signed-off-by: Joel Dice <[email protected]>

---------

Signed-off-by: Joel Dice <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

show more ...