Use traps when checking initial table/memory bounds (#12929)

Instead of using a custom error string this enables fuzzing to, for
example, see that a `Trap` was returned and consider the fuzz test ca

Use traps when checking initial table/memory bounds (#12929)

Instead of using a custom error string this enables fuzzing to, for
example, see that a `Trap` was returned and consider the fuzz test case
a normal failure. These code paths are only executed when `bulk_memory`
is disabled which is pretty rare, and also explains why it's come up in
fuzzing only just now after #12883.

show more ...

Fix table64 initialization when bulk memory is disabled (#12894)

* Fix table64 initialization when bulk memory is disabled

This commit fixes a panic in the host during instantiation when the
`bulk_

Fix table64 initialization when bulk memory is disabled (#12894)

* Fix table64 initialization when bulk memory is disabled

This commit fixes a panic in the host during instantiation when the
`bulk_memory` wasm feature is disabled. In this mode the initialization
of tables/memories is slightly different and a refactoring for 64-bit
support wasn't applied to this code path, meaning that it resulted in a
panic instead of properly handling 64-bit tables.

* Fix clippy

show more ...

Reduce type complexity of `InstanceAllocator` async functions (#12887)

This is a follow-on to #12849 to try to simplify some of the resulting
signatures a bit. Notably the `Result<..., OutOfMemory>`

Reduce type complexity of `InstanceAllocator` async functions (#12887)

This is a follow-on to #12849 to try to simplify some of the resulting
signatures a bit. Notably the `Result<..., OutOfMemory>` is now packaged
up directly into the output future, so the functions still retain a sort
of "async trait" feel even though they're still incompatible with
`#[async_trait]` (and can't be defined with that anyway).

show more ...

Use explicit boxing for InstanceAllocator async methods (#12849)

Change `allocate_memory` and `allocate_table` in the `InstanceAllocator` trait
from `async fn`s to regular `fn`s that return `Result<

Use explicit boxing for InstanceAllocator async methods (#12849)

Change `allocate_memory` and `allocate_table` in the `InstanceAllocator` trait
from `async fn`s to regular `fn`s that return `Result<Pin<Box<dyn Future<...>>>,
OutOfMemory>`.

This avoids the implicit `Box::new` allocation that `#[async_trait]` generates
when calling these methods through `dyn InstanceAllocator`, which would panic on
OOM instead of returning an error. Now the boxing is done explicitly via
`try_new::<Box<_>>` which returns `Err(OutOfMemory)` on allocation failure.

show more ...

Use `TryPrimaryMap` in `Instance` (#12848)

Introduce `wasmtime::Store::try_new`, which handles OOM (#12415)

* Introduce `wasmtime::Store::try_new`, which handles OOM

`Store::new` is an infallible constructor, so there is not a direct way to

Introduce `wasmtime::Store::try_new`, which handles OOM (#12415)

* Introduce `wasmtime::Store::try_new`, which handles OOM

`Store::new` is an infallible constructor, so there is not a direct way to make
it return an error on OOM. Additionally, it is one of the most-used functions in
the Wasmtime embedder API, so changing its signature to return a `Result` is a
non-starter -- it would cause way too much pain. So instead we define
`Store::try_new` which returns a `Result` and make `Store::new` call and unwrap
that new constructor.

Part of https://github.com/bytecodealliance/wasmtime/issues/12069

* update disas tests and fix winch

* Disable concurrency support in `Store::try_new` OOM test

* Add attributes that were lost in rebase

show more ...

Remove unnecessary `memory_tys` param from `vm::Instance::new` (#12414)

We already have this information in the `InstanceAllocationRequest`, and by
avoiding a parameter we can never accidentally pas

Remove unnecessary `memory_tys` param from `vm::Instance::new` (#12414)

We already have this information in the `InstanceAllocationRequest`, and by
avoiding a parameter we can never accidentally pass the wrong memory types.

show more ...

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Don't unwrap results of const-eval (#11557)

While all const expressions executed at runtime are valid they can still
fail due to GC OOM or failure to allocate more GC memory. As a result
all `unwrap

Don't unwrap results of const-eval (#11557)

While all const expressions executed at runtime are valid they can still
fail due to GC OOM or failure to allocate more GC memory. As a result
all `unwrap()` on `eval(...)` is removed in favor of propagating
upwards the result.

Closes #11469

show more ...

Add PoolingAllocatorMetrics (#11490)

This exposes some basic runtime metrics derived from the internal state
of a `PoolingInstanceAllocator`.

Two new atomics were added to PoolingInstanceAllocator:

Add PoolingAllocatorMetrics (#11490)

This exposes some basic runtime metrics derived from the internal state
of a `PoolingInstanceAllocator`.

Two new atomics were added to PoolingInstanceAllocator: `live_memories`
and `live_tables`. While these counts could be derived from existing
state it would require acquiring mutexes on some inner state.

show more ...

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "widened" into encompassing the limiter on
the `T` in `StoreInner<T>`, for example, by using the self-pointer
located in an instance or the store. This fix is done by threading
`&mut StoreOpaque` as a parameter separately from a
`StoreResourceLimiter`. This means that various callers now take a new
`Option<&mut StoreResourceLimiter<'_>>` parameter in various locations.

Closes #11409

* Fix gc-less build

show more ...

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `Store

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `StoreOpaque::allocate_instance` is now an `async fn`.
This function does not actually use the `async`-ness just yet so it's a
noop from that point of view, but this propagates outwards to enough
locations that I wanted to split this off to make future changes more
digestable.

Notably some creation functions here such as making an `Instance`,
`Table`, or `Memory` are refactored internally to use this new `async`
function. Annotations of `assert_ready` or `one_poll` are used as
appropriate as well.

For reference this commit was benchmarked with our `instantiation.rs`
benchmark in the pooling allocator and shows no changes relative to the
original baseline from before-`async`-PRs.

* Make table/memory creation `async` functions

This commit is a large-ish refactor which is made possible by the many
previous refactorings to internals w.r.t. async-in-Wasmtime. The end
goal of this change is that table and memory allocation are both `async`
functions. Achieving this, however, required some refactoring to enable
it to work:

* To work with `Send` neither function can close over `dyn VMStore`.
This required changing their `Option<&mut dyn VMStore>` arugment to
`Option<&mut StoreResourceLimiter<'_>>`
* Somehow a `StoreResourceLimiter` needed to be acquired from an
`InstanceAllocationRequest`. Previously the store was stored here as
an unsafe raw pointer, but I've refactored this now so
`InstanceAllocationRequest` directly stores `&StoreOpaque` and
`Option<&mut StoreResourceLimiter>` meaning it's trivial to acquire
them. This additionally means no more `unsafe` access of the store
during instance allocation (yay!).
* Now-redundant fields of `InstanceAllocationRequest` were removed since
they can be safely inferred from `&StoreOpaque`. For example passing
around `&Tunables` is now all gone.
* Methods upwards from table/memory allocation to the
`InstanceAllocator` trait needed to be made `async`. This includes new
`#[async_trait]` methods for example.
* `StoreOpaque::ensure_gc_store` is now an `async` function. This
internally carries a new `unsafe` block carried over from before with
the raw point passed around in `InstanceAllocationRequest`. A future
PR will delete this `unsafe` block, it's just temporary.

I attempted a few times to split this PR up into separate commits but
everything is relatively intertwined here so this is the smallest
"atomic" unit I could manage to land these changes and refactorings.

* Shuffle `async-trait` dep

* Fix configured build

show more ...

Make const-expr evaluation `async` (#11468)

* Make const-expr evaluation `async`

This commit is extracted from #11430 to accurately reflect how
const-expr evaluation is an async operation due to GC

Make const-expr evaluation `async` (#11468)

* Make const-expr evaluation `async`

This commit is extracted from #11430 to accurately reflect how
const-expr evaluation is an async operation due to GC pauses that may
happen. The changes in this commit are:

* Const-expr evaluation is, at its core, now an `async` function.
* To leverage this new `async`-ness all internal operations are switched
from `*_maybe_async` to `*_async` meaning all the `*_maybe_async`
methods can be removed.
* Some libcalls using `*_maybe_async` are switch to using `*_async` plus
the `block_on!` utility to help jettison more `*_maybe_async` methods.
* Instance initialization is now an `async` function. This is
temporarily handled with `block_on` during instance initialization to
avoid propagating the `async`-ness further upwards. This `block_on`
will get deleted in future refactorings.
* Const-expr evaluation has been refactored slightly to enable having a
fast path in global initialization which skips an `await` point
entirely, achieving performance-parity in benchmarks prior to this commit.

This ended up fixing a niche issue with GC where if a wasm execution was
suspended during `table.init`, for example, during a const-expr
evaluation triggering a GC then if the wasm execution was cancelled it
would panic the host. This panic was because the GC operation returned
`Result` but it was `unwrap`'d as part of the const-expr evaluation
which can fail not only to invalid-ness but also due to "computation is
cancelled" traps.

* Fix configured build

* Undo rebase mistake

show more ...

Use another RAII guard for instance allocation (#11466)

Forgotten from #11459 and extracted from #11430, uses an RAII guard
instead of a closure to handle errors.

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
wi

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
with a number of provided impls. The `InstanceAllocator` trait is
implemented for everything implementing `InstanceAllocatorImpl` to force
users to be unable to override the default methods. When adding `async`
support internally to Wasmtime these are going to need to be
`#[async_trait]`-annotated-traits which adds a cost to `async` functions
as a future needs to be heap-allocated.

The goal of this commit is to make this future `async`-ification a bit
more optimal. Notably the `InstanceAllocator` trait is removed and
replaced with inherent methods on `impl dyn InstanceAllocatorImpl`.
After that the previous `InstanceAllocatorImpl` trait was renamed to
`InstanceAllocator` meaning that there's just one `InstanceAllocator`
trait which has inherent methods which cannot be overridden. A
consequence of this is that the inherent methods are also forced to do
virtual dispatch unlike before where they would internally use
monomorphization to have static dispatch. Given the complexity of
instance allocation this is expected to be a negligible cost, however.

The main benefit is that `allocate_module`, `allocate_tables`, and
`allocate_memories` all get to be native `async` functions without the
cost of `#[async_trait]`. Only allocation of a single table/memory will
require an allocation of a future which in profiling helps reduce the
cost of instantiation slightly.

Note that `#[async_trait]` is not currently used, this commit is just
preparation for its eventual use.

show more ...

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation for example to use `Val` instead of `ValRaw`. Previously
the usage of `ValRaw` meant that wasm was disallowed from performing a
GC during const evaluation, but currently constant expressions can
indeed perform a GC. The goal of this commit is to lift this limitation.
This is expected to be a minor slowdown for modules that hit this path,
but most modules shouldn't hit this in a hot loop since LLVM doesn't
generate modules that use this branch of const eval.

The usage of `Val` brings a number of benefits and refactorings
associated with it:

* Const-evaluation is generally safer than before since everything is
higher-level.
* GC types in const-eval were almost already using `Val` meaning that
there's actually fewer conversions now.
* Instantiation code was refactored to use `wasmtime::*`-API types
instead of low-level VM types. This deduplicates a good deal and lifts
complexity out of the raw VM bits.

Another issue that this commit fixes is to change how table
initialization is modeled internally in
`vm::Instance::table_init_segment`. Previously this was done by removing
the tables from an instance to get a split borrow into the store and the
table. This is not valid though because if, during initialization, a GC
is performed then the table is not present to find roots through the
table. This function is refactored to scope borrows to within a loop
instead of over a loop via various refactorings and such and usage of
higher level APIs. This is again, like above, expected to pessimize
performance but this is also not known to be a hot path for modules at
this time.

* Remove the `TableElement` type

This commit is a refactoring of how tables work within Wasmtime to avoid
funneling table elements through a `TableElement` enum internally.
Instead methods are "exploded" to `grow_{gc_ref,func,cont}` which means,
for example, funcrefs don't need a GcStore. The main motivation for this
change was to avoid the idiom where `TableElement` represents a cloned,
but unrooted, GC reference.

Prior to this commit there were a number of subtle bugs in the table
code for Wasmtime where write barriers were forgotten on `table.init`,
`table.set` (via the embedder API), and `table.grow`. While `table.fill`
correctly handled the GC references it was awkward to get everything
else working consistently so I opted to remove `TableElement` entirely
to make it more clear that `&VMGcRef` is ubiquitously used meaning that
the write barriers, for example, are the same as other parts of the
Wasmtime codebase.

This has a few extra tests for "make sure this doesn't leak" to ensure
that GC works correctly with new barriers in place.

* Fix some lints and warnings

* Fix wmemcheck build

* Review comments

* Optimize const eval and global initialization

* Fix compile

* Fix lints

show more ...

Minimize lazy allocation of the GC store (#11411)

* Minimize lazy allocation of the GC store

This commit is an effort to minimize the number of entrypoints which
might lazily allocate a GC store. T

Minimize lazy allocation of the GC store (#11411)

* Minimize lazy allocation of the GC store

This commit is an effort to minimize the number of entrypoints which
might lazily allocate a GC store. The is currently done through
`StoreOpaque::gc_store_mut` but this method is very commonly used
meaning that there are many many places to audit for lazily allocating a
GC store. The reason that this needs an audit is that lazy allocation
is an async operation right now that must be on a fiber and is something
I'm looking to fix as part of #11262.

This commit performs a few refactorings to achieve this:

* `gc_store_mut` is renamed to `ensure_gc_store`. This is intended to be
an `async` function in the future and clearly demarcates where lazy
allocation of a GC store is occurring.

* `require_gc_store{,_mut}` is now added which is a pure accessor of the
GC store with no lazy allocation. Most locations previously using
`gc_store_mut` are updated to use this instead.

Documentation is added to store methods to clearly indicate which ones
are allocating and which ones should only be called in a context where
allocation should already have happened.

* Fix configured build

* Relax GC store restrictions in more places

* Review comments on documentation

* Move `ensure_gc_store` calls during instantiation

Instead update `needs_gc_heap` with the tables that are added to a
module and rely on instantiation to create the GC heap.

* Shuffle around some code

* Fix CI and review comments

* Add in a few more i31 cases for externref

show more ...

Make table/memory allocation functions safe (#11320)

These were previously marked as `unsafe` trait methods with a
requirement that the memory/table shape must be validated ahead of time.
Neither th

Make table/memory allocation functions safe (#11320)

These were previously marked as `unsafe` trait methods with a
requirement that the memory/table shape must be validated ahead of time.
Neither the ondemand nor pooling allocator actually has an unsafe
contract to uphold with respect to this and both may assert/reject
non-validated shapes but memory unsafety won't happen as a result.
Consequently these functions are made safe.

Instance allocation functions are adjusted to reflect how the
correctness of `imports` is required for the functions to be safe.

show more ...

Deny `unsafe_op_in_unsafe_fn` in `wasmtime::vm::instance` (#11267)

Turn the lint on and add some safety comments where appropriate.

cc #11180

More table safety improvements (#11255)

* More table safety improvements

This is some more progress on #11179 aimed at improving the safety of
management of tables internally within Wasmtime:

* `I

More table safety improvements (#11255)

* More table safety improvements

This is some more progress on #11179 aimed at improving the safety of
management of tables internally within Wasmtime:

* `Instance::table_index` is removed as it can be replaced with data
stored directly in the `VMTableImport` now.
* `Instance::get_table` now returns `&mut Table`
* `Instance::get_defined_table_with_lazy_init` now returns `&mut Table`
* `Instance::with_defined_table_index_and_instance` now directly returns
`DefinedTableIndex` plus `Pin<&mut Instance>`, codifying the ability
to "laterally move" between instances.
* `Instance::table_init_segment` was refactored to "take" the tables
during initialization and replace them afterwards, resolving the split
borrow issue and removing an `unsafe` block in the function.

cc #11179

* Improve safety of `Table::copy`

This commit fixes an issue in the previous commit with respect to Miri
and Stacked Borrows. This does so by improving the safety of the
`Table::copy`-related functions to all work mostly on safe code rather
than unsafe references. Some minor amount of unsafety is still present
but it is now clearly documented and easier to verify.

* Fix tests

show more ...

Make `vm::Instance::get_defined_table_with_lazy_init` return a reference (#11215)

This commit updates the `get_defined_table_with_lazy_init` method to
return a safe reference instead of a raw refere

Make `vm::Instance::get_defined_table_with_lazy_init` return a reference (#11215)

This commit updates the `get_defined_table_with_lazy_init` method to
return a safe reference instead of a raw reference. This then
cascaded outwards to updating more locations to handle this new safe
reference as well. While this makes some further progress on #11179
there are a number of locations that will require using raw pointers
until more refactoring is done.

show more ...

Implement exception objects. (#11230)

* WIP: Working exception objects

* Clean build with gc disabled (`cargo check -p wasmtime --no-default-features --features runtime`).

* Review feedback.

* St

Implement exception objects. (#11230)

* WIP: Working exception objects

* Clean build with gc disabled (`cargo check -p wasmtime --no-default-features --features runtime`).

* Review feedback.

* Stub out C-API support.

* Fix Clippy complaints.

* Fix dead-code warning in c-api build.

* Actually fix 27->26 reserved bit rename and test.

* Fix exnref doc-test.

* fix fuzzing build

* fix feature-flagging on Instance::id

* Bless disas test diff due to reserved-bits change.

* Review feedback.

show more ...

Inch towards making tables safer (#11212)

This is a small step towards #11179 which starts off by making
`vm::Instance::get_defined_table` a safe function. That required
updating one location which

Inch towards making tables safer (#11212)

This is a small step towards #11179 which starts off by making
`vm::Instance::get_defined_table` a safe function. That required
updating one location which required both a `&mut Table` and a
`&mut GcStore` at the same time with a new helper on the store. More of
these sorts of helpers are likely going to be necessary but IMO it's a
small price to pay for safety.

show more ...

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previou

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previously forcibly allowed it. The reason
this was allowed was that when the workspace was first migrated the
Wasmtime crate had too many instances that I was willing to fix. I've
now come back around and tried to fix everything.

In short: ideally delete `#[allow]`, otherwise use `#[expect]`,
otherwise use `#[allow]`.

prtest:full

* Adjust some directives

* Fix some warnings

* Fix stack switching size tests on unix

* Don't have a conditional `Drop` impl

* Force `testing_freelist` method to be used

Too lazy to write `#[cfg]`, but not too lazy to write a test.

show more ...

Crack down on mutability and ownership of `vm::Instance` (#10943)

* Crack down on mutability and ownership of `vm::Instance`

This commit represents more effort to bring safety to `vm::Instance`
and

Crack down on mutability and ownership of `vm::Instance` (#10943)

* Crack down on mutability and ownership of `vm::Instance`

This commit represents more effort to bring safety to `vm::Instance`
and, eventually, `ComponentInstance`. This is specifically addressing
two points of safety around `vm::Instance`:

* Previously ownership of this was murky where `InstanceHandle` sort of
represented ownership but sort of didn't either through the
`InstanceHandle::clone` method. Now `InstanceHandle` has a destructor
for instances and no longer has `clone`, so there's one exclusive
owner of an instance.

* Previously `&mut Instance` was liberally passed around, but this is
not sound because certain fields cannot be mutated (e.g. runtime
offset information). While not a perfect solution this PR switches to
using `Pin<&mut Instance>` everywhere instead. This prevents safe
access to `&mut Instance` and we hand-write accessors to individual
fields. Notably we omit mutable access to the `runtime_info` field.

This naturally involved a lot of refactoring internally, but notably
this started bringing up preexisting issues around how there are
locations in the codebase that simultaneously have `&mut Instance` and
`&mut StoreOpaque` which is technically not sound due to being able to
get back to the instance from the store. Some issues here were address
by passing around indices more often such as in instance initialization
and const-expr evaluation.

Note that all proxy methods on `InstanceHandle` are also all removed now
and there's now only two: `get` and `get_mut`. This reflects how
`InstanceHandle` should in general no longer be used and instead
`Instance` itself, and some pointer-to thereof, should be exclusively
used.

cc #10933

* Fix stack-switching-less build

* Fix wmemcheck build

show more ...