Reduce type complexity of `InstanceAllocator` async functions (#12887)

This is a follow-on to #12849 to try to simplify some of the resulting
signatures a bit. Notably the `Result<..., OutOfMemory>`

Reduce type complexity of `InstanceAllocator` async functions (#12887)

This is a follow-on to #12849 to try to simplify some of the resulting
signatures a bit. Notably the `Result<..., OutOfMemory>` is now packaged
up directly into the output future, so the functions still retain a sort
of "async trait" feel even though they're still incompatible with
`#[async_trait]` (and can't be defined with that anyway).

show more ...

Handle OOM in `{Func,Memory,Table,Global}::new` and when calling an instance's exported function (#12855)

* Use `try_new` for `Box<dyn RuntimeLinearMemory>` in `DefaultMemoryCreator`

* Use `TryPrim

Handle OOM in `{Func,Memory,Table,Global}::new` and when calling an instance's exported function (#12855)

* Use `try_new` for `Box<dyn RuntimeLinearMemory>` in `DefaultMemoryCreator`

* Use `TryPrimaryMap` for `host_globals` in `Store`

* Add `Func::try_wrap` and use `try_new` for `Box<HostFunc>`

Add `Func::try_wrap` as a fallible version of `Func::wrap` that returns an error
on out-of-memory instead of panicking. `Func::wrap` now delegates to `try_wrap`.

Also use `try_new::<Box<_>>` instead of `Box::new` for `HostFunc`.

* Use `bumpalo`'s `try_alloc` for `FuncRefs`

* Use `try_new` for `Arc<Module>` in "trampoline" code

* Test that we handle OOM in `{Func,Memory,Table,Global}::new` and when calling an instance's exported function

* cargo fmt

show more ...

Use explicit boxing for InstanceAllocator async methods (#12849)

Change `allocate_memory` and `allocate_table` in the `InstanceAllocator` trait
from `async fn`s to regular `fn`s that return `Result<

Use explicit boxing for InstanceAllocator async methods (#12849)

Change `allocate_memory` and `allocate_table` in the `InstanceAllocator` trait
from `async fn`s to regular `fn`s that return `Result<Pin<Box<dyn Future<...>>>,
OutOfMemory>`.

This avoids the implicit `Box::new` allocation that `#[async_trait]` generates
when calling these methods through `dyn InstanceAllocator`, which would panic on
OOM instead of returning an error. Now the boxing is done explicitly via
`try_new::<Box<_>>` which returns `Err(OutOfMemory)` on allocation failure.

show more ...

Use OOM-handling `PrimaryMap` in `wasmtime_environ::Module` (#12606)

Refactor `String` usage in `wasmtime_environ::Module` (#12565)

* Refactor `String` usage in `wasmtime_environ::Module`

Do not hold regular `String`s; instead use our own OOM-handling
`wasmtime_core

Refactor `String` usage in `wasmtime_environ::Module` (#12565)

* Refactor `String` usage in `wasmtime_environ::Module`

Do not hold regular `String`s; instead use our own OOM-handling
`wasmtime_core::alloc::String` or, even better, an interned `Atom` from the
`StringPool`.

Also avoid `IndexMap`, as it doesn't handle OOMs.

* Use OOM-handling `IndexMap` in `wasmtime_environ::Module`

* review feedback

show more ...

Share empty `ModuleRuntimeInfo`s across all stores in an engine (#12409)

* Share empty `ModuleRuntimeInfo`s across all stores in an engine

This avoids an `Arc`- and `Box`-allocation during `Store`

Share empty `ModuleRuntimeInfo`s across all stores in an engine (#12409)

* Share empty `ModuleRuntimeInfo`s across all stores in an engine

This avoids an `Arc`- and `Box`-allocation during `Store` creation.

* Fix no-runtime build

* Pin x86-64_macos build to nightly to avoid a rustc bug

show more ...

Do not re-export `anyhow!` in the `wasmtime::prelude` (#12298)

We are trying to move to `format_err!` instead.

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtim

Migrate the `wasmtime` crate to `wasmtime_environ::error::*` (#12231)

* Migrate the `wasmtime` crate to `wasmtime_environ::error::*`

Instead of `anyhow::Error`.

This commit re-exports the `wasmtime_environ::error` as the `wasmtime::error`
module, updates the prelude to include these new error-handling types, redirects
our top-level `wasmtime::{Error, Result}` re-exports to re-export
`wasmtime::error::{Error, Result}`, and updates various use sites that were
directly using `anyhow` to use the new `wasmtime` versions.

This process also required updating the component macro and wit-bindgen macro to
use the new error types instead of `anyhow`.

Part of https://github.com/bytecodealliance/wasmtime/issues/12069

* Replace wasmtime::error::Thing with wasmtime::Thing where it makes sense

* cargo fmt

* Move `crate::error::Thing` to `crate::Thing` where it makes sense

show more ...

Remove `AllCallFunc` (#11694)

* Remove `AllCallFunc`

And add `FuncKeyKind` and `FuncKeyNamespace` types.

Split out from https://github.com/bytecodealliance/wasmtime/pull/11630

* fix warning and d

Remove `AllCallFunc` (#11694)

* Remove `AllCallFunc`

And add `FuncKeyKind` and `FuncKeyNamespace` types.

Split out from https://github.com/bytecodealliance/wasmtime/pull/11630

* fix warning and disriminant gap

* add module arg to `Module::new()` calls in tests

show more ...

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "widened" into encompassing the limiter on
the `T` in `StoreInner<T>`, for example, by using the self-pointer
located in an instance or the store. This fix is done by threading
`&mut StoreOpaque` as a parameter separately from a
`StoreResourceLimiter`. This means that various callers now take a new
`Option<&mut StoreResourceLimiter<'_>>` parameter in various locations.

Closes #11409

* Fix gc-less build

show more ...

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `Store

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `StoreOpaque::allocate_instance` is now an `async fn`.
This function does not actually use the `async`-ness just yet so it's a
noop from that point of view, but this propagates outwards to enough
locations that I wanted to split this off to make future changes more
digestable.

Notably some creation functions here such as making an `Instance`,
`Table`, or `Memory` are refactored internally to use this new `async`
function. Annotations of `assert_ready` or `one_poll` are used as
appropriate as well.

For reference this commit was benchmarked with our `instantiation.rs`
benchmark in the pooling allocator and shows no changes relative to the
original baseline from before-`async`-PRs.

* Make table/memory creation `async` functions

This commit is a large-ish refactor which is made possible by the many
previous refactorings to internals w.r.t. async-in-Wasmtime. The end
goal of this change is that table and memory allocation are both `async`
functions. Achieving this, however, required some refactoring to enable
it to work:

* To work with `Send` neither function can close over `dyn VMStore`.
This required changing their `Option<&mut dyn VMStore>` arugment to
`Option<&mut StoreResourceLimiter<'_>>`
* Somehow a `StoreResourceLimiter` needed to be acquired from an
`InstanceAllocationRequest`. Previously the store was stored here as
an unsafe raw pointer, but I've refactored this now so
`InstanceAllocationRequest` directly stores `&StoreOpaque` and
`Option<&mut StoreResourceLimiter>` meaning it's trivial to acquire
them. This additionally means no more `unsafe` access of the store
during instance allocation (yay!).
* Now-redundant fields of `InstanceAllocationRequest` were removed since
they can be safely inferred from `&StoreOpaque`. For example passing
around `&Tunables` is now all gone.
* Methods upwards from table/memory allocation to the
`InstanceAllocator` trait needed to be made `async`. This includes new
`#[async_trait]` methods for example.
* `StoreOpaque::ensure_gc_store` is now an `async` function. This
internally carries a new `unsafe` block carried over from before with
the raw point passed around in `InstanceAllocationRequest`. A future
PR will delete this `unsafe` block, it's just temporary.

I attempted a few times to split this PR up into separate commits but
everything is relatively intertwined here so this is the smallest
"atomic" unit I could manage to land these changes and refactorings.

* Shuffle `async-trait` dep

* Fix configured build

show more ...

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
wi

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
with a number of provided impls. The `InstanceAllocator` trait is
implemented for everything implementing `InstanceAllocatorImpl` to force
users to be unable to override the default methods. When adding `async`
support internally to Wasmtime these are going to need to be
`#[async_trait]`-annotated-traits which adds a cost to `async` functions
as a future needs to be heap-allocated.

The goal of this commit is to make this future `async`-ification a bit
more optimal. Notably the `InstanceAllocator` trait is removed and
replaced with inherent methods on `impl dyn InstanceAllocatorImpl`.
After that the previous `InstanceAllocatorImpl` trait was renamed to
`InstanceAllocator` meaning that there's just one `InstanceAllocator`
trait which has inherent methods which cannot be overridden. A
consequence of this is that the inherent methods are also forced to do
virtual dispatch unlike before where they would internally use
monomorphization to have static dispatch. Given the complexity of
instance allocation this is expected to be a negligible cost, however.

The main benefit is that `allocate_module`, `allocate_tables`, and
`allocate_memories` all get to be native `async` functions without the
cost of `#[async_trait]`. Only allocation of a single table/memory will
require an allocation of a future which in profiling helps reduce the
cost of instantiation slightly.

Note that `#[async_trait]` is not currently used, this commit is just
preparation for its eventual use.

show more ...

Raise `unsafe_op_in_unsafe_fn` further in Wasmtime (#11322)

* Raise `unsafe_op_in_unsafe_fn` further in Wasmtime

Now it's at `wasmtime::runtime`, not just `wasmtime::runtime::vm`.

* Review comments

Make table/memory allocation functions safe (#11320)

These were previously marked as `unsafe` trait methods with a
requirement that the memory/table shape must be validated ahead of time.
Neither th

Make table/memory allocation functions safe (#11320)

These were previously marked as `unsafe` trait methods with a
requirement that the memory/table shape must be validated ahead of time.
Neither the ondemand nor pooling allocator actually has an unsafe
contract to uphold with respect to this and both may assert/reject
non-validated shapes but memory unsafety won't happen as a result.
Consequently these functions are made safe.

Instance allocation functions are adjusted to reflect how the
correctness of `imports` is required for the functions to be safe.

show more ...

Avoid cloning and dropping `Arc`s in `LocalMemory::vmmemory` (#11148)

* Avoid cloning and dropping `Arc`s in `LocalMemory::vmmemory`

Getting the base pointer from the underlying `dyn RuntimeLinearM

Avoid cloning and dropping `Arc`s in `LocalMemory::vmmemory` (#11148)

* Avoid cloning and dropping `Arc`s in `LocalMemory::vmmemory`

Getting the base pointer from the underlying `dyn RuntimeLinearMemory` involved
getting a `MemoryBase` which is potentially an `MmapOffset` which itself
contains an `Arc<Mmap>`, and we would then call `base.as_non_null()` to turn
this into a raw pointer, and then we would drop the `MemoryBase` which
ultimately drops the `Arc<Mmap>`. So that's an `Arc` clone and drop just to get
a `VMMemoryDefinition`, which is just a pointer and a length, essentially a
slice of the linear memory. And, among other places, we call
`LocalMemory::vmmemory` to get the GC heap's memory base and bound every time we
access a GC object from Rust (for example, during collections).

Altogether, this removes another 30% of runtime from the testcase in
https://github.com/bytecodealliance/wasmtime/issues/11141

* Construct `VMMemoryDefinition`s in the `RuntimeLinearMemory` implementation

Instead of in `LocalMemory` via calling various `RuntimeLinearMemory` methods,
as `LocalMemory` has a `dyn RuntimeLinearMemory` object so those calls are all
indirect.

show more ...

Fix globals/tables holding their registered types (#11103)

This commit fixes an issue where host-created tables and globals with
concrete reference types previously did not keep their associated typ

Fix globals/tables holding their registered types (#11103)

This commit fixes an issue where host-created tables and globals with
concrete reference types previously did not keep their associated type
registrations alive for the duration of the table or global itself. This
could lead to runtime panics when reflecting on their type and
additionally lead to some type confusion about the global/table itself.
As described in #11102 this is not a security issue, just a bug that
needs fixing.

Closes #11102

show more ...

Remove Stored from wasmtime::Instance (#10909)

* Remove the "export cache" on instances

This is now a relic of the past now that conversion from the internal
to external representation of Wasmtime

Remove Stored from wasmtime::Instance (#10909)

* Remove the "export cache" on instances

This is now a relic of the past now that conversion from the internal
to external representation of Wasmtime items is free. This is
effectively dead code that is no longer needed.

* Remove `Stored` from `wasmtime::Instance`

Powered by all previous commits this is a near-trivial change where an
`Instance` is now more-or-less "just" an `InstanceId`. Additionally the
`host_state: Box<dyn Any>` is no longer needed within `vm::Instance` so
that was removed as well.

show more ...

Update pinned nightly used in CI (#10893)

* Update pinned nightly used in CI

Resolves some various warnings cropping up from rustc/clippy. This does
trigger what for us is a false positive (I belie

Update pinned nightly used in CI (#10893)

* Update pinned nightly used in CI

Resolves some various warnings cropping up from rustc/clippy. This does
trigger what for us is a false positive (I believe) dead code warning
which I've annotated as `#[allow]` for now (alas this can't be
`#[expect]` because it's a nighly-only warning).

* Remove dead methods when component-model is disabled

show more ...

Store an `InstanceId` in `vm::Instance` (#10872)

* Store an `InstanceId` in `vm::Instance`

This commit is the beginning of an effort to clean up `unsafe`-related
code about how we manage instances

Store an `InstanceId` in `vm::Instance` (#10872)

* Store an `InstanceId` in `vm::Instance`

This commit is the beginning of an effort to clean up `unsafe`-related
code about how we manage instances in the guts of Wasmtime. This doesn't
have any affect on the public API but it's will result eventually in
changes to the internals of types in the public API. This change is also
done in preparation for simplifying `Stored<T>` and how it works in
Wasmtime to avoid requiring allocations (aka pushing onto a `StoreData`
vector) when converting from Wasmtime's internal representation to the
external representation of tables, globals, memories, etc.

For now all this commit does is add `id: InstanceId` to the
`vm::Instance` structure. This led to some refactorings to ensure that
it's correctly listed by centralizing various methods to allocate an
instance into one helper method which manages this configuration.

* Review comments

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Reuse Wasm linear memories code for GC heaps (#10503)

* Reuse code for Wasm linear memories for GC heaps

Instead of bespoke code paths and structures for Wasm GC, this commit makes it
so that we no

Reuse Wasm linear memories code for GC heaps (#10503)

* Reuse code for Wasm linear memories for GC heaps

Instead of bespoke code paths and structures for Wasm GC, this commit makes it
so that we now reuse VM structures like `VMMemoryDefinition` and bounds-checking
logic. Notably, we also reuse all the associated bounds-checking optimizations
and, when possible, virtual-memory techniques to completely elide them.

Furthermore, this commit adds support for growing GC heaps, reusing the
machinery for growing memories, and makes it so that GC heaps always start out
empty. This allows us to properly delay allocating the GC heap's storage until a
GC object is actually allocated.

Fixes #9350

* fix c api compilation

* use assert_contains

* remove no-longer-necessary extra memory config from limiter tests

* Helper for retry-after-maybe-async-gc in libcalls

* Clean up some comments

* fix wasmtime-fuzzing and no-gc compilation

* fix examples

* fix no-gc+compiler build

* fix build without pooling allocator

* fix +cranelift +gc-drc -gc-null builds

* fix table hash key stability test

* fix oracle usage of `ExternRef::new`

* fix +gc -gc-null -gc-drc build

* fix wasmtime-fuzzing

* make `StorePtr` wrap a `NonNull`

* Fix some doc tests

* Remove some unnecessary retry helpers now that `FooRef::new` will auto-gc

* fix things after rebase

* Reorganize collection/growth methods for GC heap

* rename BoundsCheck variants

* fix cfg'ing of gc only code

* Fix doc tests

* fix one more gc cfg

* disable GC heap OOM test on non-64-bit targets

show more ...

Do not reorder/pack struct fields (#10463)

* Do not reorder/pack struct fields

We were not considering that when a type T is a subtype of type U, that all of
the fields that T and U share must be a

Do not reorder/pack struct fields (#10463)

* Do not reorder/pack struct fields

We were not considering that when a type T is a subtype of type U, that all of
the fields that T and U share must be at the same offsets. Our
reordering/packing algorithm did not consider this. So we stop reordering struct
fields for now.

This also means that we cannot guarantee that all GC refs in an object are
contiguously packed anymore. Instead, we maintain a map keyed by
`VMSharedTypeIndex`, whose value is all the offsets of outgoing GC
references for instances of that type. Tracing looks up the current object's
type in this map, and then extracts each of the GC edges.

This new tracing scheme means that we no longer need to store the number of
outgoing edges in the GC object header, and can put the object size in there
again, reducing the DRC header from 24 to 16 bytes.

Fixes #10459
Fixes #10464

* Correctly handle `Layout` errors in `libcalls::gc_alloc_raw`

show more ...

Enable warnings if `gc` is disabled (#10149)

* Enable warnings if `gc` is disabled

Continuation of work in #10131. This additionally handles turning off
`gc-null` and `gc-drc` and the various combi

Enable warnings if `gc` is disabled (#10149)

* Enable warnings if `gc` is disabled

Continuation of work in #10131. This additionally handles turning off
`gc-null` and `gc-drc` and the various combinations within.

* Fix some more warnings

* Fix a feature combo build

show more ...

Move MemoryImageSource::map_at to mmap module (#9687)

* Simplify mmap interface slightly

Return a single `SendSyncPtr` -- the platform-independent context converts to
the various raw pointer types

Move MemoryImageSource::map_at to mmap module (#9687)

* Simplify mmap interface slightly

Return a single `SendSyncPtr` -- the platform-independent context converts to
the various raw pointer types as desired. This simplifies upcoming work where I
wanted to return a `SendSyncPtr`.

* Move MemoryImageSource::map_at to mmap module

This is part of the work to centralize memory management into the `mmap`
module. This commit introduces a few structures which aid in that process, and
starts converting one of the functions (`MemoryImageSource::map_at`) into this
module.

The structures introduced are:

* `MemoryBase`: `RuntimeLinearMemory::base_ptr` is now
`RuntimeLinearMemory::base`, which returns a `MemoryBase`. This is either a
raw pointer or an `MmapOffset` as described below.

* `MmapOffset`: A combination of a reference to an mmap and an offset into it.
Logically represents a pointer into a mapped section of memory.

In future work, we'll move more image-mapping code over to `Mmap` instances.

show more ...

Remove the `MemoryStyle` enum (#9583)

* Remove the `MemoryStyle` enum

There are no more major users left, only a few minor pieces here and
there. This should complete the refactoring to delete the

Remove the `MemoryStyle` enum (#9583)

* Remove the `MemoryStyle` enum

There are no more major users left, only a few minor pieces here and
there. This should complete the refactoring to delete the "static" and
"dynamic" terminology from Wasmtime and all we're left with are linear
memories with a few knobs in `Tunables`.

* Fix test compile

show more ...