Fix `OwnedRooted<T>::wasm_ty_store` missing `i31ref` check (#12451)

Force callers to handle allocation failure in `Slab` (#12450)

* Add `Vec::drain` method

* Force callers to handle allocation failure in `Slab`

Forbid rustdoc warnings in CI (#12420)

* Forbid rustdoc warnings in CI

This commit corrects our handling of rustdoc flags in CI to ensure that
warnings indeed fire. Additionally this changes our fl

Forbid rustdoc warnings in CI (#12420)

* Forbid rustdoc warnings in CI

This commit corrects our handling of rustdoc flags in CI to ensure that
warnings indeed fire. Additionally this changes our flags to pass
`-Dwarnings` to ensure that we have warning-free doc builds when all
features are enabled at least.

There were quite a lot of preexisting issues to fix, so this
additionally goes through and fixes all the warnings that cropped up.

* Update nightly toolchain again

prtest:full

* Update another nightly

* Fix a warning in generated code

show more ...

Move `wasmtime-slab` crate into `wasmtime-core` (#12407)

Consolidating core data structures in one location

Do not re-export `anyhow!` in the `wasmtime::prelude` (#12298)

We are trying to move to `format_err!` instead.

GC: replace ManuallyRooted with OwnedRooted. (#11514)

* GC: replace ManuallyRooted with OwnedRooted.

This implements the ideas from #11445: it replaces `ManuallyRooted`,
which requires an explicit

GC: replace ManuallyRooted with OwnedRooted. (#11514)

* GC: replace ManuallyRooted with OwnedRooted.

This implements the ideas from #11445: it replaces `ManuallyRooted`,
which requires an explicit unroot action with a mut borrow of the store
(making it impossible to implement in a standard `Drop` impl), with
`OwnedRooted`, which holds an `Arc` only to a small auxiliary memory
allocation (an `Arc<()>`) and uses this externalized "liveness flag" to
allow for a `Store`-less drop. These liveness flags are scanned during a
"trim" pass, which happens both when new owned roots are created, and
just before a GC.

This should greatly increase safety for host-side users of GC: it
provides a way to have a handle whose ownership works like any other
Rust value, alive until dropped. It is still not quite as efficient as
LIFO-scoped handles (by analogy, for the same reason that
individually-freed RAII types are not as efficient as arena allocation),
so those remain for efficiency-minded users that have a clear picture of
reference lifetimes.

At some later time we may wish to use `OwnedRooted` exclusively in our
public APIs rather than `Rooted`, and we may wish to rename `Rooted` to
`ScopedRooted`, but I haven't done either of those things yet.

I opted to *replace* `ManuallyRooted` rather than add a third kind of
root, after discussion with fitzgen. One implication of this is that the
C API's `anyref` and `externref` types are now 24 or 20 bytes rather
than 16 (because of the `Arc` pointer), and correspondingly the Val
union grew to that size. I *believe* this is an acceptable tradeoff, but
I'm happy to put `ManuallyRooted` back if not.

Fixes #11445.

* Review feedback.

* Fix for riscv32imac: loosen asserts on struct size slightly to allow for different padding.

* C API: use a `*const ()` to pass the liveness-flag Arc through C.

* Add some additional documentation warning about unrooting in the C API.

* Fix size-of test on 32-bit platforms.

* New amortized algorithm for root trimming.

* core::cmp, not std::cmp

* Always set high-water mark, even if eager.

* Make val size-assert tolerant of padding bytes in crates/c-api/src/val.rs too.

show more ...

Delete `StoreOpaque::traitobj_mut` (#11444)

* Delete `StoreOpaque::traitobj_mut`

This was a fundamentally unsound function because it is widening a
mutable borrow to encompass more than it original

Delete `StoreOpaque::traitobj_mut` (#11444)

* Delete `StoreOpaque::traitobj_mut`

This was a fundamentally unsound function because it is widening a
mutable borrow to encompass more than it originally contained. Removal
of its usage falls into a few buckets here:

* Many instances of `store.0.traitobj_mut()` were starting from a
`&mut StoreInner<T>`, going to `&mut StoreOpaque`, then going back to
`&mut dyn VMStore`. These are replaced with `store.0` as
`&mut StoreInner<T>` directly coerces into `&mut dyn VMStore`.

* Some fiber-related helpers were updated to take any store-opaque-thing
which encapsulates the ability to start/resume a borrow with a part of
the store, but you only get that part of the store during the fiber
itself. This means there's no widening necessary.

* Some methods previously taking `&mut StoreOpaque` are now
appropriately widened to `&mut dyn VMStore`.

This all enables full deletion of this function which prevents
accidentally ever tripping over its unsound-ness.

* Fix configured build

* Another fix for a configured build

* Fix a warning

* Don't hold `VMStore` live over await points

* Review comments

* Move `AsStoreOpaque` to `store.rs` and use it in GC

* Fix configured build

* Switch lint annotation

show more ...

Minimize lazy allocation of the GC store (#11411)

* Minimize lazy allocation of the GC store

This commit is an effort to minimize the number of entrypoints which
might lazily allocate a GC store. T

Minimize lazy allocation of the GC store (#11411)

* Minimize lazy allocation of the GC store

This commit is an effort to minimize the number of entrypoints which
might lazily allocate a GC store. The is currently done through
`StoreOpaque::gc_store_mut` but this method is very commonly used
meaning that there are many many places to audit for lazily allocating a
GC store. The reason that this needs an audit is that lazy allocation
is an async operation right now that must be on a fiber and is something
I'm looking to fix as part of #11262.

This commit performs a few refactorings to achieve this:

* `gc_store_mut` is renamed to `ensure_gc_store`. This is intended to be
an `async` function in the future and clearly demarcates where lazy
allocation of a GC store is occurring.

* `require_gc_store{,_mut}` is now added which is a pure accessor of the
GC store with no lazy allocation. Most locations previously using
`gc_store_mut` are updated to use this instead.

Documentation is added to store methods to clearly indicate which ones
are allocating and which ones should only be called in a context where
allocation should already have happened.

* Fix configured build

* Relax GC store restrictions in more places

* Review comments on documentation

* Move `ensure_gc_store` calls during instantiation

Instead update `needs_gc_heap` with the tables that are added to a
module and rely on instantiation to create the GC heap.

* Shuffle around some code

* Fix CI and review comments

* Add in a few more i31 cases for externref

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions (#10560)

* Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions

Rather than forcing all callers to

Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions (#10560)

* Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions

Rather than forcing all callers to check for `GcHeapOutOfMemory`, trigger a GC,
and then try again. This does force us to define `*_async` variations for when
async is enabled, however; it's ultimately worth it.

* cargo fmt

* review feedback and fix tests

* fix +runtime -gc build

* more feedback and build cfg fixes

* remove copy-paste assertion that doesn't apply to this method

* move assertion into retry methods

show more ...

Analyze whether a module requires a GC heap during its compilation (#10508)

* Analyze whether a module requires a GC heap during its compilation

And avoid forcing the allocation of a GC heap for mo

Analyze whether a module requires a GC heap during its compilation (#10508)

* Analyze whether a module requires a GC heap during its compilation

And avoid forcing the allocation of a GC heap for modules that will not use it.

Split out from https://github.com/bytecodealliance/wasmtime/issues/9350

* Simplify condition for allocating GC heap

show more ...

Avoid some unnecessary reduces and extends around GC libcalls (#10458)

We are logically returning a `VMGcRef` from these libcalls, which is a
`NonZeroU32` under the covers. We were representing that

Avoid some unnecessary reduces and extends around GC libcalls (#10458)

We are logically returning a `VMGcRef` from these libcalls, which is a
`NonZeroU32` under the covers. We were representing that value as a `u32`, and
then to represent traps and panics, we packed that into a `u64` and if the `u64`
was negative that meant there was a trap/panic but otherwise we could unpack the
bottom half and that was our result.

But because our GC refs are non-zero, we don't need to extend to a `u64` to get
a sentinel for traps/panics; we can just use zero. That's what this commit does.

show more ...

Transitively dec-ref objects in the DRC collector (#10401)

* Transitively dec-ref objects in the DRC collector

Previously, the DRC collector only performed shallow reference counting where if
an ob

Transitively dec-ref objects in the DRC collector (#10401)

* Transitively dec-ref objects in the DRC collector

Previously, the DRC collector only performed shallow reference counting where if
an object's ref count was decremented and reached zero, then we would deallocate
that object, but we would not decrement the reference counts of any other object
that was referenced by it. This commit changes that, so that now we will
decrement the reference counts of other objects referenced by an object that is
being deallocated.

This requires some method of tracing an object's outgoing edges. To avoid
needing to keep around and dispatch upon type-specific information about which
struct fields are GC references, for example, we instead pack all GC references
at the start of the object, just after its header. We additionally keep a count
of how many GC references an object has in its header. This allows us to create
a slice of any object's GC references, and we can uniformly trace any GC
object's outgoing edges. We don't need to reflect on the GC object's actual
type.

A final detail: the DRC collector was previously storing the object size in the
common GC header's reserved bits. It now stores the number of GC refs there
instead, and the object size is stored in its own field in the DRC-specific
header, next to the ref count. This is nice because, since we are making the DRC
header larger anyways, it raises the DRC collector's implementation limit on
object size from `u27::MAX` to `u32::MAX`.

Fixes #9701

* update disas tests

* Remove newly-added debug-assert that isn't true in certain edge cases

Sometimes we need to deallocate uninitialized objects when initializing their
fields fails, and this assertion trips.

* fix some tests by returning an error of the correct kind of trap

rather than a generic anyhow error

* fix tracing GC refs on big-endian architectures

We can't cast to a slice, we need to do conversion from little-endian bytes.

show more ...

Make `GcStore::expose_gc_ref_to_wasm` return the raw GC ref (#10375)

Every caller of `expose_gc_ref_to_wasm` except for one does this weird dance
where they get the raw representation of the GC ref

Make `GcStore::expose_gc_ref_to_wasm` return the raw GC ref (#10375)

Every caller of `expose_gc_ref_to_wasm` except for one does this weird dance
where they get the raw representation of the GC ref before giving up ownership
of it to the `expose_gc_ref_to_wasm` call, and then do something with the raw GC
ref (like return it to Wasm or whatever).

This small refactoring boxes up that dance inside `expose_gc_ref_to_wasm` so
that callers don't have to do it themselves anymore. Makes things that much more
concise and harder to mess up.

show more ...

Refactor `vm::Instance` for better `Store` safety (#9565)

* Refactor `vm::Instance` for better `Store` safety

At various times, for example in libcalls, we need to go from a raw `vmctx`
pointer tha

Refactor `vm::Instance` for better `Store` safety (#9565)

* Refactor `vm::Instance` for better `Store` safety

At various times, for example in libcalls, we need to go from a raw `vmctx`
pointer that Wasm gave us to both an `&mut Instance` and an `&mut
Store{Opaque,Context}`. This is a pretty fundamental unsafe aspect of
implementing a language runtimne in Rust, but we can tighten things up a bit and
make them a little safer than they currently are. This commit is such an
attempt and is notably tackling the issue of creating multiple store borrows
after we have an instance borrow.

This commit makes the following changes:

* `Instance` doesn't expose a method to get the raw `*mut dyn VMStore` pointer
or otherwise create a store borrow.

* You cannot construct an `Instance` directly from a `vmctx` pointer.

* There is now an `InstanceAndStore` type that represents unique access to both
an `Instance` and a `Store`.

* You can (with `unsafe`) create an `InstanceAndStore` from a raw `vmctx`
pointer. This generally only happens inside a couple "bottlenecks", not all
throughout the codebase, like the shared plumbing code for libcalls.

* The `InstanceAndStore` can be unpacked into a mutable borrow of an `Instance`
and a mutable borrow of a `Store`. This unpacking takes holds a mutable borrow
of the original `InstanceAndStore`, so double borrows of the store are no
longer a concern, so long as you don't use `unsafe` to create new
`InstanceAndStore`s for the same `vmctx`.

* All `Instance` methods and functions that previously would unsafely turn the
`Instance`'s internal store pointer into a store borrow to perform some action
now take a store argument, and this store is threaded around as necessary.

Altogether, I feel that we've ended up with an architecture that is a safety
improvement over where we were previously, and will help us properly avoid Rust
UB.

For what its worth, I do not think this is the end state. I foresee at least two
additional follow ups that I unfortunately do not currently have time for:

1. Create `ComponentInstanceAndStore`, which is the same thing that this commit
did but for components. I started on this but ran out of fuel while trying to
update macros for all the component shims and transcoders.

2. Stop dealing with `&mut vm::Table`s and `&mut vm::Global`s and all that in
the runtime. Instead just deal with indices, similar to how things are
structured in the host API level. This refactoring was not previously possible
due to (1) the `wasmtime` versus `wasmtime-runtime` crate split and (2) the lack
of `StoreOpaque`s threaded through the VM internals. The first blocker was
addressed a few months ago, this commit removes the second blocker. This will
still be a pretty large refactoring though, but I think ultimately will be worth
it.

* Fix wmemcheck build

* Remove `'static` from the `dyn VMStore` accessors; fix no-gc build

show more ...

Implement the Wasm GC instructions for converting between `anyref` and `externref` (#9435)

* Implement the Wasm GC instructions for converting between `anyref` and `externref`

This commit implement

Implement the Wasm GC instructions for converting between `anyref` and `externref` (#9435)

* Implement the Wasm GC instructions for converting between `anyref` and `externref`

This commit implements two instructions:

1. `any.convert_extern`
2. `extern.convert_any`

These instructions are used to convert between `anyref` and `externref`
values. The `any.convert_extern` instruction takes an `anyref` value and
converts it to an `externref` value. The `extern.convert_any` instruction takes
an `externref` value and converts it to an `anyref` value.

Rather than implementing wrapper objects -- for example an `struct
AnyOfExtern(ExternRef)` type that is a subtype of `AnyRef` -- we instead reuse
the same representation converted references as their unconverted reference. For
example, `(any.convert_extern my_externref)` is identical to the original
`my_externref` value. This means that we don't actually emit any clif
instructions to implement these Wasm instructions; they are no-ops!

Wasm code remains none-the-wiser because it cannot directly test for the
difference between, for example, a `my_anyref` and the result of
`(extern.convert_any my_anyref)` because they are in two different type
hierarchies, so any direct `ref.test` would be invalid. The Wasm code would have
to convert one into the other's type hierarchy, at which point it doesn't know
whether wrapping/unwrapping took place.

We did need some changes at the host API and host API implementation levels,
however:

* We needed to relax the requirement that a `wasmtime::AnyRef` only wraps a
`VMGcRef` that points to an object that is a subtype of `anyref` and similar for
`wasmtime::ExternRef`.

* We needed to make the `wasmtime::ExternRef::data[_mut]` methods return an
option of their associated host data, since any `externref` resulting from
`(extern.convert_any ...)` does not have any associated host data. (This change
would have been required either way, even if we used wrapper objects.)

* fix tests

* fix wasmtime-environ tests

show more ...

Implement the `array.copy` Wasm GC instruction (#9389)

* Implement the `array.copy` Wasm GC instruction

This also involved fixing some `VMGcRef` initialization and GC barrier code for
globals. That

Implement the `array.copy` Wasm GC instruction (#9389)

* Implement the `array.copy` Wasm GC instruction

This also involved fixing some `VMGcRef` initialization and GC barrier code for
globals. That in turn involved making global initialization fallible, which
meant that it needed to be pulled out of `vmctx` initialization and put into
instance initialization.

Co-Authored-By: Alex Crichton <[email protected]>

* fix compile error due to code moving where a trait method was no longer in scope

* use the result of `MaybeUninit::write`

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Add constant evaluation support for the `struct.new[_default]` Wasm GC instructions (#9282)

* Add constant evaluation support for the `struct.new[_default]` Wasm GC instructions

* fix no-gc builds

Add constant evaluation support for the `struct.new[_default]` Wasm GC instructions (#9282)

* Add constant evaluation support for the `struct.new[_default]` Wasm GC instructions

* fix no-gc builds

* wrap const evaluation in `AutoAssertNoGc`

* add assertions that we access `VMGcRef`s in native-endian

* fix an endian mismatch bug between compiled wasm code and host code

show more ...

Switch to new "user" stack maps and use `i32` for GC refs in Wasmtime (#9082)

This moves Wasmtime over from the old, regalloc-based stack maps system to the
new "user" stack maps system.

Removing t

Switch to new "user" stack maps and use `i32` for GC refs in Wasmtime (#9082)

This moves Wasmtime over from the old, regalloc-based stack maps system to the
new "user" stack maps system.

Removing the old regalloc-based stack maps system is left for follow-up work.

show more ...

Improve unsupported architecture errors in Wasmtime (#9039)

Don't rely purely on `compile_error!` which doesn't halt compilation.
Instead fill out some stubs which won't ever actually get compiled b

Improve unsupported architecture errors in Wasmtime (#9039)

Don't rely purely on `compile_error!` which doesn't halt compilation.
Instead fill out some stubs which won't ever actually get compiled but
prevent other compiler errors from cropping up.

show more ...

Reduce typo count (#8951)

Remove `unchecked_*` methods on `Rooted`s for getting `VMGcRef`s (#8948)

Because they take a shared borrow of the store and return a shared borrow of
the `VMGcRef` with the same lifetime, and becaus

Remove `unchecked_*` methods on `Rooted`s for getting `VMGcRef`s (#8948)

Because they take a shared borrow of the store and return a shared borrow of
the `VMGcRef` with the same lifetime, and because performing a GC requires a
mutable borrow of the store, there actually was no reason to differentiate
between checked and unchecked methods or require `AutoAssertNoGc` arguments.

Fixes https://github.com/bytecodealliance/wasmtime/issues/8940

show more ...

Deduplicate `WasmTy` implementations for GC-managed types (#8946)

Fixes https://github.com/bytecodealliance/wasmtime/issues/8942

Introduce `wasmtime::StructRef` and allocating Wasm GC structs (#8933)

* Introduce `wasmtime::StructRef` and allocating Wasm GC structs

This commit introduces the `wasmtime::StructRef` type and sup

Introduce `wasmtime::StructRef` and allocating Wasm GC structs (#8933)

* Introduce `wasmtime::StructRef` and allocating Wasm GC structs

This commit introduces the `wasmtime::StructRef` type and support for allocating
Wasm GC structs from the host. This commit does *not* add support for the
`struct.new` family of Wasm instructions; guests still cannot allocate Wasm GC
objects yet, but initial support should be pretty straightforward after this
commit lands.

The `StructRef` type has everything you expect from other value types in the
`wasmtime` crate:

* A method to get its type or check whether it matches a given type

* An implementation of `WasmTy` so that it can be used with `Func::wrap`-style
APIs

* The ability to upcast it into an `AnyRef` and to do checked downcasts in the
opposite direction

There are, additionally, methods for getting, setting, and enumerating a
`StructRef`'s fields.

To allocate a `StructRef`, we need proof that the struct type we are allocating
is being kept alive for the duration that the allocation may live. This is
required for many reasons, but a basic example is getting a struct instance's
type from the embedder API: this does a type-index-to-`StructType` lookup and
conversion and if the type wasn't kept alive, then the type-index lookup will
result in what is logically a use-after-free bug. This won't be a problem for
Wasm guests (when we get around to implementing allocation for them) since their
module defines the type, the store holds onto its instances' modules, and the
allocation cannot outlive the store. For the host, we need another method of
keeping the object's type alive, since it might be that the host defined the
type and there is no module that also defined it, let alone such a module that
is being kept alive in the store.

The solution to the struct-type-lifetime problem that this commit implements for
hosts is for the store to hold a hash set of `RegisteredType`s specifically for
objects which were allocated via the embedder API. But we also don't want to do
a hash lookup on every allocation, so we also implement a `StructRefPre` type. A
`StructRefPre` is proof that the embedder has inserted a `StructType`'s inner
`RegisteredType` into a store. Structurally, it is a pair of the struct type and
a store id. All `StructRef` allocation methods require a `StructRefPre`
argument, which does a fast store id check, rather than a whole hash table
insertion.

I opted to require `StructRefPre` in all allocation cases -- even though this
has the downside of always forcing callers to create one before they allocate,
even if they are only allocating a single object -- because of two
reasons. First, this avoids needing to define duplicate methods, with and
without a `StructRefPre` argument. Second, this avoids a performance footgun in
the API where users don't realize that they *can* avoid extra work by creating a
single `StructRefPre` and then using it multiple times. Anecdotally, I've heard
multiple people complain about instantiation being slower than advertised but it
turns out they weren't using `InstancePre`, and I'd like to avoid that situation
for allocation if we can.

* Move `allow(missing_docs)` up to `gc::disabled` module instead of each `impl`

* Rename `cast` to `unchecked_cast`

* fix `GcHeapOutOfMemory` error example in doc example

* document additional error case for `StructRef::new`

* Use `unpack` method instead of open-coding it

* deallocate on failed initialization

* Refactor field access methods to share more code

And define `fields()` in terms of `field()` rather than the other way around.

* Add upcast methods from structref to anyref

* Remove duplicate type checking and add clarifying comments about initializing vs writing fields

* make the `PodValType` trait safe

* fix benchmarks build

* prtest:full

* add miri ignores to new tests that call into wasm

show more ...

Wasmtime: Pop GC LIFO roots even when there is no GC heap (#8899)

* Wasmtime: Pop GC LIFO roots even when there is no GC heap

We can create and root `i31ref`s without ever allocating a GC heap for

Wasmtime: Pop GC LIFO roots even when there is no GC heap (#8899)

* Wasmtime: Pop GC LIFO roots even when there is no GC heap

We can create and root `i31ref`s without ever allocating a GC heap for the
store, so we can't guard popping LIFO roots on the presence of a GC heap or else
we risk unbounded growth of the LIFO root set.

* Fix build with the gc feature disabled

show more ...