Exceptions: implement C API. (#12861)

* Exceptions: implement C API.

This PR implements C (and C++) API support for Wasm exceptions, one
final remaining hurdle (aside from fuzz-testing) for making

Exceptions: implement C API. (#12861)

* Exceptions: implement C API.

This PR implements C (and C++) API support for Wasm exceptions, one
final remaining hurdle (aside from fuzz-testing) for making exceptions
tier-1 and on-by-default.

* Review feedback, and add exnref case to `wasmtime_val_t`.

* Review feedback: GC feature guard.

* clang-format

* add docs to exn.hh.

* Remove tag size asserts: broken on 32-bit platforms, but not needed for correctness wrt C struct.

show more ...

Fix externref/anyref ownership in C/C++ API (#11799)

* Fix externref/anyref ownership in C/C++ API

This commit is a follow-up to #11514 which was discovered through
failing tests in the wasmtime-py

Fix externref/anyref ownership in C/C++ API (#11799)

* Fix externref/anyref ownership in C/C++ API

This commit is a follow-up to #11514 which was discovered through
failing tests in the wasmtime-py repository when updating to Wasmtime
37.0.0. Effectively a combination of bugs in the Rust API meant that it
wasn't possible to use `externref` or `anyref` bindings correctly. The
Rust changes in this commit are:

* `wasmtime_val_unroot` correctly drops the value now as opposed to
effectively being a noop from before (typo of using `as_externref` vs
`from_externref`).
* `wasmtime_{anyref,externref,val}_t` now have a `Drop` implementation
in Rust to correctly drop them if a value in Rust is dropped. This is
required to correctly manage memory in the `wasmtime_func_{call,new}`
implementations, for example.
* `wasmtime_{anyref,externref,val}_clone` no longer have an unnecessary
context parameter.
* `wasmtime_{anyref,externref,val}_unroot` no longer have an unnecessary
context parameter.

Changes in the C/C++ APIs are:

* `Result::{ok,err}_ref` APIs were added in addition to the preexisting
rvalue accessors.
* Loading/storing typed arguments now has an overload for `const T&` and
`T&&` which behaves differently. Notably transferring ownership for
`T&&` and not for `const T&`. This means that passing parameters when
calling a wasm function uses `const T&`, but passing results from a
host import uses `T&&`.
* `TypedFunc::call` now uses `const Params&` instead of `Params` to
explicitly specify it doesn't modify the parameters and forces using
the `const T&` store method.
* `Store::gc` is now a convenience method for `store.context().gc()`
* `ExternRef`, `AnyRef`, and `Val` now have ownership semantics and
destructors. This matches the spirit of #11514 for Rust but models it
in C++ as well. This required filling out move/copy
constructors/assignments.
* The explicit `ExternRef` now takes `std::any` instead of `T`.
* Minor issues related to ownership are fixed in `Val` bindings.

Valgrind was used to ensure that there were no leaks for the test suite
which additionally resulted in a number of `*_delete` calls being added
to tests using the C API (accidental omissions).

The original goal of this change was to be a patch release for 37.0.1 to
enable updating wasmtime-py to the 37.0.x releases of Wasmtime. In the
end though the changes here were broad enough that I no longer feel that
this is a good idea, so wasmtime-py will be skipping the 37 version of
Wasmtime.

* Run `clang-format`

prtest:full

show more ...

cranelift: stack-switching support (#11003)

* cranelift: stack-switching support

This initial commit represents the "pr2" base commit with
minimal merge conflicts resolved. Due to OOB conflicts, t

cranelift: stack-switching support (#11003)

* cranelift: stack-switching support

This initial commit represents the "pr2" base commit with
minimal merge conflicts resolved. Due to OOB conflicts, this
commit is not functional as-is, but using it as a base in
order to allow for easier reviewing of the delta from
this commit to what will be used for the PR against upstream.

Co-authored-by: Daniel Hillerström <[email protected]>
Co-authored-by: Paul Osborne <[email protected]>

* cranelift: stack-switching updates pass 1

This first set of changes updates the base pr in order to
compiled and pass basic checks (compile, clippy, fmt) with
the biggest part of the change being to eliminate injection
of tracing/assertions in JIT'ed code.

* cranelift: stack-switching: restore original visibility for a few func_environ members

* cranelift: stack-switching conditional compilation

At this point, the only bit we really branch on is what we
do in order to avoid problems tying into wasmtime_environ.
This is basd on the approach and macro used by the gc code for
converting presence/absence of the cranelift feature flag to
cranelift compile time. This is a bit of a half-measure for now
as we still compile most stack-switching code in cranelift, but
this does enough to avoid causing problems with missing definitions
in wasmtime_environ.

* cranelift: avoid "as" casts in stack-switching

Replace either with infallible From or fallible, panicing
TryFrom alternatives where required.

* cranelift: cleanup stack-switching control_effect signatures

After removing emission of runtime trace logging and assertions,
there were several unused parameters. Remove those from the
ControlEffect signatures completely.

* cranelift: rename stack-switching VMArray to VMHostArray

This matches a change to the mirrored runtime type in
the upstream changes.

* stack-switching: fix typo

Co-authored-by: Daniel Hillerström <[email protected]>

* stack-switching: used Index impl for get_stack_slot_data

* stack-switching: use smallvec over vec in several cases

* stack-switching: avoid resumetable naming confusion

* stack-switching: cleanup unused params from unchecked_get_continuation

The extra parameters here used to be used for emitting runtime
assertions, but with those gone we just had unused params
and lifetimes, clean those out.

* stack_switching: simplify store_data_entries assertion

* stack-switching: simplify translate_table_{grow,fill} control flow

* stack-switching: remove translate_resume_throw stub

There's already a stub elsewhere and this is not called, when
exceptions are added and it is time to revisit, this method
can be restored.

* stack-switching: compute control_context_size based on target triple

* stack-switching: VMHostArrayRef updates

Rename VMHostArray -> VMHostArrayRef
Change impl to compute address with offset upfront rather than
on each load.

* stack-switching: move cranelift code to live under func_environ

This matches the directory structure for gc and aids in visibility
for a few members required by stack-switching code in cranelift.

* stack-switching: formatting fix

* stack-switching: reduce visibility on a few additional items

* stack-switching: simplify contobj fatptr con/de-struction

* stack-switching: add disas tests to cover new instructions

* stack-switching: fix layout of VMContObj

In the course of the various runtime updates, the layout of the
runtime VMContObj got switched around. This resulted in failures
when doing certain table operations on continuations.

This change fixes that layout problem and adds some tests with
offsets to avoid the problem. Due to the way that we interact
with the VMContObj in cranelift, we don't use these offsets outside
of the tests.

* Fix formatting of merge conflict resolution

* cranelift: remove ir::function::get_stack_slot_data

This method isn't required as sized_stack_slots is already pub.

* stack-switching: reduce visibility of a couple func_environ methods

* stack-switching: define VMContObj as two words

This change migrates VMContObj and its usages in cranelift and runtime
to work with the VMContObj fat pointer as two words in order to better
target different architectures (still gated to x86_64 for now).

To support this, a size type was plumbed into the builtins function
signature types (as is done for component types) that maps to
usize.

* fixup! stack-switching: define VMContObj as two words

* stack-switching: add stub Val::ContRef

This type is not fully complete until continuation/gc integration
is revisited (#10248) but without these changes, test cases are
now failing on panics as we need some representation of
continuation references in the runtime Val enumeration.

Runtime errors with TODO notes are added for the stubbed
code paths to revisit later.

* fixup! stack-switching: add stub Val::ContRef

* fixup! stack-switching: add stub Val::ContRef

* fixup! stack-switching: define VMContObj as two words

prtest:full

* stack-switching: don't conflate host and target pointer sizes

Disas tests were failing on i686 targeting x86_64 as the size of the
host pointer was leaking into what we were using to do codegen
in a few paths. This patch is a bit of a hack as it seems like
using a generic <T> for T: *mut u8 (as an example) is a bit
questionable. To keep things small, I do a hacky typecheck to map
pointers to the target pointer size here.

* stack-switching: VMHostArray entry sizes based off env PtrSize

Revisiting the previous commit with an approach that should be
less brittle.

---------

Co-authored-by: Frank Emrich <[email protected]>
Co-authored-by: Daniel Hillerström <[email protected]>

show more ...

GC: replace ManuallyRooted with OwnedRooted. (#11514)

* GC: replace ManuallyRooted with OwnedRooted.

This implements the ideas from #11445: it replaces `ManuallyRooted`,
which requires an explicit

GC: replace ManuallyRooted with OwnedRooted. (#11514)

* GC: replace ManuallyRooted with OwnedRooted.

This implements the ideas from #11445: it replaces `ManuallyRooted`,
which requires an explicit unroot action with a mut borrow of the store
(making it impossible to implement in a standard `Drop` impl), with
`OwnedRooted`, which holds an `Arc` only to a small auxiliary memory
allocation (an `Arc<()>`) and uses this externalized "liveness flag" to
allow for a `Store`-less drop. These liveness flags are scanned during a
"trim" pass, which happens both when new owned roots are created, and
just before a GC.

This should greatly increase safety for host-side users of GC: it
provides a way to have a handle whose ownership works like any other
Rust value, alive until dropped. It is still not quite as efficient as
LIFO-scoped handles (by analogy, for the same reason that
individually-freed RAII types are not as efficient as arena allocation),
so those remain for efficiency-minded users that have a clear picture of
reference lifetimes.

At some later time we may wish to use `OwnedRooted` exclusively in our
public APIs rather than `Rooted`, and we may wish to rename `Rooted` to
`ScopedRooted`, but I haven't done either of those things yet.

I opted to *replace* `ManuallyRooted` rather than add a third kind of
root, after discussion with fitzgen. One implication of this is that the
C API's `anyref` and `externref` types are now 24 or 20 bytes rather
than 16 (because of the `Arc` pointer), and correspondingly the Val
union grew to that size. I *believe* this is an acceptable tradeoff, but
I'm happy to put `ManuallyRooted` back if not.

Fixes #11445.

* Review feedback.

* Fix for riscv32imac: loosen asserts on struct size slightly to allow for different padding.

* C API: use a `*const ()` to pass the liveness-flag Arc through C.

* Add some additional documentation warning about unrooting in the C API.

* Fix size-of test on 32-bit platforms.

* New amortized algorithm for root trimming.

* core::cmp, not std::cmp

* Always set high-water mark, even if eager.

* Make val size-assert tolerant of padding bytes in crates/c-api/src/val.rs too.

show more ...

Implement exception objects. (#11230)

* WIP: Working exception objects

* Clean build with gc disabled (`cargo check -p wasmtime --no-default-features --features runtime`).

* Review feedback.

* St

Implement exception objects. (#11230)

* WIP: Working exception objects

* Clean build with gc disabled (`cargo check -p wasmtime --no-default-features --features runtime`).

* Review feedback.

* Stub out C-API support.

* Fix Clippy complaints.

* Fix dead-code warning in c-api build.

* Actually fix 27->26 reserved bit rename and test.

* Fix exnref doc-test.

* fix fuzzing build

* fix feature-flagging on Instance::id

* Bless disas test diff due to reserved-bits change.

* Review feedback.

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Add release binaries for 32-bit platforms (#10025)

This commit notably adds release binaries for:

* `i686-unknown-linux-gnu` - named `i686-linux`
* `i686-pc-windows-msvc` - named `i686-windows`
* `

Add release binaries for 32-bit platforms (#10025)

This commit notably adds release binaries for:

* `i686-unknown-linux-gnu` - named `i686-linux`
* `i686-pc-windows-msvc` - named `i686-windows`
* `armv7-unknown-linux-gnueabihf` - named `armv7-linux`

These artifacts all use Pulley for their execution of WebAssembly and
have been added to the tiers listing under tier 3.

show more ...

Enable `unsafe-attr-outside-unsafe` 2024 edition lint (#9964)

* Enable `unsafe-attr-outside-unsafe` 2024 edition lint

This commit enables the `unsafe-attr-outside-unsafe` lint in rustc used
in tran

Enable `unsafe-attr-outside-unsafe` 2024 edition lint (#9964)

* Enable `unsafe-attr-outside-unsafe` 2024 edition lint

This commit enables the `unsafe-attr-outside-unsafe` lint in rustc used
in transitioning to the 2024 edition. This requires that the
`#[no_mangle]` attribute is replaced in favor of `#[unsafe(no_mangle)]`.
This mostly affects the C API of wasmtime and most of the changes here
are a simple search/replace.

* Another attribute update

* Fix command adapter build

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

c-api: Remove allocations from `wasmtime_val_t` (#8451)

* c-api: Remove allocations from `wasmtime_val_t`

This commit redesigns how GC references work in the C API. previously
`wasmtime_{any,extern

c-api: Remove allocations from `wasmtime_val_t` (#8451)

* c-api: Remove allocations from `wasmtime_val_t`

This commit redesigns how GC references work in the C API. previously
`wasmtime_{any,extern}ref_t` were both opaque pointers in the C API
represented as a `Box`. Wasmtime did not, however, provide the ability
to deallocate just the `Box` part. This was intended to be handled with
unrooting APIs but unrooting requires a `wasmtime_context_t` parameter,
meaning that destructors of types in other languages don't have a
suitable means of managing the memory around the
`wasmtime_{any,extern}ref_t` which might lead to leaks.

This PR takes an alternate approach for the representation of these
types in the C API. Instead of being an opaque pointer they're now
actual structures with definitions in the header file. These structures
mirror the internals in Rust and Rust is tagged to ensure that changes
are reflected in the C API as well. This is similar to how
`wasmtime_func_t` matches `wasmtime::Func`. This enables embedders to
not need to worry about memory management of these values outside of the
manual rooting otherwise required.

The hope is that this will reduce the likelihood of memory leaks and
otherwise not make it any harder to manage references in the C API.

* Run clang-format

* Review comments

* Replace macros with inline functions

* Add explicit accessors/constructors for the C API

* Fix C example

* Fix doc link

* Fix some doc issues

prtest:full

* Fix doxygen links

show more ...

c-api: Tidy up some `wasmtime_func_t` usage (#8461)

* c-api: Tidy up some `wasmtime_func_t` usage

Try to avoid using a `transmute` in the implementation of the C API and
instead use a `union` like

c-api: Tidy up some `wasmtime_func_t` usage (#8461)

* c-api: Tidy up some `wasmtime_func_t` usage

Try to avoid using a `transmute` in the implementation of the C API and
instead use a `union` like is being done in #8451. Additionally add some
helper functions to the header to work with null funcref values instead
of only documenting the implementation.

* Try to fix doxygen

show more ...

c-api: Create `RootScope` where necessary (#8374)

* c-api: Create `RootScope` where necessary

This commit changes the `wasmtime_val_t::{from_val, to_val}` methods to
take a `RootScope` instead of a

c-api: Create `RootScope` where necessary (#8374)

* c-api: Create `RootScope` where necessary

This commit changes the `wasmtime_val_t::{from_val, to_val}` methods to
take a `RootScope` instead of any `AsContextMut`. This then required a
number of refactorings in callers to ensure that a `RootScope` was
created for any function that needed one. This is required to ensure
that GC references in the C API aren't forced to live for the entire
lifetime of the store.

This additionally added `*_unrooted` variants which do the same thing
but don't require `RootScope`. This was needed for when the C API calls
out to the embedder through a function call because a new `RootScope`
wouldn't work for return values (they're bound to a scope within the
closure when we want them to outlive the closure). In these situations
though we know a `RootScope` is already present at the entrypoint.

Closes #8367

* Review comments

show more ...

c-api: Fix alignment of `wasmtime_val_*` (#8363)

* c-api: Fix alignment of `wasmtime_val_*`

This commit fixes an issue where `wasmtime_val_raw_t` had an incorrect
alignment. In Rust `ValRaw` contai

c-api: Fix alignment of `wasmtime_val_*` (#8363)

* c-api: Fix alignment of `wasmtime_val_*`

This commit fixes an issue where `wasmtime_val_raw_t` had an incorrect
alignment. In Rust `ValRaw` contains a `u128` which has an alignment of
16 but in C the representation had a smaller alignment meaning that the
alignment of the two structures was different. This was seen to cause
alignment faults when structure were passed from C++ to Rust, for
example.

This commit changes the Rust representation of `ValRaw`'s `v128` field
to do the same as C which is to use `[u8; 16]`. This avoids the need to
raise the alignment in C which appears to be nontrivial. Cranelift is
appropriately adjusted to understand that loads/stores from `ValRaw` are
no longer aligned. Technically this only applies to the `v128` field but
it's not too bad to apply it to the other fields as well.

* Try alternate syntax for alignof

show more ...

wasmtime-c-api: Add support for GC references in `wasmtime.h` APIs (#8346)

Restores support for `externref` in `wasmtime_val_t`, methods for manipulating
them and getting their wrapped host data, an

wasmtime-c-api: Add support for GC references in `wasmtime.h` APIs (#8346)

Restores support for `externref` in `wasmtime_val_t`, methods for manipulating
them and getting their wrapped host data, and examples/tests for these things.

Additionally adds support for `anyref` in `wasmtime_val_t`, clone/delete methods
similar to those for `externref`, and a few `i31ref`-specific methods. Also adds
C and Rust example / test for working with `anyref`.

show more ...

`wasmtime-c-api`: Improve non-support of GC references in `wasm.h` APIs (#8345)

* c-api: Better differentiate between `wasm.h` and `wasmtime.h` APIs

This renames some types and adds some type alias

`wasmtime-c-api`: Improve non-support of GC references in `wasm.h` APIs (#8345)

* c-api: Better differentiate between `wasm.h` and `wasmtime.h` APIs

This renames some types and adds some type aliases to help us better distinguish
between `wasm.h` APIs and `wasmtime.h` APIs, primarily for `Store`-related
types. In general, `WasmFoo` is related to `wasm.h` and `WasmtimeFoo` is related
to `wasmtime.h`.

* `StoreRef` -> `WasmStoreRef`
* Introduce the `WasmStore[Data]` and `WasmStoreContext[Mut]` aliases
* `StoreData` -> `WasmtimeStoreData`
* `CStoreContext[Mut]` -> `WasmtimeStoreContext[Mut]`
* Introduce the `Wasmtime{Store,Caller}` aliases

* c-api: Improve non-support of GC references in `wasm.h` APIs

A couple small tweaks: error message improvements, exhaustive matching, etc...

show more ...

c-api: Better differentiate between `wasm.h` and `wasmtime.h` APIs (#8344)

This renames some types and adds some type aliases to help us better distinguish
between `wasm.h` APIs and `wasmtime.h` API

c-api: Better differentiate between `wasm.h` and `wasmtime.h` APIs (#8344)

This renames some types and adds some type aliases to help us better distinguish
between `wasm.h` APIs and `wasmtime.h` APIs, primarily for `Store`-related
types. In general, `WasmFoo` is related to `wasm.h` and `WasmtimeFoo` is related
to `wasmtime.h`.

* `StoreRef` -> `WasmStoreRef`
* Introduce the `WasmStore[Data]` and `WasmStoreContext[Mut]` aliases
* `StoreData` -> `WasmtimeStoreData`
* `CStoreContext[Mut]` -> `WasmtimeStoreContext[Mut]`
* Introduce the `Wasmtime{Store,Caller}` aliases

show more ...

Add `GcRuntime` and `GcCompiler` traits; `i31ref` support (#8196)

\### The `GcRuntime` and `GcCompiler` Traits

This commit factors out the details of the garbage collector away from the rest
of the

Add `GcRuntime` and `GcCompiler` traits; `i31ref` support (#8196)

\### The `GcRuntime` and `GcCompiler` Traits

This commit factors out the details of the garbage collector away from the rest
of the runtime and the compiler. It does this by introducing two new traits,
very similar to a subset of [those proposed in the Wasm GC RFC], although not
all equivalent functionality has been added yet because Wasmtime doesn't
support, for example, GC structs yet:

[those proposed in the Wasm GC RFC]: https://github.com/bytecodealliance/rfcs/blob/main/accepted/wasm-gc.md#defining-the-pluggable-gc-interface

1. The `GcRuntime` trait: This trait defines how to create new GC heaps, run
collections within them, and execute the various GC barriers the collector
requires.

Rather than monomorphize all of Wasmtime on this trait, we use it
as a dynamic trait object. This does imply some virtual call overhead and
missing some inlining (and resulting post-inlining) optimization
opportunities. However, it is *much* less disruptive to the existing embedder
API, results in a cleaner embedder API anyways, and we don't believe that VM
runtime/embedder code is on the hot path for working with the GC at this time
anyways (that would be the actual Wasm code, which has inlined GC barriers
and direct calls and all of that). In the future, once we have optimized
enough of the GC that such code is ever hot, we have options we can
investigate at that time to avoid these dynamic virtual calls, like only
enabling one single collector at build time and then creating a static type
alias like `type TheOneGcImpl = ...;` based on the compile time
configuration, and using this type alias in the runtime rather than a dynamic
trait object.

The `GcRuntime` trait additionally defines a method to reset a GC heap, for
use by the pooling allocator. This allows reuse of GC heaps across different
stores. This integration is very rudimentary at the moment, and is missing
all kinds of configuration knobs that we should have before deploying Wasm GC
in production. This commit is large enough as it is already! Ideally, in the
future, I'd like to make it so that GC heaps receive their memory region,
rather than allocate/reserve it themselves, and let each slot in the pooling
allocator's memory pool be *either* a linear memory or a GC heap. This would
unask various capacity planning questions such as "what percent of memory
capacity should we dedicate to linear memories vs GC heaps?". It also seems
like basically all the same configuration knobs we have for linear memories
apply equally to GC heaps (see also the "Indexed Heaps" section below).

2. The `GcCompiler` trait: This trait defines how to emit CLIF that implements
GC barriers for various operations on GC-managed references. The Rust code
calls into this trait dynamically via a trait object, but since it is
customizing the CLIF that is generated for Wasm code, the Wasm code itself is
not making dynamic, indirect calls for GC barriers. The `GcCompiler`
implementation can inline the parts of GC barrier that it believes should be
inline, and leave out-of-line calls to rare slow paths.

All that said, there is still only a single implementation of each of these
traits: the existing deferred reference-counting (DRC) collector. So there is a
bunch of code motion in this commit as the DRC collector was further isolated
from the rest of the runtime and moved to its own submodule. That said, this was
not *purely* code motion (see "Indexed Heaps" below) so it is worth not simply
skipping over the DRC collector's code in review.

\### Indexed Heaps

This commit does bake in a couple assumptions that must be shared across all
collector implementations, such as a shared `VMGcHeader` that all objects
allocated within a GC heap must begin with, but the most notable and
far-reaching of these assumptions is that all collectors will use "indexed
heaps".

What we are calling indexed heaps are basically the three following invariants:

1. All GC heaps will be a single contiguous region of memory, and all GC objects
will be allocated within this region of memory. The collector may ask the
system allocator for additional memory, e.g. to maintain its free lists, but
GC objects themselves will never be allocated via `malloc`.

2. A pointer to a GC-managed object (i.e. a `VMGcRef`) is a 32-bit offset into
the GC heap's contiguous region of memory. We never hold raw pointers to GC
objects (although, of course, we have to compute them and use them
temporarily when actually accessing objects). This means that deref'ing GC
pointers is equivalent to deref'ing linear memory pointers: we need to add a
base and we also check that the GC pointer/index is within the bounds of the
GC heap. Furthermore, compressing 64-bit pointers into 32 bits is a fairly
common technique among high-performance GC
implementations[^compressed-oops][^v8-ptr-compression] so we are in good
company.

3. Anything stored inside the GC heap is untrusted. Even each GC reference that
is an element of an `(array (ref any))` is untrusted, and bounds checked on
access. This means that, for example, we do not store the raw pointer to an
`externref`'s host object inside the GC heap. Instead an `externref` now
stores an ID that can be used to index into a side table in the store that
holds the actual `Box<dyn Any>` host object, and accessing that side table is
always checked.

[^compressed-oops]: See ["Compressed OOPs" in
OpenJDK.](https://wiki.openjdk.org/display/HotSpot/CompressedOops)

[^v8-ptr-compression]: See [V8's pointer
compression](https://v8.dev/blog/pointer-compression).

The good news with regards to all the bounds checking that this scheme implies
is that we can use all the same virtual memory tricks that linear memories use
to omit explicit bounds checks. Additionally, (2) means that the sizes of GC
objects is that much smaller (and therefore that much more cache friendly)
because they are only holding onto 32-bit, rather than 64-bit, references to
other GC objects. (We can, in the future, support GC heaps up to 16GiB in size
without losing 32-bit GC pointers by taking advantage of `VMGcHeader` alignment
and storing aligned indices rather than byte indices, while still leaving the
bottom bit available for tagging as an `i31ref` discriminant. Should we ever
need to support even larger GC heap capacities, we could go to full 64-bit
references, but we would need explicit bounds checks.)

The biggest benefit of indexed heaps is that, because we are (explicitly or
implicitly) bounds checking GC heap accesses, and because we are not otherwise
trusting any data from inside the GC heap, we greatly reduce how badly things
can go wrong in the face of collector bugs and GC heap corruption. We are
essentially sandboxing the GC heap region, the same way that linear memory is a
sandbox. GC bugs could lead to the guest program accessing the wrong GC object,
or getting garbage data from within the GC heap. But only garbage data from
within the GC heap, never outside it. The worse that could happen would be if we
decided not to zero out GC heaps between reuse across stores (which is a valid
trade off to make, since zeroing a GC heap is a defense-in-depth technique
similar to zeroing a Wasm stack and not semantically visible in the absence of
GC bugs) and then a GC bug would allow the current Wasm guest to read old GC
data from the old Wasm guest that previously used this GC heap. But again, it
could never access host data.

Taken altogether, this allows for collector implementations that are nearly free
from `unsafe` code, and unsafety can otherwise be targeted and limited in scope,
such as interactions with JIT code. Most importantly, we do not have to maintain
critical invariants across the whole system -- invariants which can't be nicely
encapsulated or abstracted -- to preserve memory safety. Such holistic
invariants that refuse encapsulation are otherwise generally a huge safety
problem with GC implementations.

\### `VMGcRef` is *NOT* `Clone` or `Copy` Anymore

`VMGcRef` used to be `Clone` and `Copy`. It is not anymore. The motivation here
was to be sure that I was actually calling GC barriers at all the correct
places. I couldn't be sure before. Now, you can still explicitly copy a raw GC
reference without running GC barriers if you need to and understand why that's
okay (aka you are implementing the collector), but that is something you have to
opt into explicitly by calling `unchecked_copy`. The default now is that you
can't just copy the reference, and instead call an explicit `clone` method (not
*the* `Clone` trait, because we need to pass in the GC heap context to run the
GC barriers) and it is hard to forget to do that accidentally. This resulted in
a pretty big amount of churn, but I am wayyyyyy more confident that the correct
GC barriers are called at the correct times now than I was before.

\### `i31ref`

I started this commit by trying to add `i31ref` support. And it grew into the
whole traits interface because I found that I needed to abstract GC barriers
into helpers anyways to avoid running them for `i31ref`s, so I figured that I
might as well add the whole traits interface. In comparison, `i31ref` support is
much easier and smaller than that other part! But it was also difficult to pull
apart from this commit, sorry about that!

---------------------

Overall, I know this is a very large commit. I am super happy to have some
synchronous meetings to walk through this all, give an overview of the
architecture, answer questions directly, etc... to make review easier!

prtest:full

show more ...

Define garbage collection rooting APIs (#8011)

* Define garbage collection rooting APIs

Rooting prevents GC objects from being collected while they are actively being
used.

We have a few sometimes

Define garbage collection rooting APIs (#8011)

* Define garbage collection rooting APIs

Rooting prevents GC objects from being collected while they are actively being
used.

We have a few sometimes-conflicting goals with our GC rooting APIs:

1. Safety: It should never be possible to get a use-after-free bug because the
user misused the rooting APIs, the collector "mistakenly" determined an
object was unreachable and collected it, and then the user tried to access
the object. This is our highest priority.

2. Moving GC: Our rooting APIs should moving collectors (such as generational
and compacting collectors) where an object might get relocated after a
collection and we need to update the GC root's pointer to the moved
object. This means we either need cooperation and internal mutability from
individual GC roots as well as the ability to enumerate all GC roots on the
native Rust stack, or we need a level of indirection.

3. Performance: Our rooting APIs should generally be as low-overhead as
possible. They definitely shouldn't require synchronization and locking to
create, access, and drop GC roots.

4. Ergonomics: Our rooting APIs should be, if not a pleasure, then at least not
a burden for users. Additionally, the API's types should be `Sync` and `Send`
so that they work well with async Rust.

For example, goals (3) and (4) are in conflict when we think about how to
support (2). Ideally, for ergonomics, a root would automatically unroot itself
when dropped. But in the general case that requires holding a reference to the
store's root set, and that root set needs to be held simultaneously by all GC
roots, and they each need to mutate the set to unroot themselves. That implies
`Rc<RefCell<...>>` or `Arc<Mutex<...>>`! The former makes the store and GC root
types not `Send` and not `Sync`. The latter imposes synchronization and locking
overhead. So we instead make GC roots indirect and require passing in a store
context explicitly to unroot in the general case. This trades worse ergonomics
for better performance and support for moving GC and async Rust.

Okay, with that out of the way, this module provides two flavors of rooting
API. One for the common, scoped lifetime case, and another for the rare case
where we really need a GC root with an arbitrary, non-LIFO/non-scoped lifetime:

1. `RootScope` and `Rooted<T>`: These are used for temporarily rooting GC
objects for the duration of a scope. Upon exiting the scope, they are
automatically unrooted. The internal implementation takes advantage of the
LIFO property inherent in scopes, making creating and dropping `Rooted<T>`s
and `RootScope`s super fast and roughly equivalent to bump allocation.

This type is vaguely similar to V8's [`HandleScope`].

[`HandleScope`]: https://v8.github.io/api/head/classv8_1_1HandleScope.html

Note that `Rooted<T>` can't be statically tied to its context scope via a
lifetime parameter, unfortunately, as that would allow the creation and use
of only one `Rooted<T>` at a time, since the `Rooted<T>` would take a borrow
of the whole context.

This supports the common use case for rooting and provides good ergonomics.

2. `ManuallyRooted<T>`: This is the fully general rooting API used for holding
onto non-LIFO GC roots with arbitrary lifetimes. However, users must manually
unroot them. Failure to manually unroot a `ManuallyRooted<T>` before it is
dropped will result in the GC object (and everything it transitively
references) leaking for the duration of the `Store`'s lifetime.

This type is roughly similar to SpiderMonkey's [`PersistentRooted<T>`],
although they avoid the manual-unrooting with internal mutation and shared
references. (Our constraints mean we can't do those things, as mentioned
explained above.)

[`PersistentRooted<T>`]: http://devdoc.net/web/developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/JSAPI_reference/JS::PersistentRooted.html

At the end of the day, both `Rooted<T>` and `ManuallyRooted<T>` are just tagged
indices into the store's `RootSet`. This indirection allows working with Rust's
borrowing discipline (we use `&mut Store` to represent mutable access to the GC
heap) while still allowing rooted references to be moved around without tying up
the whole store in borrows. Additionally, and crucially, this indirection allows
us to update the *actual* GC pointers in the `RootSet` and support moving GCs
(again, as mentioned above).

* Reorganize GC-related submodules in `wasmtime-runtime`

* Reorganize GC-related submodules in `wasmtime`

* Use `Into<StoreContext[Mut]<'a, T>` for `Externref::data[_mut]` methods

* Run rooting tests under MIRI

* Make `into_abi` take an `AutoAssertNoGc`

* Don't use atomics to update externref ref counts anymore

* Try to make lifetimes/safety more-obviously correct

Remove some transmute methods, assert that `VMExternRef`s are the only valid
`VMGcRef`, etc.

* Update extenref constructor examples

* Make `GcRefImpl::transmute_ref` a non-default trait method

* Make inline fast paths for GC LIFO scopes

* Make `RootSet::unroot_gc_ref` an `unsafe` function

* Move Hash and Eq for Rooted, move to impl methods

* Remove type parameter from `AutoAssertNoGc`

Just wrap a `&mut StoreOpaque` directly.

* Make a bunch of internal `ExternRef` methods that deal with raw `VMGcRef`s take `AutoAssertNoGc` instead of `StoreOpaque`

* Fix compile after rebase

* rustfmt

* revert unrelated egraph changes

* Fix non-gc build

* Mark `AutoAssertNoGc` methods inline

* review feedback

* Temporarily remove externref support from the C API

Until we can add proper GC rooting.

* Remove doxygen reference to temp deleted function

* Remove need to `allow(private_interfaces)`

* Fix call benchmark compilation

show more ...

Wasmtime: Finish support for the typed function references proposal (#7943)

* Wasmtime: Finish support for the typed function references proposal

While we supported the function references proposal

Wasmtime: Finish support for the typed function references proposal (#7943)

* Wasmtime: Finish support for the typed function references proposal

While we supported the function references proposal inside Wasm, we didn't
support it on the "outside" in the Wasmtime embedder APIs. So much of the work
here is exposing typed function references, and their type system updates, in
the embedder API. These changes include:

* `ValType::FuncRef` and `ValType::ExternRef` are gone, replaced with the
introduction of the `RefType` and `HeapType` types and a
`ValType::Ref(RefType)` variant.

* `ValType` and `FuncType` no longer implement `Eq` and `PartialEq`. Instead
there are `ValType::matches` and `FuncType::matches` methods which check
directional subtyping. I also added `ValType::eq` and `FuncType::eq` static
methods for the rare case where someone needs to check precise equality, but
that is almost never actually the case, 99.99% of the time you want to check
subtyping.

* There are also public `Val::matches_ty` predicates for checking if a value is
an instance of a type, as well as internal helpers like
`Val::ensure_matches_ty` that return a formatted error if the value does not
match the given type. These helpers are used throughout Wasmtime internals
now.

* There is now a dedicated `wasmtime::Ref` type that represents reference
values. Table operations have been updated to take and return `Ref`s rather
than `Val`s.

Furthermore, this commit also includes type registry changes to correctly manage
lifetimes of types that reference other types. This wasn't previously an issue
because the only thing that could reference types that reference other types was
a Wasm module that added all the types that could reference each other at the
same time and removed them all at the same time. But now that the previously
discussed work to expose these things in the embedder API is done, type lifetime
management in the registry becomes a little trickier because the embedder might
grab a reference to a type that references another type, and then unload the
Wasm module that originally defined that type, but then the user should still be
able use that type and the other types it transtively references. Before, we
were refcounting individual registry entries. Now, we still are refcounting
individual entries, but now we are also accounting for type-to-type references
and adding a new type to the registry will increment the refcounts of each of
the types that it references, and removing a type from the registry will
decrement the refcounts of each of the types it references, and then recursively
(logically, not literally) remove any types whose refcount has now reached zero.

Additionally, this PR adds support for subtyping to `Func::typed`- and
`Func::wrap`-style APIs. For result types, you can always use a supertype of the
WebAssembly function's actual declared return type in `Func::typed`. And for
param types, you can always use a subtype of the Wasm function's actual declared
param type. Doing these things essentially erases information but is always
correct. But additionally, for functions which take a reference to a concrete
type as a parameter, you can also use the concrete type's supertype. Consider a
WebAssembly function that takes a reference to a function with a concrete type:
`(ref null <func type index>)`. In this scenario, there is no static
`wasmtime::Foo` Rust type that corresponds to that particular Wasm-defined
concrete reference type because Wasm modules are loaded dynamically at
runtime. You *could* do `f.typed::<Option<NoFunc>, ()>()`, and while that is
correctly typed and valid, it is often overly restrictive. The only value you
could call the resulting typed function with is the null function reference, but
we'd like to call it with non-null function references that happen to be of the
correct type. Therefore, `f.typed<Option<Func>, ()>()` is also allowed in this
case, even though `Option<Func>` represents `(ref null func)` which is the
supertype, not subtype, of `(ref null <func type index>)`. This does imply some
minimal dynamic type checks in this case, but it is supported for better
ergonomics, to enable passing non-null references into the function.

We can investigate whether it is possible to use generic type parameters and
combinators to define Rust types that precisely match concrete reference types
in future, follow-up pull requests. But for now, we've made things usable, at
least.

Finally, this also takes the first baby step towards adding support for the Wasm
GC proposal. Right now the only thing that is supported is `nofunc` references,
and this was mainly to make testing function reference subtyping easier. But
that does mean that supporting `nofunc` references entailed also adding a
`wasmtime::NoFunc` type as well as the `Config::wasm_gc(enabled)` knob. So we
officially have an in-progress implementation of Wasm GC in Wasmtime after this
PR lands!

Fixes https://github.com/bytecodealliance/wasmtime/issues/6455

* Fix WAT in test to be valid

* Check that dependent features are enabled for function-references and GC

* Remove unnecessary engine parameters from a few methods

Ever since `FuncType`'s internal `RegisteredType` holds onto its own `Engine`,
we don't need these anymore.

Still useful to keep the `Engine` parameter around for the `ensure_matches`
methods because that can be used to check correct store/engine usage for
embedders.

* Add missing dependent feature enabling for some tests

* Remove copy-paste bit from test

* match self to show it is uninhabited

* Add a missing `is_v128` method

* Short circuit a few func type comparisons

* Turn comment into part of doc comment

* Add test for `Global::new` and subtyping

* Add tests for embedder API, tables, and subtyping

* Add an embedder API test for setting globals and subtyping

* Construct realloc's type from its index, rather than from scratch

* Help LLVM better optimize our dynamic type checks in `TypedFunc::call_raw`

* Fix call benchmark compilation

* Change `WasmParams::into_abi` to take the whole func type instead of iter of params

* Fix doc links

prtest:full

* Fix size assertion on s390x

show more ...

Add support for `v128` to the typed function API (#7010)

* Add support for `v128` to the typed function API

This commit adds a Rust type `V128` which corresponds to the wasm
`v128` type. This is in

Add support for `v128` to the typed function API (#7010)

* Add support for `v128` to the typed function API

This commit adds a Rust type `V128` which corresponds to the wasm
`v128` type. This is intended to perhaps one day have accessors for
lanes of various sizes but in the meantime only supports conversion back
and forth between `u128`. The intention of this type is to allow
platforms to perform typed between to functions that take or return
`v128` wasm values.

Previously this was not implemented because it's a bit tricky ABI-wise.
Typed functions work by passing arguments in registers which requires
the calling convention to match in both Cranelift and in Rust. This
should be the case for supported platforms and the default calling
convention, especially now that the wasm calling convention is separate
from the platform calling convention. This does mean, however, that this
feature can only be supported on x86_64 and AArch64. Currently neither
s390x nor RISC-V have a means of supporting the vector calling
convention since the vector types aren't available on stable in Rust
itself. This means that it's now unfortunately possible to write a
Wasmtime embedding that compiles on x86_64 that doesn't compile on s390x
for example, but given how niche this feature is that seems like an ok
tradeoff for now and by the time it's a problem Rust might have native
stable support for vector types on these platforms.

prtest:full

* Fix compile of C API

* Conditionally enable typed v128 tests

* Review comments

* Fix compiler warnings

show more ...

Make Wasmtime compatible with Stacked Borrows in MIRI (#6338)

* Make Wasmtime compatible with Stacked Borrows in MIRI

The fact that Wasmtime executes correctly under Tree Borrows but not
Stacked B

Make Wasmtime compatible with Stacked Borrows in MIRI (#6338)

* Make Wasmtime compatible with Stacked Borrows in MIRI

The fact that Wasmtime executes correctly under Tree Borrows but not
Stacked Borrows is a bit suspect and given what I've since learned about
the aliasing models I wanted to give it a stab to get things working
with Stacked Borrows. It turns out that this wasn't all that difficult,
but required two underlying changes:

* First the implementation of `Instance::vmctx` is now specially crafted
in an intentional way to preserve the provenance of the returned
pointer. This way all `&Instance` pointers will return a `VMContext`
pointer with the same provenance and acquiring the pointer won't
accidentally invalidate all prior pointers.

* Second the conversion from `VMContext` to `Instance` has been updated
to work with provenance and such. Previously the conversion looked
like `&mut VMContext -> &mut Instance`, but I think this didn't play
well with MIRI because `&mut VMContext` has no provenance over any
data since it's zero-sized. Instead now the conversion is from `*mut
VMContext` to `&mut Instance` where we know that `*mut VMContext` has
provenance over the entire instance allocation. This shuffled a fair
bit around to handle the new closure-based API to prevent escaping
pointers, but otherwise no major change other than the structure and
the types in play.

This commit additionally picks up a dependency on the `sptr` crate which
is a crate for prototyping strict-provenance APIs in Rust. This is I
believe intended to be upstreamed into Rust one day (it's in the
standard library as a Nightly-only API right now) but in the meantime
this is a stable alternative.

* Clean up manual `unsafe impl Send` impls

This commit adds a new wrapper type `SendSyncPtr<T>` which automatically
impls the `Send` and `Sync` traits based on the `T` type contained.
Otherwise it works similarly to `NonNull<T>`. This helps clean up a
number of manual annotations of `unsafe impl {Send,Sync} for ...`
throughout the runtime.

* Remove pointer-to-integer casts with tables

In an effort to enable MIRI's "strict provenance" mode this commit
removes the integer-to-pointer casts in the runtime `Table`
implementation for Wasmtime. Most of the bits were already there to
track all this, so this commit plumbed around the various pointer types
and with the help of the `sptr` crate preserves the provenance of all
related pointers.

* Remove integer-to-pointer casts in CoW management

The `MemoryImageSlot` type stored a `base: usize` field mostly because I
was too lazy to have a `Send`/`Sync` type as a pointer, so this commit
updates it to use `SendSyncPtr<u8>` and then plumbs the pointer-ness
throughout the implementation. This removes all integer-to-pointer casts
and has pointers stores as actual pointers when they're at rest.

* Remove pointer-to-integer casts in "raw" representations

This commit changes the "raw" representation of `Func` and `ExternRef`
to a `*mut c_void` instead of the previous `usize`. This is done to
satisfy MIRI's requirements with strict provenance, properly marking the
intermediate value as a pointer rather than round-tripping through
integers.

* Minor remaining cleanups

* Switch to Stacked Borrows for MIRI on CI

Additionally enable the strict-provenance features to force warnings
emitted today to become errors.

* Fix a typo

* Replace a negative offset with `sub`

* Comment the sentinel value

* Use NonNull::dangling

show more ...

Add `*_unchecked` variants of `Func` APIs for the C API (#3350)

* Add `*_unchecked` variants of `Func` APIs for the C API

This commit is what is hopefully going to be my last installment within

Add `*_unchecked` variants of `Func` APIs for the C API (#3350)

* Add `*_unchecked` variants of `Func` APIs for the C API

This commit is what is hopefully going to be my last installment within
the saga of optimizing function calls in/out of WebAssembly modules in
the C API. This is yet another alternative approach to #3345 (sorry) but
also contains everything necessary to make the C API fast. As in #3345
the general idea is just moving checks out of the call path in the same
style of `TypedFunc`.

This new strategy takes inspiration from previously learned attempts
effectively "just" exposes how we previously passed `*mut u128` through
trampolines for arguments/results. This storage format is formalized
through a new `ValRaw` union that is exposed from the `wasmtime` crate.
By doing this it made it relatively easy to expose two new APIs:

* `Func::new_unchecked`
* `Func::call_unchecked`

These are the same as their checked equivalents except that they're
`unsafe` and they work with `*mut ValRaw` rather than safe slices of
`Val`. Working with these eschews type checks and such and requires
callers/embedders to do the right thing.

These two new functions are then exposed via the C API with new
functions, enabling C to have a fast-path of calling/defining functions.
This fast path is akin to `Func::wrap` in Rust, although that API can't
be built in C due to C not having generics in the same way that Rust
has.

For some benchmarks, the benchmarks here are:

* `nop` - Call a wasm function from the host that does nothing and
returns nothing.
* `i64` - Call a wasm function from the host, the wasm function calls a
host function, and the host function returns an `i64` all the way out to
the original caller.
* `many` - Call a wasm function from the host, the wasm calls
host function with 5 `i32` parameters, and then an `i64` result is
returned back to the original host
* `i64` host - just the overhead of the wasm calling the host, so the
wasm calls the host function in a loop.
* `many` host - same as `i64` host, but calling the `many` host function.

All numbers in this table are in nanoseconds, and this is just one
measurement as well so there's bound to be some variation in the precise
numbers here.

| Name | Rust | C (before) | C (after) |
|-----------|------|------------|-----------|
| nop | 19 | 112 | 25 |
| i64 | 22 | 207 | 32 |
| many | 27 | 189 | 34 |
| i64 host | 2 | 38 | 5 |
| many host | 7 | 75 | 8 |

The main conclusion here is that the C API is significantly faster than
before when using the `*_unchecked` variants of APIs. The Rust
implementation is still the ceiling (or floor I guess?) for performance
The main reason that C is slower than Rust is that a little bit more has
to travel through memory where on the Rust side of things we can
monomorphize and inline a bit more to get rid of that. Overall though
the costs are way way down from where they were originally and I don't
plan on doing a whole lot more myself at this time. There's various
things we theoretically could do I've considered but implementation-wise
I think they'll be much more weighty.

* Tweak `wasmtime_externref_t` API comments

show more ...

Implement RFC 11: Redesigning Wasmtime's APIs (#2897)

Implement Wasmtime's new API as designed by RFC 11. This is quite a large commit which has had lots of discussion externally, so for more inform

Implement RFC 11: Redesigning Wasmtime's APIs (#2897)

Implement Wasmtime's new API as designed by RFC 11. This is quite a large commit which has had lots of discussion externally, so for more information it's best to read the RFC thread and the PR thread.

show more ...

Fix `wasm_val_copy` for null funcref/externref (#2041)

This commit fixes `Clone for wasm_val_t` to avoid attempting to chase a
null pointer. It also fixes the implementation for `FuncRef` values by

Fix `wasm_val_copy` for null funcref/externref (#2041)

This commit fixes `Clone for wasm_val_t` to avoid attempting to chase a
null pointer. It also fixes the implementation for `FuncRef` values by
cloning their internal `wasm_ref_t` as well.

show more ...

Fix leaking funcrefs in the C API (#2040)

This commit adds a case to the destructor of `wasm_val_t` to be sure to
deallocate the `Box<wasm_ref_t>`.