Deduplicate const-eval fast paths in Wasmtime (#12406)

Closes #12243

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Do not re-export `anyhow!` in the `wasmtime::prelude` (#12298)

We are trying to move to `format_err!` instead.

Debug: create private code memories per store when debugging is enabled. (#12051)

* Debug: create private code memories per store when debugging is enabled.

This will allow patching code to implem

Debug: create private code memories per store when debugging is enabled. (#12051)

* Debug: create private code memories per store when debugging is enabled.

This will allow patching code to implement e.g. breakpoints. (That is,
for now the copies are redundant, but soon they will not be.)

This change follows the discussion [here] and offline to define a few
types that better encapsulate the distinction we want to enforce.
Basically, there is almost never a bare `CodeMemory`; they are always
wrapped in an `EngineCode` or `StoreCode`, the latter being a per-store
instance of the former. Accessors are moved to the relevant place so
that, for example, one cannot get a pointer to a Wasm function's body
without being in the context of a `Store` where the containing module
has been registered. The registry then returns a `ModuleWithCode` that
boxes up a `Module` reference and `StoreCode` together for cases where
we need both the metadata from the module and the raw code to derive
something.

The only case where we return raw code pointers to the `EngineCode`
directly have to do with Wasm-to-array trampolines: in some cases, e.g.
`InstancePre` pre-creating data structures with references to host
functions, it breaks our expected performance characteristics to make
the function pointers store-specific. This is fine as long as the
Wasm-to-array trampolines never bake in direct calls to Wasm functions;
the strong invariant is that Wasm functions never execute from
`EngineCode` directly. Some parts of the component runtime would also
have to be substantially refactored if we wanted to do away with this
exception.

The per-`Store` module registry is substantially refactored in this PR.
I got rid of the modules-without-code distinction (the case where a
module only has trampolines and no defined functions still works fine),
and organized the BTreeMaps to key on start address rather than end
address, which I find a little more intuitive (one then queries with the
dual to the range -- 0-up-to-PC and last entry found).

[here]: https://github.com/bytecodealliance/wasmtime/pull/12051#pullrequestreview-3493711812

* Review feedback: do not assume a reasonable code alignment; error when it cannot be known

* Review feedback: fail properly in profiler when we are cloning code

* Fix guest-profiler C API.

* Review feedback: make private-code representation impossible in non-debugging-support builds.

* Add TODO comment referencing issue for cloning only .text.

* clang-format

* Review feedback: add back Component::image_range.

* Review feedback: error on registering profiling metadata when debug is enabled.

* rustfmt

* Remove early bail on profiling-data registration when debugging is enabled: this always happens so we cannot error out.

show more ...

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "widened" into encompassing the limiter on
the `T` in `StoreInner<T>`, for example, by using the self-pointer
located in an instance or the store. This fix is done by threading
`&mut StoreOpaque` as a parameter separately from a
`StoreResourceLimiter`. This means that various callers now take a new
`Option<&mut StoreResourceLimiter<'_>>` parameter in various locations.

Closes #11409

* Fix gc-less build

show more ...

Make const-expr evaluation `async` (#11468)

* Make const-expr evaluation `async`

This commit is extracted from #11430 to accurately reflect how
const-expr evaluation is an async operation due to GC

Make const-expr evaluation `async` (#11468)

* Make const-expr evaluation `async`

This commit is extracted from #11430 to accurately reflect how
const-expr evaluation is an async operation due to GC pauses that may
happen. The changes in this commit are:

* Const-expr evaluation is, at its core, now an `async` function.
* To leverage this new `async`-ness all internal operations are switched
from `*_maybe_async` to `*_async` meaning all the `*_maybe_async`
methods can be removed.
* Some libcalls using `*_maybe_async` are switch to using `*_async` plus
the `block_on!` utility to help jettison more `*_maybe_async` methods.
* Instance initialization is now an `async` function. This is
temporarily handled with `block_on` during instance initialization to
avoid propagating the `async`-ness further upwards. This `block_on`
will get deleted in future refactorings.
* Const-expr evaluation has been refactored slightly to enable having a
fast path in global initialization which skips an `await` point
entirely, achieving performance-parity in benchmarks prior to this commit.

This ended up fixing a niche issue with GC where if a wasm execution was
suspended during `table.init`, for example, during a const-expr
evaluation triggering a GC then if the wasm execution was cancelled it
would panic the host. This panic was because the GC operation returned
`Result` but it was `unwrap`'d as part of the const-expr evaluation
which can fail not only to invalid-ness but also due to "computation is
cancelled" traps.

* Fix configured build

* Undo rebase mistake

show more ...

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation for example to use `Val` instead of `ValRaw`. Previously
the usage of `ValRaw` meant that wasm was disallowed from performing a
GC during const evaluation, but currently constant expressions can
indeed perform a GC. The goal of this commit is to lift this limitation.
This is expected to be a minor slowdown for modules that hit this path,
but most modules shouldn't hit this in a hot loop since LLVM doesn't
generate modules that use this branch of const eval.

The usage of `Val` brings a number of benefits and refactorings
associated with it:

* Const-evaluation is generally safer than before since everything is
higher-level.
* GC types in const-eval were almost already using `Val` meaning that
there's actually fewer conversions now.
* Instantiation code was refactored to use `wasmtime::*`-API types
instead of low-level VM types. This deduplicates a good deal and lifts
complexity out of the raw VM bits.

Another issue that this commit fixes is to change how table
initialization is modeled internally in
`vm::Instance::table_init_segment`. Previously this was done by removing
the tables from an instance to get a split borrow into the store and the
table. This is not valid though because if, during initialization, a GC
is performed then the table is not present to find roots through the
table. This function is refactored to scope borrows to within a loop
instead of over a loop via various refactorings and such and usage of
higher level APIs. This is again, like above, expected to pessimize
performance but this is also not known to be a hot path for modules at
this time.

* Remove the `TableElement` type

This commit is a refactoring of how tables work within Wasmtime to avoid
funneling table elements through a `TableElement` enum internally.
Instead methods are "exploded" to `grow_{gc_ref,func,cont}` which means,
for example, funcrefs don't need a GcStore. The main motivation for this
change was to avoid the idiom where `TableElement` represents a cloned,
but unrooted, GC reference.

Prior to this commit there were a number of subtle bugs in the table
code for Wasmtime where write barriers were forgotten on `table.init`,
`table.set` (via the embedder API), and `table.grow`. While `table.fill`
correctly handled the GC references it was awkward to get everything
else working consistently so I opted to remove `TableElement` entirely
to make it more clear that `&VMGcRef` is ubiquitously used meaning that
the write barriers, for example, are the same as other parts of the
Wasmtime codebase.

This has a few extra tests for "make sure this doesn't leak" to ensure
that GC works correctly with new barriers in place.

* Fix some lints and warnings

* Fix wmemcheck build

* Review comments

* Optimize const eval and global initialization

* Fix compile

* Fix lints

show more ...

Remove `cranelift_entity::{Signed, Unsigned}` (#11400)

Use `*::cast_{un,}signed` in the Rust standard library stabilized in
1.87.

Make `Val::to_raw` a safe function (#11319)

This commit updates Wasmtime's core `Val::to_raw` function a safe
function. This was previously marked as `unsafe` with documentation that
the raw pointer

Make `Val::to_raw` a safe function (#11319)

This commit updates Wasmtime's core `Val::to_raw` function a safe
function. This was previously marked as `unsafe` with documentation that
the raw pointer could be invalid, but that's not a reason for the
function itself to be `unsafe`. Usage of the returned value is still
`unsafe`, but simply acquiring the value is not itself an unsafe
operation.

This additionally marks a number of GC-related `from_raw` functions as
safe. Wasmtime's GC is safe in the face of heap corruption, so it's
memory safe to pass in any 32-bit value. Documentation still indicates
that panics are possible, however.

show more ...

Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm` (#11312)

* Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm`

Slowly expanding this lint to more of the crate.

prtest:full

* Fix lin

Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm` (#11312)

* Deny `unsafe_op_in_unsafe_fn` in `wasmtime::runtime::vm`

Slowly expanding this lint to more of the crate.

prtest:full

* Fix lints in custom module

* Fix some lints with miri

* Fix non-VM build

* Fix arm windows

show more ...

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as wel

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as well.

* Update dependency directive in `Cargo.toml`

show more ...

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previou

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previously forcibly allowed it. The reason
this was allowed was that when the workspace was first migrated the
Wasmtime crate had too many instances that I was willing to fix. I've
now come back around and tried to fix everything.

In short: ideally delete `#[allow]`, otherwise use `#[expect]`,
otherwise use `#[allow]`.

prtest:full

* Adjust some directives

* Fix some warnings

* Fix stack switching size tests on unix

* Don't have a conditional `Drop` impl

* Force `testing_freelist` method to be used

Too lazy to write `#[cfg]`, but not too lazy to write a test.

show more ...

Crack down on mutability and ownership of `vm::Instance` (#10943)

* Crack down on mutability and ownership of `vm::Instance`

This commit represents more effort to bring safety to `vm::Instance`
and

Crack down on mutability and ownership of `vm::Instance` (#10943)

* Crack down on mutability and ownership of `vm::Instance`

This commit represents more effort to bring safety to `vm::Instance`
and, eventually, `ComponentInstance`. This is specifically addressing
two points of safety around `vm::Instance`:

* Previously ownership of this was murky where `InstanceHandle` sort of
represented ownership but sort of didn't either through the
`InstanceHandle::clone` method. Now `InstanceHandle` has a destructor
for instances and no longer has `clone`, so there's one exclusive
owner of an instance.

* Previously `&mut Instance` was liberally passed around, but this is
not sound because certain fields cannot be mutated (e.g. runtime
offset information). While not a perfect solution this PR switches to
using `Pin<&mut Instance>` everywhere instead. This prevents safe
access to `&mut Instance` and we hand-write accessors to individual
fields. Notably we omit mutable access to the `runtime_info` field.

This naturally involved a lot of refactoring internally, but notably
this started bringing up preexisting issues around how there are
locations in the codebase that simultaneously have `&mut Instance` and
`&mut StoreOpaque` which is technically not sound due to being able to
get back to the instance from the store. Some issues here were address
by passing around indices more often such as in instance initialization
and const-expr evaluation.

Note that all proxy methods on `InstanceHandle` are also all removed now
and there's now only two: `get` and `get_mut`. This reflects how
`InstanceHandle` should in general no longer be used and instead
`Instance` itself, and some pointer-to thereof, should be exclusively
used.

cc #10933

* Fix stack-switching-less build

* Fix wmemcheck build

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Update spec test suite (#10611)

Now that wasm-tools is updated it's possible to run the latest version
again. This notably required implementing the `extern.convert_any` and
`any.convert_extern` ins

Update spec test suite (#10611)

Now that wasm-tools is updated it's possible to run the latest version
again. This notably required implementing the `extern.convert_any` and
`any.convert_extern` instructions in a const-expression context.

show more ...

Reuse Wasm linear memories code for GC heaps (#10503)

* Reuse code for Wasm linear memories for GC heaps

Instead of bespoke code paths and structures for Wasm GC, this commit makes it
so that we no

Reuse Wasm linear memories code for GC heaps (#10503)

* Reuse code for Wasm linear memories for GC heaps

Instead of bespoke code paths and structures for Wasm GC, this commit makes it
so that we now reuse VM structures like `VMMemoryDefinition` and bounds-checking
logic. Notably, we also reuse all the associated bounds-checking optimizations
and, when possible, virtual-memory techniques to completely elide them.

Furthermore, this commit adds support for growing GC heaps, reusing the
machinery for growing memories, and makes it so that GC heaps always start out
empty. This allows us to properly delay allocating the GC heap's storage until a
GC object is actually allocated.

Fixes #9350

* fix c api compilation

* use assert_contains

* remove no-longer-necessary extra memory config from limiter tests

* Helper for retry-after-maybe-async-gc in libcalls

* Clean up some comments

* fix wasmtime-fuzzing and no-gc compilation

* fix examples

* fix no-gc+compiler build

* fix build without pooling allocator

* fix +cranelift +gc-drc -gc-null builds

* fix table hash key stability test

* fix oracle usage of `ExternRef::new`

* fix +gc -gc-null -gc-drc build

* fix wasmtime-fuzzing

* make `StorePtr` wrap a `NonNull`

* Fix some doc tests

* Remove some unnecessary retry helpers now that `FooRef::new` will auto-gc

* fix things after rebase

* Reorganize collection/growth methods for GC heap

* rename BoundsCheck variants

* fix cfg'ing of gc only code

* Fix doc tests

* fix one more gc cfg

* disable GC heap OOM test on non-64-bit targets

show more ...

Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions (#10560)

* Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions

Rather than forcing all callers to

Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions (#10560)

* Automatically trigger GC in `{Array,Extern,Struct}Ref` allocation functions

Rather than forcing all callers to check for `GcHeapOutOfMemory`, trigger a GC,
and then try again. This does force us to define `*_async` variations for when
async is enabled, however; it's ultimately worth it.

* cargo fmt

* review feedback and fix tests

* fix +runtime -gc build

* more feedback and build cfg fixes

* remove copy-paste assertion that doesn't apply to this method

* move assertion into retry methods

show more ...

`AnyRef::from_raw` needs to clone its GC ref (#10374)

Like `ExternRef::from_raw` does.

This is because while `to_raw` clones the GC ref, it also exposes it to Wasm,
inserting it into the DRC collec

`AnyRef::from_raw` needs to clone its GC ref (#10374)

Like `ExternRef::from_raw` does.

This is because while `to_raw` clones the GC ref, it also exposes it to Wasm,
inserting it into the DRC collector's table, which effectively gives ownership
of that clone to Wasm. Then, if we don't clone in `from_raw`, when a GC is
triggered, the DRC collector will see that Wasm isn't holding the ref alive
anymore and will decref (and perhaps even deallocate) it, which leaves the
`AnyRef` we constructed via `from_raw` dangling.

Fixes #10182

show more ...

Wasm GC: Fix an incorrect assertion and canonicalize types for runtime usage in ExternType::from_wasmtime (#10223)

* Fix assertion in `PartialEq` for `RegisteredType` again

It is possible for two `

Wasm GC: Fix an incorrect assertion and canonicalize types for runtime usage in ExternType::from_wasmtime (#10223)

* Fix assertion in `PartialEq` for `RegisteredType` again

It is possible for two `WasmSubType`s to be equal to each other, as far as
`derive(PartialEq)` is concerned, but still different from each other if they
are in different rec groups or even if they are at different indices within the
same rec group. The assertion mistakenly did not permit either of these,
however.

Fixes #9714

* Canonicalize all types for runtime usage when creating `wasmtime::{Module,Component}`s

Rather than canonicalizing them on demand in functions like
`{Func,Global,Table}Type::from_wasmtime` and other places. Instead, we do it in
one place, up front, so that it is very unlikely we miss anything. Doing this
involves changing some things from `ModuleInternedTypeIndex`es to
`EngineOrModuleTypeIndex`es in `wasmtime_environ`, which means that a bunch of
uses of those things need to unwrap the appropriate kind of type index at usage
sites (e.g. compilation uses will unwrap `ModuleInternedTypeIndex`es, runtime
uses usage will unwrap `VMSharedTypeIndex`es). And it additionally required
implementing the `TypeTrace` trait for a handful of things to unlock the
provided `canonicalize_for_runtime_usage` trait method for those things.

All this machinery is required to avoid an assertion failure in the regression
test introduced in the previous commit, which was triggered because we were
failing to canonicalize type indices inside `ExternType`s for runtime usage on
some code paths. We shouldn't have to play that kind of whack-a-mole in the
future, thanks to this new approach.

* Fix a warning in no-default-features builds

* Fix another warning in weird cfg builds

show more ...

Enable warnings if `gc` is disabled (#10149)

* Enable warnings if `gc` is disabled

Continuation of work in #10131. This additionally handles turning off
`gc-null` and `gc-drc` and the various combi

Enable warnings if `gc` is disabled (#10149)

* Enable warnings if `gc` is disabled

Continuation of work in #10131. This additionally handles turning off
`gc-null` and `gc-drc` and the various combinations within.

* Fix some more warnings

* Fix a feature combo build

show more ...

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today th

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today this is not possible because
Pulley does not properly respect either strict or permissive provenance
models. The goal of this refactoring is to enable fixing this in a
future commit that doesn't touch everything in the codebase. Instead
everything is touched here in this commit.

The basic problem with Pulley is that it is incompatible with the strict
provenance model of Rust which means that we'll be using "exposed
provenance" APIs to satisfy Rust's soundness requirements. In this model
we must explicitly call `ptr.expose_provenance()` on any pointers
which are exposed to compiled code. Arguably we should also be already
doing this for natively-compiled code but I am not certain about how
strictly this is required.

Currently in Wasmtime today we call `ptr.expose_provenance()` nowhere.
It also turns out, though, that we share quite a few pointers in quite a
few places with compiled code. This creates a bit of a problem! The
solution settled on in this commit looks like:

* A new marker trait, `VmSafe`, is introduced. This trait is used to
represent "safe to share with compiled code" types and enumerates some
properties such as defined ABIs, primitives wrappers match primitive
ABIs, and notably "does not contain a pointer".

* A new type, `VmPtr<T>`, is added to represent pointers shared with
compiled code. Internally for now this is just `SendSyncPtr<T>` but in
the future it will be `usize`. By using `SendSyncPtr<T>` it shouldn't
actually really change anything today other than requiring a lot of
refactoring to get the types to line up.

* The core `vmctx_plus_offset*` methods are updated to require
`T: VmSafe`. Previously they allowed any `T` which is relatively
dangerous to store any possible Rust type in Cranelift-accessible
areas.

These three fundamental changes were introduced in this commit. All
further changes were refactoring necessary to get everything working
after these changes. For example many types in `vmcontext.rs`, such as
`VMFuncRef`, have changed to using `VmPtr<T>` instead of `NonNull<T>` or
`*mut T`. This is a pretty expansive change which resulted in touching a
lot of places.

One premise of `VmPtr<T>` is that it's non-null. This was an
additional refactoring that updated a lot of places where previously
`*mut T` was used and now either `VmPtr<T>` or `NonNull<T>` is used.

In the end the intention is that `VmPtr<T>` is used whenever pointers
are store in memory that can be accessed from Cranelift. When operating
inside of the runtime `NonNull<T>` or `SendSyncPtr<T>` is preferred
instead.

As a final note, no provenance changes have actually happened yet. For
example this doesn't fix Pulley in MIRI. What it does enable, though, is
that the future commit to fix Pulley in MIRI will be much smaller with
this having already landed.

* Run the full test suite in PR CI

prtest:full

* Minor CI issues

* Fix no_std build

* Fix miri build

* Don't use `VmPtr` in FFI function signatures

Use `NonNull` or `*mut u8` as appropriate for function signatures
instead. It shouldn't be required to use `VmPtr` during the handoff to
compiled code as we've already annotated the pointer going out.

* Fix rebase conflict

* Review comments

show more ...

Use `NonNull` for `VMFuncRef` in more places (#9677)

* Use `NonNull` for `VMFuncRef` in more places

This commit switches to using `NonNull<VMFuncRef>` in more places
instead of `*mut VMFuncRef`. A

Use `NonNull` for `VMFuncRef` in more places (#9677)

* Use `NonNull` for `VMFuncRef` in more places

This commit switches to using `NonNull<VMFuncRef>` in more places
instead of `*mut VMFuncRef`. A few minor locations benefitted from this
by removing some otherwise extraneous checks but most places have been
updated to mostly do the same as before. The goal of this commit is to
make it more clear about what returns a nullable reference and what
never returns a nullable reference. Additionally some constructors now
unconditionally work with `NonNull<T>` instead of checking for null
internally.

* Fix a refactoring typo

show more ...

Refactor `vm::Instance` for better `Store` safety (#9565)

* Refactor `vm::Instance` for better `Store` safety

At various times, for example in libcalls, we need to go from a raw `vmctx`
pointer tha

Refactor `vm::Instance` for better `Store` safety (#9565)

* Refactor `vm::Instance` for better `Store` safety

At various times, for example in libcalls, we need to go from a raw `vmctx`
pointer that Wasm gave us to both an `&mut Instance` and an `&mut
Store{Opaque,Context}`. This is a pretty fundamental unsafe aspect of
implementing a language runtimne in Rust, but we can tighten things up a bit and
make them a little safer than they currently are. This commit is such an
attempt and is notably tackling the issue of creating multiple store borrows
after we have an instance borrow.

This commit makes the following changes:

* `Instance` doesn't expose a method to get the raw `*mut dyn VMStore` pointer
or otherwise create a store borrow.

* You cannot construct an `Instance` directly from a `vmctx` pointer.

* There is now an `InstanceAndStore` type that represents unique access to both
an `Instance` and a `Store`.

* You can (with `unsafe`) create an `InstanceAndStore` from a raw `vmctx`
pointer. This generally only happens inside a couple "bottlenecks", not all
throughout the codebase, like the shared plumbing code for libcalls.

* The `InstanceAndStore` can be unpacked into a mutable borrow of an `Instance`
and a mutable borrow of a `Store`. This unpacking takes holds a mutable borrow
of the original `InstanceAndStore`, so double borrows of the store are no
longer a concern, so long as you don't use `unsafe` to create new
`InstanceAndStore`s for the same `vmctx`.

* All `Instance` methods and functions that previously would unsafely turn the
`Instance`'s internal store pointer into a store borrow to perform some action
now take a store argument, and this store is threaded around as necessary.

Altogether, I feel that we've ended up with an architecture that is a safety
improvement over where we were previously, and will help us properly avoid Rust
UB.

For what its worth, I do not think this is the end state. I foresee at least two
additional follow ups that I unfortunately do not currently have time for:

1. Create `ComponentInstanceAndStore`, which is the same thing that this commit
did but for components. I started on this but ran out of fuel while trying to
update macros for all the component shims and transcoders.

2. Stop dealing with `&mut vm::Table`s and `&mut vm::Global`s and all that in
the runtime. Instead just deal with indices, similar to how things are
structured in the host API level. This refactoring was not previously possible
due to (1) the `wasmtime` versus `wasmtime-runtime` crate split and (2) the lack
of `StoreOpaque`s threaded through the VM internals. The first blocker was
addressed a few months ago, this commit removes the second blocker. This will
still be a pretty large refactoring though, but I think ultimately will be worth
it.

* Fix wmemcheck build

* Remove `'static` from the `dyn VMStore` accessors; fix no-gc build

show more ...

threads: add `WasmCompositeInnerType` layer (#9520)

* threads: add `WasmCompositeInnerType` layer

As in `wasm-tools`, push the type down a layer so that
`WasmCompositeType` can be marked shared. Th

threads: add `WasmCompositeInnerType` layer (#9520)

* threads: add `WasmCompositeInnerType` layer

As in `wasm-tools`, push the type down a layer so that
`WasmCompositeType` can be marked shared. This leaves several holes to
be filled in by future commits (all marked with `TODO: handle shared`);
in effect, this change allows `WasmCompositeType` to be marked shared
but mostly does not wire up the sharedness during translation.

* review: remove TODOs

* review: remove more TODOs

* review: refactor to use `GcTypeLayouts::gc_layout`

* review: propagate sharedness

* review: fail if an unwrapped type is shared

* review: propagate sharedness more

* review: redefine accessors as unshared

* fix: add `DisabledLayouts` to satisfy `GcRuntime::layouts()`

* review: panic on shared

* review: remove unnecessary assert

show more ...

Implement the `array.copy` Wasm GC instruction (#9389)

* Implement the `array.copy` Wasm GC instruction

This also involved fixing some `VMGcRef` initialization and GC barrier code for
globals. That

Implement the `array.copy` Wasm GC instruction (#9389)

* Implement the `array.copy` Wasm GC instruction

This also involved fixing some `VMGcRef` initialization and GC barrier code for
globals. That in turn involved making global initialization fallible, which
meant that it needed to be pulled out of `vmctx` initialization and put into
instance initialization.

Co-Authored-By: Alex Crichton <[email protected]>

* fix compile error due to code moving where a trait method was no longer in scope

* use the result of `MaybeUninit::write`

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...