Use `TryPrimaryMap` in `Instance` (#12848)

fix: correct various typos (#12807)

Signed-off-by: Ho Kim <[email protected]>

Rename our OOM-handling `EntitySet` to `TryEntitySet` (#12722)

Remove unused `wasmtime::runtime::vm::Instance::exports` method (#12579)

Introduce `wasmtime::Store::try_new`, which handles OOM (#12415)

* Introduce `wasmtime::Store::try_new`, which handles OOM

`Store::new` is an infallible constructor, so there is not a direct way to

Introduce `wasmtime::Store::try_new`, which handles OOM (#12415)

* Introduce `wasmtime::Store::try_new`, which handles OOM

`Store::new` is an infallible constructor, so there is not a direct way to make
it return an error on OOM. Additionally, it is one of the most-used functions in
the Wasmtime embedder API, so changing its signature to return a `Result` is a
non-starter -- it would cause way too much pain. So instead we define
`Store::try_new` which returns a `Result` and make `Store::new` call and unwrap
that new constructor.

Part of https://github.com/bytecodealliance/wasmtime/issues/12069

* update disas tests and fix winch

* Disable concurrency support in `Store::try_new` OOM test

* Add attributes that were lost in rebase

show more ...

Don't pass module into `initialize_vmctx` (#12412)

* Don't pass module into `initialize_vmctx`

Get it from `self`, which means we can't ever pass in the wrong module.

* Avoid cloning an `Arc` via

Don't pass module into `initialize_vmctx` (#12412)

* Don't pass module into `initialize_vmctx`

Get it from `self`, which means we can't ever pass in the wrong module.

* Avoid cloning an `Arc` via pin tricks

show more ...

Remove unnecessary `memory_tys` param from `vm::Instance::new` (#12414)

We already have this information in the `InstanceAllocationRequest`, and by
avoiding a parameter we can never accidentally pas

Remove unnecessary `memory_tys` param from `vm::Instance::new` (#12414)

We already have this information in the `InstanceAllocationRequest`, and by
avoiding a parameter we can never accidentally pass the wrong memory types.

show more ...

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Use `self` offsets to initialize an `Instance`'s vmctx (#12378)

Rather than passing them in, which allows for the possiblity of passing the
wrong offsets in.

add missing may-block checks for sync-to-sync guest-to-guest calls (#12282)

* add missing may-block checks for sync-to-sync guest-to-guest calls

Previously, we weren't updating or checking the may-

add missing may-block checks for sync-to-sync guest-to-guest calls (#12282)

* add missing may-block checks for sync-to-sync guest-to-guest calls

Previously, we weren't updating or checking the may-block status of a task
across sync-to-sync, guest-to-guest calls, meaning we were allowing blocking in
cases we shouldn't have.

This fixes that by adding a new `task_may_block` field to `VMComponentContext`,
plus code to update it every time we switch threads or do a sync-to-sync,
guest-to-guest call. We use that field as the source of truth about whether a
blocking operation is permitted.

I've updated various tests to match, and Luke has an item on his to-do list to
add sad-path coverage for various cases to the upstream `component-model` test
suite.

* address review feedback and fix component_instance_size_limit test

* remove `TaskMayBlock` type per review feedback

* bless disas tests

show more ...

Debug: create private code memories per store when debugging is enabled. (#12051)

* Debug: create private code memories per store when debugging is enabled.

This will allow patching code to implem

Debug: create private code memories per store when debugging is enabled. (#12051)

* Debug: create private code memories per store when debugging is enabled.

This will allow patching code to implement e.g. breakpoints. (That is,
for now the copies are redundant, but soon they will not be.)

This change follows the discussion [here] and offline to define a few
types that better encapsulate the distinction we want to enforce.
Basically, there is almost never a bare `CodeMemory`; they are always
wrapped in an `EngineCode` or `StoreCode`, the latter being a per-store
instance of the former. Accessors are moved to the relevant place so
that, for example, one cannot get a pointer to a Wasm function's body
without being in the context of a `Store` where the containing module
has been registered. The registry then returns a `ModuleWithCode` that
boxes up a `Module` reference and `StoreCode` together for cases where
we need both the metadata from the module and the raw code to derive
something.

The only case where we return raw code pointers to the `EngineCode`
directly have to do with Wasm-to-array trampolines: in some cases, e.g.
`InstancePre` pre-creating data structures with references to host
functions, it breaks our expected performance characteristics to make
the function pointers store-specific. This is fine as long as the
Wasm-to-array trampolines never bake in direct calls to Wasm functions;
the strong invariant is that Wasm functions never execute from
`EngineCode` directly. Some parts of the component runtime would also
have to be substantially refactored if we wanted to do away with this
exception.

The per-`Store` module registry is substantially refactored in this PR.
I got rid of the modules-without-code distinction (the case where a
module only has trampolines and no defined functions still works fine),
and organized the BTreeMaps to key on start address rather than end
address, which I find a little more intuitive (one then queries with the
dual to the range -- 0-up-to-PC and last entry found).

[here]: https://github.com/bytecodealliance/wasmtime/pull/12051#pullrequestreview-3493711812

* Review feedback: do not assume a reasonable code alignment; error when it cannot be known

* Review feedback: fail properly in profiler when we are cloning code

* Fix guest-profiler C API.

* Review feedback: make private-code representation impossible in non-debugging-support builds.

* Add TODO comment referencing issue for cloning only .text.

* clang-format

* Review feedback: add back Component::image_range.

* Review feedback: error on registering profiling metadata when debug is enabled.

* rustfmt

* Remove early bail on profiling-data registration when debugging is enabled: this always happens so we cannot error out.

show more ...

Prevent using shared memories with `Memory` (#12022)

* Prevent using shared memories with `Memory`

This commit fixes a few issues where it was possible to represent a wasm
shared linear memory with

Prevent using shared memories with `Memory` (#12022)

* Prevent using shared memories with `Memory`

This commit fixes a few issues where it was possible to represent a wasm
shared linear memory with the `wasmtime::Memory` type. This is not sound
because `wasmtime::Memory` provides safe Rust access to the bytes where
that is not possible with wasm shared memories. Shared memories in Rust
must be represented by `SharedMemory`, not `wasmtime::Memory`.

Specifically this commit prevents two vectors of this happening:

1. `Memory::new` now requires that the memory type specified is
non-shared. Instead `SharedMemory::new` must be used instead.

2. Core dumps now skip over shared memories when iterating over all
memories in the store. Supporting shared memories is left to a future
feature request for now.

* CI fixes

show more ...

Wasmtime: implement debug instrumentation and basic host API to examine runtime state. (#11769)

* Wasmtime: implement debug instrumentation and basic host API to examine runtime state.

This PR impl

Wasmtime: implement debug instrumentation and basic host API to examine runtime state. (#11769)

* Wasmtime: implement debug instrumentation and basic host API to examine runtime state.

This PR implements ideas from the [recent RFC] to serve as the basis
for Wasm (guest) debugging: it adds a stackslot to each function
translated from Wasm, stores to replicate Wasm VM state in the
stackslot as the program runs, and metadata to describe the format of
that state and allow reading it out at runtime.

As an initial user of this state, this PR adds a basic "stack view"
API that, from host code that has been called from Wasm, can examine
Wasm frames currently on the stack and read out all of their locals
and stack slots.

Note in particular that this PR does not include breakpoints,
watchpoints, stepped execution, or any sort of user interface for any
of this; it is only a foundation.

This PR still has a few unsatisfying bits that I intend to address:

- The "stack view" performs some O(n) work when the view is initially
taken, computing some internal data per frame. This is forced by the
current design of `Backtrace`, which takes a closure and walks that
closure over stack frames eagerly (rather than work as an
iterator). It's got some impressive iterator-chain stuff going on
internally, so refactoring it to the latter approach might not
be *too* bad, but I haven't tackled it yet.

A O(1) stack view, that is, one that does work only for frames as
the host API is used to walk up the stack, is desirable because some
use-cases may want to quickly examine e.g. only the deepest
frame (say, running with a breakpoint condition that needs to read a
particular local's value after each step).

- It includes a new `Config::compiler_force_inlining()` option that is
used only for testing that we get the correct frames after
inlining. I couldn't get the existing flags to work on a Wasmtime
config level and suspect there may be an existing bug there; I will
try to split out a fix for it.

This PR renames the existing `debug` option to `native_debug`, to
distinguish it from the new approach.

[recent RFC]: https://github.com/bytecodealliance/rfcs/pull/44

* Update to new APIs on Cranelift side.

* Test update.

* Adjust objdump printing of InstPos on frame progpoints; and adjust progpoint collapsing.

* Convert to iterator form.

* Fix path in native-debug tests (debug -> native_debug rename).

* Enforce that `debug_instrumentation` can only be enabled when feature is enabled.

* Add missing assert.

* Use builtin knob for forcing intra-module inlining instead.

* Review feedback:

- Make StackView own the current frame rather than handing it out. This
prevents the current frame (`FrameView`) from walking away and hiding
somewhere it shouldn't, to be used unsoundly later.
- Assert no-GC during stack walk.

* Merge debug-instrumentation hooks on FuncEnvironment into before/after hooks.

* Review feedback: avoid downcasting funcs twice.

* Add debug feature to `wasmtime` crate's defaults.

* Review feedback: u32s for local and stack indices in debug host API.

* Use *const u8 as stack pointers and `with_exposed_provenance` in debug API.

* Remove some `srcloc` plumbing in Wasm translator.

* Rename native-debug back to debug, and make the new thing "guest debugging".

* rustfmt in debugging test.

* fix disas test after guest-debug CLI option rename.

* Review feedback: no separate debug-instrumentation hooks on FuncEnvironment.

* Review feedback: update doc comment on `Config::guest_debug`.

* Review feedback: rename `generate_debuginfo` to `debug_native` in tunables.

* Review feedback: miscellaneous comments.

* Review comment: fix wording in safety conditions.

* revert wasi-common submodule update

* Properly root values in debug frame slots.

Fixes #11841.

* Fix non-`debug`-feature build.

* Review feedback: naming.

* Ignore tests that compile modules in miri.

show more ...

Refine raw `VMContext` helpers (#11670)

This redefines `Instance::from_vmctx` as a function from
`NonNull<VMContext>` to `NonNull<Instance>` to canonicalize the single
location that pointer arithmet

Refine raw `VMContext` helpers (#11670)

This redefines `Instance::from_vmctx` as a function from
`NonNull<VMContext>` to `NonNull<Instance>` to canonicalize the single
location that pointer arithmetic is done but otherwise require callers
to handle the safety requirements of safe references and such.

show more ...

Don't unwrap results of const-eval (#11557)

While all const expressions executed at runtime are valid they can still
fail due to GC OOM or failure to allocate more GC memory. As a result
all `unwrap

Don't unwrap results of const-eval (#11557)

While all const expressions executed at runtime are valid they can still
fail due to GC OOM or failure to allocate more GC memory. As a result
all `unwrap()` on `eval(...)` is removed in favor of propagating
upwards the result.

Closes #11469

show more ...

Make core wasm libcalls sound (#11496)

* Make core wasm libcalls sound

This commit updates the signature of core wasm libcalls to look more
like component libcalls where the instance argument is ju

Make core wasm libcalls sound (#11496)

* Make core wasm libcalls sound

This commit updates the signature of core wasm libcalls to look more
like component libcalls where the instance argument is just an id, not
an actual pointer. This is required to make them sound because otherwise
it's possible to, in safe Rust, acquire two mutable pointers to the same
instance. Implementing this change is made possible by the many many
previous refactors to how all of these internals work. All that was
required here was changing type signatures and minor updates to the
order of operations inside of libcalls.

Closes #11178

* Fix wmemcheck build

show more ...

WebAssembly exception-handling support. (#11326)

* WebAssembly exception-handling support.

This PR introduces support for the [Wasm exception-handling proposal],
which introduces a conventional try

WebAssembly exception-handling support. (#11326)

* WebAssembly exception-handling support.

This PR introduces support for the [Wasm exception-handling proposal],
which introduces a conventional try/catch mechanism to WebAssembly. The
PR supports modules that use `try_table` to register handlers for a
lexical scope; and provides `throw` and `throw_ref` that allocate (in
the first case) and throw exception objects.

This PR builds on top of the work in #10510 for Cranelift-level
exception support, #10919 for an unwinder, and #11230 for exception
objects built on top of GC, in addition a bunch of smaller fix and
enabling PRs around those.

[Wasm exception-handling proposal]: https://github.com/WebAssembly/exception-handling/

prtest:full

* Permit UnwindToWasm to have unused fields in Pulley builds (for now).

* Resolve miri-caught reborrowing issue.

* Ignore exceptions tests in miri for now (Pulley not supported).

* Use wasmtime_test on exceptions tests.

* Get tests passing on pulley platforms

* Add a check to `supports_host` for the generated test and assert
failure also when that is false.
* Remove `pulley_unsupported` test as it falls out of `#[wasmtime_test]`
* Remove `exceptions_store` helper as it falls out of `#[wasmtime_test]`
* Remove miri annotations as they fall out of `#[wasmtime_test]`

* Remove dead import

* Skip some unsupported tests entirely in `#[wasmtime_test]`

If the selected compiler doesn't support the host at all then there's no
need to run it. Actually running it could misinterpret `CraneliftNative`
as "run with pulley" otherwise, so avoid such false negatives.

* Cranelift: dynamic contexts: account for outgoing-args area.

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "widened" into encompassing the limiter on
the `T` in `StoreInner<T>`, for example, by using the self-pointer
located in an instance or the store. This fix is done by threading
`&mut StoreOpaque` as a parameter separately from a
`StoreResourceLimiter`. This means that various callers now take a new
`Option<&mut StoreResourceLimiter<'_>>` parameter in various locations.

Closes #11409

* Fix gc-less build

show more ...

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `Store

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `StoreOpaque::allocate_instance` is now an `async fn`.
This function does not actually use the `async`-ness just yet so it's a
noop from that point of view, but this propagates outwards to enough
locations that I wanted to split this off to make future changes more
digestable.

Notably some creation functions here such as making an `Instance`,
`Table`, or `Memory` are refactored internally to use this new `async`
function. Annotations of `assert_ready` or `one_poll` are used as
appropriate as well.

For reference this commit was benchmarked with our `instantiation.rs`
benchmark in the pooling allocator and shows no changes relative to the
original baseline from before-`async`-PRs.

* Make table/memory creation `async` functions

This commit is a large-ish refactor which is made possible by the many
previous refactorings to internals w.r.t. async-in-Wasmtime. The end
goal of this change is that table and memory allocation are both `async`
functions. Achieving this, however, required some refactoring to enable
it to work:

* To work with `Send` neither function can close over `dyn VMStore`.
This required changing their `Option<&mut dyn VMStore>` arugment to
`Option<&mut StoreResourceLimiter<'_>>`
* Somehow a `StoreResourceLimiter` needed to be acquired from an
`InstanceAllocationRequest`. Previously the store was stored here as
an unsafe raw pointer, but I've refactored this now so
`InstanceAllocationRequest` directly stores `&StoreOpaque` and
`Option<&mut StoreResourceLimiter>` meaning it's trivial to acquire
them. This additionally means no more `unsafe` access of the store
during instance allocation (yay!).
* Now-redundant fields of `InstanceAllocationRequest` were removed since
they can be safely inferred from `&StoreOpaque`. For example passing
around `&Tunables` is now all gone.
* Methods upwards from table/memory allocation to the
`InstanceAllocator` trait needed to be made `async`. This includes new
`#[async_trait]` methods for example.
* `StoreOpaque::ensure_gc_store` is now an `async` function. This
internally carries a new `unsafe` block carried over from before with
the raw point passed around in `InstanceAllocationRequest`. A future
PR will delete this `unsafe` block, it's just temporary.

I attempted a few times to split this PR up into separate commits but
everything is relatively intertwined here so this is the smallest
"atomic" unit I could manage to land these changes and refactorings.

* Shuffle `async-trait` dep

* Fix configured build

show more ...

Make const-expr evaluation `async` (#11468)

* Make const-expr evaluation `async`

This commit is extracted from #11430 to accurately reflect how
const-expr evaluation is an async operation due to GC

Make const-expr evaluation `async` (#11468)

* Make const-expr evaluation `async`

This commit is extracted from #11430 to accurately reflect how
const-expr evaluation is an async operation due to GC pauses that may
happen. The changes in this commit are:

* Const-expr evaluation is, at its core, now an `async` function.
* To leverage this new `async`-ness all internal operations are switched
from `*_maybe_async` to `*_async` meaning all the `*_maybe_async`
methods can be removed.
* Some libcalls using `*_maybe_async` are switch to using `*_async` plus
the `block_on!` utility to help jettison more `*_maybe_async` methods.
* Instance initialization is now an `async` function. This is
temporarily handled with `block_on` during instance initialization to
avoid propagating the `async`-ness further upwards. This `block_on`
will get deleted in future refactorings.
* Const-expr evaluation has been refactored slightly to enable having a
fast path in global initialization which skips an `await` point
entirely, achieving performance-parity in benchmarks prior to this commit.

This ended up fixing a niche issue with GC where if a wasm execution was
suspended during `table.init`, for example, during a const-expr
evaluation triggering a GC then if the wasm execution was cancelled it
would panic the host. This panic was because the GC operation returned
`Result` but it was `unwrap`'d as part of the const-expr evaluation
which can fail not only to invalid-ness but also due to "computation is
cancelled" traps.

* Fix configured build

* Undo rebase mistake

show more ...

Make memory growth an async function (#11460)

* Make memory growth an `async` function

This is an analog of #11442 but for memories. This had a little more
impact due to memories being hooked into

Make memory growth an async function (#11460)

* Make memory growth an `async` function

This is an analog of #11442 but for memories. This had a little more
impact due to memories being hooked into GC operations. Further
refactoring of GC operations to make them safer/more-async is deferred
to a future PR and for now it's "no worse than before". This is another
step towards #11430 and enables removing a longstanding `unsafe` block
in `runtime/memory.rs` which previously could not be removed.

One semantic change from this is that growth of a shared memory no
longer uses an async limiter. This is done to keep growth of a shared
memory consistent with creation of a shared memory where no limits are
applied. This is due to the cross-store nature of shared memories which
means that we can't tie growth to any one particular store. This
additionally fixes an issue where an rwlock write guard was otherwise
held across a `.await` point which creates a non-`Send` future, closing
a possible soundness hole in Wasmtime.

* Fix threads-disabled build

* Review comments

show more ...

Require a store in `catch_unwind_and_record_trap` (#11441)

* Require a store in `catch_unwind_and_record_trap`

This commit does some preparatory refactoring for #11326 to ensure that
a store is ava

Require a store in `catch_unwind_and_record_trap` (#11441)

* Require a store in `catch_unwind_and_record_trap`

This commit does some preparatory refactoring for #11326 to ensure that
a store is available when trap information is being processed. Currently
this doesn't leverage the new parameter but it should be leverage-able
in #11326.

* Review comments

show more ...

Make table growth a true `async fn` (#11442)

* Make table growth a true `async fn`

Upon further refactoring and thinking about #11430 I've realized that we
might be able to sidestep `T: Send` on th

Make table growth a true `async fn` (#11442)

* Make table growth a true `async fn`

Upon further refactoring and thinking about #11430 I've realized that we
might be able to sidestep `T: Send` on the store entirely which would be
quite the boon if it can be pulled off. The realization I had is that
the main reason for this was `&mut dyn VMStore` on the stack, but that
itself is actually a bug in Wasmtime (#11178) and shouldn't be done.
The functions which have this on the stack should actually ONLY have the
resource limiter, if configured. This means that while the
`ResourceLimiter{,Async}` traits need a `Send` supertrait that's
relatively easy to add without much impact. My hunch is that plumbing
this through to the end will enable all the benefits of #11430 without
requiring adding `T: Send` to the store.

This commit starts out on this journey by making table growth a true
`async fn`. A new internal type is added to represent a store's limiter
which is plumbed to growth functions. This represents a hierarchy of
borrows that look like:

* `StoreInner<T>`
* `StoreResourceLimiter<'_>`
* `StoreOpaque`
* `Pin<&mut Instance>`
* `&mut vm::Table`

This notably, safely, allows operating on `vm::Table` with a
`StoreResourceLimiter` at the same time. This is exactly what's needed
and prevents needing to have `&mut dyn VMStore`, the previous argument,
on the stack.

This refactoring cleans up `unsafe` blocks in table growth right
now which manually uses raw pointers to work around the borrow checker.
No more now!

I'll note as well that this is just an incremental step. What I plan on
doing next is handling other locations like memory growth, memory
allocation, and table allocation. Each of those will require further
refactorings to ensure that things like GC are correctly accounted for
so they're going to be split into separate PRs. Functionally though this
PR should have no impact other than a fiber is no longer required for
`Table::grow_async`.

* Remove #[cfg] gate

show more ...

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation

Refactor internals of table initialization and management (#11416)

* Refactor const-eval to use `Val`, not `ValRaw`

This commit refactors the evaluation of constant expressions during
instantiation for example to use `Val` instead of `ValRaw`. Previously
the usage of `ValRaw` meant that wasm was disallowed from performing a
GC during const evaluation, but currently constant expressions can
indeed perform a GC. The goal of this commit is to lift this limitation.
This is expected to be a minor slowdown for modules that hit this path,
but most modules shouldn't hit this in a hot loop since LLVM doesn't
generate modules that use this branch of const eval.

The usage of `Val` brings a number of benefits and refactorings
associated with it:

* Const-evaluation is generally safer than before since everything is
higher-level.
* GC types in const-eval were almost already using `Val` meaning that
there's actually fewer conversions now.
* Instantiation code was refactored to use `wasmtime::*`-API types
instead of low-level VM types. This deduplicates a good deal and lifts
complexity out of the raw VM bits.

Another issue that this commit fixes is to change how table
initialization is modeled internally in
`vm::Instance::table_init_segment`. Previously this was done by removing
the tables from an instance to get a split borrow into the store and the
table. This is not valid though because if, during initialization, a GC
is performed then the table is not present to find roots through the
table. This function is refactored to scope borrows to within a loop
instead of over a loop via various refactorings and such and usage of
higher level APIs. This is again, like above, expected to pessimize
performance but this is also not known to be a hot path for modules at
this time.

* Remove the `TableElement` type

This commit is a refactoring of how tables work within Wasmtime to avoid
funneling table elements through a `TableElement` enum internally.
Instead methods are "exploded" to `grow_{gc_ref,func,cont}` which means,
for example, funcrefs don't need a GcStore. The main motivation for this
change was to avoid the idiom where `TableElement` represents a cloned,
but unrooted, GC reference.

Prior to this commit there were a number of subtle bugs in the table
code for Wasmtime where write barriers were forgotten on `table.init`,
`table.set` (via the embedder API), and `table.grow`. While `table.fill`
correctly handled the GC references it was awkward to get everything
else working consistently so I opted to remove `TableElement` entirely
to make it more clear that `&VMGcRef` is ubiquitously used meaning that
the write barriers, for example, are the same as other parts of the
Wasmtime codebase.

This has a few extra tests for "make sure this doesn't leak" to ensure
that GC works correctly with new barriers in place.

* Fix some lints and warnings

* Fix wmemcheck build

* Review comments

* Optimize const eval and global initialization

* Fix compile

* Fix lints

show more ...

Minimize lazy allocation of the GC store (#11411)

* Minimize lazy allocation of the GC store

This commit is an effort to minimize the number of entrypoints which
might lazily allocate a GC store. T

Minimize lazy allocation of the GC store (#11411)

* Minimize lazy allocation of the GC store

This commit is an effort to minimize the number of entrypoints which
might lazily allocate a GC store. The is currently done through
`StoreOpaque::gc_store_mut` but this method is very commonly used
meaning that there are many many places to audit for lazily allocating a
GC store. The reason that this needs an audit is that lazy allocation
is an async operation right now that must be on a fiber and is something
I'm looking to fix as part of #11262.

This commit performs a few refactorings to achieve this:

* `gc_store_mut` is renamed to `ensure_gc_store`. This is intended to be
an `async` function in the future and clearly demarcates where lazy
allocation of a GC store is occurring.

* `require_gc_store{,_mut}` is now added which is a pure accessor of the
GC store with no lazy allocation. Most locations previously using
`gc_store_mut` are updated to use this instead.

Documentation is added to store methods to clearly indicate which ones
are allocating and which ones should only be called in a context where
allocation should already have happened.

* Fix configured build

* Relax GC store restrictions in more places

* Review comments on documentation

* Move `ensure_gc_store` calls during instantiation

Instead update `needs_gc_heap` with the tables that are added to a
module and rely on instantiation to create the GC heap.

* Shuffle around some code

* Fix CI and review comments

* Add in a few more i31 cases for externref

show more ...