Handle OOM in the rest of Wasmtime's non-component, -async, -compilation APIs (#12858)

* Handle OOM in more places in the public API

A bunch of random places:

* Add: `Trap::try_new` to handle OOM

Handle OOM in the rest of Wasmtime's non-component, -async, -compilation APIs (#12858)

* Handle OOM in more places in the public API

A bunch of random places:

* Add: `Trap::try_new` to handle OOM while creating traps
* Use: `TryVec` inside `Func::call_impl_do_call` and `wasm_val_raw_storage` to
hold the args and rets
* Add: `Instance::try_exports` for iterating over an instance's exports while
handling OOM
* `Linker:try_get`, like `Linker::get` but handling OOM
* `Linker:try_get_by_import`, like `Linker::get_by_import` but handling OOM
* Use `try_new` to box things in `SharedMemory::new`
* Use `TryVec` instead of `Vec` in our dynamic tables

* Add OOM tests for most of Wasmtime's public API

Excludes component-, async-, and compilation-related APIs.

* address review feedback

* fix test compilation

* fix c-api

show more ...

Adjust behavior of 4gb memories with custom page sizes (#12884)

* Adjust behavior of 4gb memories with custom page sizes

This commit adjust what happens when a linear memory grows up to 4gb
large w

Adjust behavior of 4gb memories with custom page sizes (#12884)

* Adjust behavior of 4gb memories with custom page sizes

This commit adjust what happens when a linear memory grows up to 4gb
large when custom page sizes are used. This is an open question in the
upstream proposal at WebAssembly/custom-page-sizes#45 but without any
special handling a return value of -1 is ambiguous if it succeeded or
failed. For now eagerly trap memory operations reaching these conditions
while the upstream specification question is resolved.

* Fix CI

* Debug CI failure

prtest:full

* Fix 32-bit platforms

show more ...

"Downgrade" threads support to tier 2, disable fuzzing (#12036)

* "Downgrade" threads support to tier 2, disable fuzzing

This commit is borne out of a fuzz bug that was opened recently. The
fuzz bu

"Downgrade" threads support to tier 2, disable fuzzing (#12036)

* "Downgrade" threads support to tier 2, disable fuzzing

This commit is borne out of a fuzz bug that was opened recently. The
fuzz bug specifically has to do with fallout from #12022, specifically
`SharedMemory` being used to allocated instead of `Memory`. In this
situation the resource limiter is no longer consulted meaning that
shared memories bypass this and aren't caught by OOM checks. This is
currently by design because `SharedMemory` instances don't know which
resource limiter to hook into per-store.

More generally though the implementation of wasm threads, while workable
in Wasmtime, has a number of known relatively large deficiencies. These
were not resolved prior to ungating the wasm proposal (that's on me) but
nevertheless the quality of implementation is not quite up to "tier 1
par" with the rest of what Wasmtime offers. Given this the threads
proposal is now downgraded to tier 2. To help minimize the impact of
this the wasm proposal is left enabled-by-default, but creation of a
`SharedMemory` in the Rust API requires opting-in via a new
`Config::shared_memory` method.

This commit shuffles around some documentation of wasm proposals to
split it into tier 1/2/3 instead of on/off-by-default and then adds a
column for whether the proposal is on-by-default.

* clangformat

* Fix tests

* Add tests for failed creation

Fix an issue where defined shared memories weren't gated

* Sync disabled threads stub

* Fix another test

prtest:full

* Fix fuzzing tests

* Fix dwarf tests

show more ...

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `Store

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `StoreOpaque::allocate_instance` is now an `async fn`.
This function does not actually use the `async`-ness just yet so it's a
noop from that point of view, but this propagates outwards to enough
locations that I wanted to split this off to make future changes more
digestable.

Notably some creation functions here such as making an `Instance`,
`Table`, or `Memory` are refactored internally to use this new `async`
function. Annotations of `assert_ready` or `one_poll` are used as
appropriate as well.

For reference this commit was benchmarked with our `instantiation.rs`
benchmark in the pooling allocator and shows no changes relative to the
original baseline from before-`async`-PRs.

* Make table/memory creation `async` functions

This commit is a large-ish refactor which is made possible by the many
previous refactorings to internals w.r.t. async-in-Wasmtime. The end
goal of this change is that table and memory allocation are both `async`
functions. Achieving this, however, required some refactoring to enable
it to work:

* To work with `Send` neither function can close over `dyn VMStore`.
This required changing their `Option<&mut dyn VMStore>` arugment to
`Option<&mut StoreResourceLimiter<'_>>`
* Somehow a `StoreResourceLimiter` needed to be acquired from an
`InstanceAllocationRequest`. Previously the store was stored here as
an unsafe raw pointer, but I've refactored this now so
`InstanceAllocationRequest` directly stores `&StoreOpaque` and
`Option<&mut StoreResourceLimiter>` meaning it's trivial to acquire
them. This additionally means no more `unsafe` access of the store
during instance allocation (yay!).
* Now-redundant fields of `InstanceAllocationRequest` were removed since
they can be safely inferred from `&StoreOpaque`. For example passing
around `&Tunables` is now all gone.
* Methods upwards from table/memory allocation to the
`InstanceAllocator` trait needed to be made `async`. This includes new
`#[async_trait]` methods for example.
* `StoreOpaque::ensure_gc_store` is now an `async` function. This
internally carries a new `unsafe` block carried over from before with
the raw point passed around in `InstanceAllocationRequest`. A future
PR will delete this `unsafe` block, it's just temporary.

I attempted a few times to split this PR up into separate commits but
everything is relatively intertwined here so this is the smallest
"atomic" unit I could manage to land these changes and refactorings.

* Shuffle `async-trait` dep

* Fix configured build

show more ...

Make memory growth an async function (#11460)

* Make memory growth an `async` function

This is an analog of #11442 but for memories. This had a little more
impact due to memories being hooked into

Make memory growth an async function (#11460)

* Make memory growth an `async` function

This is an analog of #11442 but for memories. This had a little more
impact due to memories being hooked into GC operations. Further
refactoring of GC operations to make them safer/more-async is deferred
to a future PR and for now it's "no worse than before". This is another
step towards #11430 and enables removing a longstanding `unsafe` block
in `runtime/memory.rs` which previously could not be removed.

One semantic change from this is that growth of a shared memory no
longer uses an async limiter. This is done to keep growth of a shared
memory consistent with creation of a shared memory where no limits are
applied. This is due to the cross-store nature of shared memories which
means that we can't tie growth to any one particular store. This
additionally fixes an issue where an rwlock write guard was otherwise
held across a `.await` point which creates a non-`Send` future, closing
a possible soundness hole in Wasmtime.

* Fix threads-disabled build

* Review comments

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Make a couple `vm::Memory` methods take `&self` instead of `&mut self` (#10543)

The `&mut self` was not necessary and also did not provide any help with using
the resulting `VMMemoryDefinition`s and

Make a couple `vm::Memory` methods take `&self` instead of `&mut self` (#10543)

The `&mut self` was not necessary and also did not provide any help with using
the resulting `VMMemoryDefinition`s and their raw pointers, so just make these
methods take `&self`.

show more ...

Enable warnings if `component-model` is disabled (#10141)

Continuation of work in #10131

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today th

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today this is not possible because
Pulley does not properly respect either strict or permissive provenance
models. The goal of this refactoring is to enable fixing this in a
future commit that doesn't touch everything in the codebase. Instead
everything is touched here in this commit.

The basic problem with Pulley is that it is incompatible with the strict
provenance model of Rust which means that we'll be using "exposed
provenance" APIs to satisfy Rust's soundness requirements. In this model
we must explicitly call `ptr.expose_provenance()` on any pointers
which are exposed to compiled code. Arguably we should also be already
doing this for natively-compiled code but I am not certain about how
strictly this is required.

Currently in Wasmtime today we call `ptr.expose_provenance()` nowhere.
It also turns out, though, that we share quite a few pointers in quite a
few places with compiled code. This creates a bit of a problem! The
solution settled on in this commit looks like:

* A new marker trait, `VmSafe`, is introduced. This trait is used to
represent "safe to share with compiled code" types and enumerates some
properties such as defined ABIs, primitives wrappers match primitive
ABIs, and notably "does not contain a pointer".

* A new type, `VmPtr<T>`, is added to represent pointers shared with
compiled code. Internally for now this is just `SendSyncPtr<T>` but in
the future it will be `usize`. By using `SendSyncPtr<T>` it shouldn't
actually really change anything today other than requiring a lot of
refactoring to get the types to line up.

* The core `vmctx_plus_offset*` methods are updated to require
`T: VmSafe`. Previously they allowed any `T` which is relatively
dangerous to store any possible Rust type in Cranelift-accessible
areas.

These three fundamental changes were introduced in this commit. All
further changes were refactoring necessary to get everything working
after these changes. For example many types in `vmcontext.rs`, such as
`VMFuncRef`, have changed to using `VmPtr<T>` instead of `NonNull<T>` or
`*mut T`. This is a pretty expansive change which resulted in touching a
lot of places.

One premise of `VmPtr<T>` is that it's non-null. This was an
additional refactoring that updated a lot of places where previously
`*mut T` was used and now either `VmPtr<T>` or `NonNull<T>` is used.

In the end the intention is that `VmPtr<T>` is used whenever pointers
are store in memory that can be accessed from Cranelift. When operating
inside of the runtime `NonNull<T>` or `SendSyncPtr<T>` is preferred
instead.

As a final note, no provenance changes have actually happened yet. For
example this doesn't fix Pulley in MIRI. What it does enable, though, is
that the future commit to fix Pulley in MIRI will be much smaller with
this having already landed.

* Run the full test suite in PR CI

prtest:full

* Minor CI issues

* Fix no_std build

* Fix miri build

* Don't use `VmPtr` in FFI function signatures

Use `NonNull` or `*mut u8` as appropriate for function signatures
instead. It shouldn't be required to use `VmPtr` during the handoff to
compiled code as we've already annotated the pointer going out.

* Fix rebase conflict

* Review comments

show more ...

Add a `signals-based-traps` Cargo compile-time feature (#9614)

* Gate signal handlers behind a new Cargo feature

This commit adds a new on-by-default Cargo feature to the `wasmtime`
crate named `si

Add a `signals-based-traps` Cargo compile-time feature (#9614)

* Gate signal handlers behind a new Cargo feature

This commit adds a new on-by-default Cargo feature to the `wasmtime`
crate named `signals-based-traps`. This is modeled after the
`Config::signals_based_traps` configuration at runtime and can be used
to statically disable the use of signal handlers in Wasmtime. This
notably reduces the number of platform dependencies that Wasmtime has
and provides a mode of avoiding relying on signals altogether.

This introduces a new `MallocMemory` which is a linear memory backed by
the system allocator. This new type of memory is enabled when virtual
memory guards are disabled and signals-based-traps are disabled. This
means that this new type of memory will be candidate for fuzzing for
example.

prtest:full

* Fix rebase conflict

* Refactor `MmapVec` documentation and representation

* Remove no-longer-needed `Arc`
* Document it may be backed by `Vec<u8>`

show more ...