Handle OOM in `{Func,Memory,Table,Global}::new` and when calling an instance's exported function (#12855)

* Use `try_new` for `Box<dyn RuntimeLinearMemory>` in `DefaultMemoryCreator`

* Use `TryPrim

Handle OOM in `{Func,Memory,Table,Global}::new` and when calling an instance's exported function (#12855)

* Use `try_new` for `Box<dyn RuntimeLinearMemory>` in `DefaultMemoryCreator`

* Use `TryPrimaryMap` for `host_globals` in `Store`

* Add `Func::try_wrap` and use `try_new` for `Box<HostFunc>`

Add `Func::try_wrap` as a fallible version of `Func::wrap` that returns an error
on out-of-memory instead of panicking. `Func::wrap` now delegates to `try_wrap`.

Also use `try_new::<Box<_>>` instead of `Box::new` for `HostFunc`.

* Use `bumpalo`'s `try_alloc` for `FuncRefs`

* Use `try_new` for `Arc<Module>` in "trampoline" code

* Test that we handle OOM in `{Func,Memory,Table,Global}::new` and when calling an instance's exported function

* cargo fmt

show more ...

Use `TryVec` in the store's `FuncRefs` structure (#12845)

Rename our OOM-handling `Vec` to `TryVec` (#12721)

* Rename our OOM-handling `Vec` to `TryVec`

* fix tests that got mangled by the LSP server

* fix more tests mangled by LSP

Handle OOM in `Linker::instantiate_pre` (#12683)

Add a helper always-`Sync` utility to Wasmtime (#11453)

* Add a helper always-`Sync` utility to Wasmtime

This commit adds a newtype wrapper to Wasmtime, `AlwaysMut`, which is
unconditionally `Sync`

Add a helper always-`Sync` utility to Wasmtime (#11453)

* Add a helper always-`Sync` utility to Wasmtime

This commit adds a newtype wrapper to Wasmtime, `AlwaysMut`, which is
unconditionally `Sync` if the stored type is `Send`. This is similar to
a `Mutex<T>` where it promotes a `Send` bound to a `Sync` bound, but
it's unlike `Mutex<T>` in that `AlwaysMut<T>` has no synchronization.
The reason that this is safe is that `AlwaysMut<T>` completely disallows
access to the underlying data through `&self` and requires `&mut self`.
This is similar to how `Mutex::get_mut` is safe, for example.

This type cleans up a preexisting `unsafe impl Sync` block in funcref
management around `SendSyncBump` (`bumpalo::Bump` is `Send`, not `Sync`,
but we only access it through `&mut self`). This then additionally
removes `unsafe impl Sync for StoreFiber` which, upon reflection, is not
sound because we don't ever constraint the store's `T` type to `Sync`,
only `Send`. This is effectively no change throughout Wasmtime, however,
as fibers are only accessed with `&mut`.

Overall this is mostly just internal refactoring to reduce the amount of
`unsafe` inside of Wasmtime and to add a new utility to use in the
future too.

* Fix configured build

* Review comments

* Fix configured build

show more ...

Raise `unsafe_op_in_unsafe_fn` further in Wasmtime (#11322)

* Raise `unsafe_op_in_unsafe_fn` further in Wasmtime

Now it's at `wasmtime::runtime`, not just `wasmtime::runtime::vm`.

* Review comments

Refactor the representation of `Func` (#10897)

* Refactor the representation of `Func`

This commit rewrites the internals of `wasmtime::Func`. The `Stored`
type is no longer used meaning that it's

Refactor the representation of `Func` (#10897)

* Refactor the representation of `Func`

This commit rewrites the internals of `wasmtime::Func`. The `Stored`
type is no longer used meaning that it's now free to create a
`wasmtime::Func` at any time. It is effectively a store-tagged
`NonNull<VMFuncRef>`. This required a few internal changes to how
functions are passed around:

* Previously the insertion of a `wasm_call`-less `VMFuncRef` was
deferred until the raw pointer was loaded. Now the insertion happens
immediately as soon as the function is placed within a store. This is
done to ensure that a `Func` corresponds to one exact `VMFuncRef` and
that's it, and lazily filled in versions within the store are still
lazily filled in but they're more eagerly allocated. This isn't
expected to have much of an impact perf-wise since all these lazily
allocated functions were already almost guaranteed to get lazily
allocated anyway.

* Filling in "holes" in `VMFuncRef`, notably the `wasm_call` field, no
longer happens lazily when the `VMFuncRef` is demanded. Instead during
instantiation a pass is made to fill in holes with the new module
being instantiated (after registration). Additionally when a
lazily-allocated `VMFuncRef` is created the module registry is checked
immediately. This means that all store-local `VMFuncRef` values are
either filled in immediately or filled in during instantiation. This
notably means that the previous logic in `Func::vmimport` is now
"just" an `.unwrap()` with a lot of comments saying why the unwrap
shouldn't panic.

* To implement this commit a previous optimization for the `Func` API was
removed as well, namely `Func::call` will become slower after this
commit. The `Func::call` API is a dynamically-typed API which requires
run-time type-checking of arguments. Previously a `FuncType` was loaded
into a cache once-per-`Func` which helped amortize the cost of using
`Func::call` repeatedly. Now, though, there's no natural place to put
such a cache since `Func` no longer has dedicated storage within a
`Store`. Historically this optimization was added for the C API before
the `*_call_unchecked` APIs existed, but nowadays the `*_call_unchecked`
APIs should suffice for performance-critical applications where needed.
In the future it might also be possible to have a hash map in the
`Store` of a `VMSharedTypeIndex` to `FuncType` which is lazily populated
based on calls to `Func::call`, but that feels a bit overkill nowadays
for a possibly rarely-used map.

* The no-longer-necessary `RootedHostFunc` type is now gone as its
unsafety and various contracts are subsumed by other preexisting
`unsafe` blocks.

* The specific drop order between `StoreOpaque::store_data` and
`StoreOpaque::rooted_host_funcs` is removed. The `rooted_host_funcs`
field now lives in the `func_refs` field and the `FuncKind` type,
where the destructors came from before, is no more.

* The `func_refs` field has grown storage locations for a variety of
"keep this thing alive as long as the store" related to functions and
such.

Closes #10868

* Remove mutability from locations that no longer need it

* Update crates/wasmtime/src/runtime/func.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

* Update crates/wasmtime/src/runtime/store/func_refs.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today th

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today this is not possible because
Pulley does not properly respect either strict or permissive provenance
models. The goal of this refactoring is to enable fixing this in a
future commit that doesn't touch everything in the codebase. Instead
everything is touched here in this commit.

The basic problem with Pulley is that it is incompatible with the strict
provenance model of Rust which means that we'll be using "exposed
provenance" APIs to satisfy Rust's soundness requirements. In this model
we must explicitly call `ptr.expose_provenance()` on any pointers
which are exposed to compiled code. Arguably we should also be already
doing this for natively-compiled code but I am not certain about how
strictly this is required.

Currently in Wasmtime today we call `ptr.expose_provenance()` nowhere.
It also turns out, though, that we share quite a few pointers in quite a
few places with compiled code. This creates a bit of a problem! The
solution settled on in this commit looks like:

* A new marker trait, `VmSafe`, is introduced. This trait is used to
represent "safe to share with compiled code" types and enumerates some
properties such as defined ABIs, primitives wrappers match primitive
ABIs, and notably "does not contain a pointer".

* A new type, `VmPtr<T>`, is added to represent pointers shared with
compiled code. Internally for now this is just `SendSyncPtr<T>` but in
the future it will be `usize`. By using `SendSyncPtr<T>` it shouldn't
actually really change anything today other than requiring a lot of
refactoring to get the types to line up.

* The core `vmctx_plus_offset*` methods are updated to require
`T: VmSafe`. Previously they allowed any `T` which is relatively
dangerous to store any possible Rust type in Cranelift-accessible
areas.

These three fundamental changes were introduced in this commit. All
further changes were refactoring necessary to get everything working
after these changes. For example many types in `vmcontext.rs`, such as
`VMFuncRef`, have changed to using `VmPtr<T>` instead of `NonNull<T>` or
`*mut T`. This is a pretty expansive change which resulted in touching a
lot of places.

One premise of `VmPtr<T>` is that it's non-null. This was an
additional refactoring that updated a lot of places where previously
`*mut T` was used and now either `VmPtr<T>` or `NonNull<T>` is used.

In the end the intention is that `VmPtr<T>` is used whenever pointers
are store in memory that can be accessed from Cranelift. When operating
inside of the runtime `NonNull<T>` or `SendSyncPtr<T>` is preferred
instead.

As a final note, no provenance changes have actually happened yet. For
example this doesn't fix Pulley in MIRI. What it does enable, though, is
that the future commit to fix Pulley in MIRI will be much smaller with
this having already landed.

* Run the full test suite in PR CI

prtest:full

* Minor CI issues

* Fix no_std build

* Fix miri build

* Don't use `VmPtr` in FFI function signatures

Use `NonNull` or `*mut u8` as appropriate for function signatures
instead. It shouldn't be required to use `VmPtr` during the handoff to
compiled code as we've already annotated the pointer going out.

* Fix rebase conflict

* Review comments

show more ...

Remove the native ABI calling convention from Wasmtime (#8629)

* Remove the native ABI calling convention from Wasmtime

This commit proposes removing the "native abi" calling convention used
in Was

Remove the native ABI calling convention from Wasmtime (#8629)

* Remove the native ABI calling convention from Wasmtime

This commit proposes removing the "native abi" calling convention used
in Wasmtime. For background this ABI dates back to the origins of
Wasmtime. Originally Wasmtime only had `Func::call` and eventually I
added `TypedFunc` with `TypedFunc::call` and `Func::wrap` for a faster
path. At the time given the state of trampolines it was easiest to call
WebAssembly code directly without any trampolines using the native ABI
that wasm used at the time. This is the original source of the native
ABI and it's persisted over time under the assumption that it's faster
than the array ABI due to keeping arguments in registers rather than
spilling them to the stack.

Over time, however, this design decision of using the native ABI has not
aged well. Trampolines have changed quite a lot in the meantime and it's
no longer possible for the host to call wasm without a trampoline, for
example. Compilations nowadays maintain both native and array
trampolines for wasm functions in addition to host functions. There's a
large split between `Func::new` and `Func::wrap`. Overall, there's quite
a lot of weight that we're pulling for the design decision of using the
native ABI.

Functionally this hasn't ever really been the end of the world.
Trampolines aren't a known issue in terms of performance or code size.
There's no known faster way to invoke WebAssembly from the host (or
vice-versa). One major downside of this design, however, is that
`Func::new` requires Cranelift as a backend to exist. This is due to the
fact that it needs to synthesize various entries in the matrix of ABIs
we have that aren't available at any other time. While this is itself
not the worst of issues it means that the C API cannot be built without
a compiler because the C API does not have access to `Func::wrap`.

Overall I'd like to reevaluate given where Wasmtime is today whether it
makes sense to keep the native ABI trampolines. Sure they're supposed to
be fast, but are they really that much faster than the array-call ABI as
an alternative? This commit is intended to measure this.

This commit removes the native ABI calling convention entirely. For
example `VMFuncRef` is now one pointer smaller. All of `TypedFunc` now
uses `*mut ValRaw` for loads/stores rather than dealing with ABI
business. The benchmarks with this PR are:

* `sync/no-hook/core - host-to-wasm - typed - nop` - 5% faster
* `sync/no-hook/core - host-to-wasm - typed - nop-params-and-results` - 10% slower
* `sync/no-hook/core - wasm-to-host - typed - nop` - no change
* `sync/no-hook/core - wasm-to-host - typed - nop-params-and-results` - 7% faster

These numbers are a bit surprising as I would have suspected no change
in both "nop" benchmarks as well as both being slower in the
params-and-results benchmarks. Regardless it is apparent that this is
not a major change in terms of performance given Wasmtime's current
state. In general my hunch is that there are more expensive sources of
overhead than reads/writes from the stack when dealing with wasm values
(e.g. trap handling, store management, etc).

Overall this commit feels like a large simplification of what we
currently do in `TypedFunc`:

* The number of ABIs that Wasmtime deals with is reduced by one. ABIs
are pretty much always tricky and having fewer moving parts should
help improve the understandability of the system.
* All of the `WasmTy` trait methods and `TypedFunc` infrastructure is
simplified. Traits now work with simple `load`/`store` methods rather
than various other flavors of conversion.
* The multi-return-value handling of the native ABI is all gone now
which gave rise to significant complexity within Wasmtime's Cranelift
translation layer in addition to the `TypedFunc` backing traits.
* This aligns components and core wasm where components always use the
array ABI and now core wasm additionally will always use the array ABI
when communicating with the host.

I'll note that this still leaves a major ABI "complexity" with respect
to native functions do not have a wasm ABI function pointer until
they're "attached" to a `Store` with a `Module`. That's required to
avoid needing Cranelift for creating host functions and that property is
still true today. This is a bit simpler to understand though now that
`Func::new` and `Func::wrap` are treated uniformly rather than one being
special-cased.

* Fix miri unsafety

prtest:full

show more ...

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTF

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTFLAGS` `--cfg` flag to be specified. With `no_std` becoming
officially supported this should provide a better onboarding experience
where the fallback custom platform is used. This will cause linker
errors if the symbols aren't implemented and searching/googling should
lead back to our docs/repo (eventually, hopefully).

* Change Wasmtime's TLS state to a single pointer

This commit updates the management of TLS to rely on just a single
pointer rather than a pair of a pointer and a `bool`. Additionally
management of the TLS state is pushed into platform-specific modules to
enable different means of managing it, namely the "custom" platform now
has a C function required to implement TLS state for Wasmtime.

* Delay conversion to `Instant` in atomic intrinsics

The `Duration` type is available in `no_std` but the `Instant` type is
not. The intention is to only support the `threads` proposal if `std` is
active but to assist with this split push the `Duration` further into
Wasmtime to avoid using a type that can't be mentioned in `no_std`.

* Gate more parts of Wasmtime on the `profiling` feature

Move `serde_json` to an optional dependency and gate the guest profiler
entirely on the `profiling` feature.

* Refactor conversion to `anyhow::Error` in `wasmtime-environ`

Have a dedicated trait for consuming `self` in addition to a
`Result`-friendly trait.

* Gate `gimli` in Wasmtime on `addr2line`

Cut down the dependency list if `addr2line` isn't enabled since then
the dependency is not used. While here additionally lift the version
requirement for `addr2line` up to the workspace level.

* Update `bindgen!` to have `no_std`-compatible output

Pull most types from Wasmtime's `__internal` module as the source of
truth.

* Use an `Option` for `gc_store` instead of `OnceCell`

No need for synchronization here when mutability is already available in
the necessary contexts.

* Enable embedder-defined host feature detection

* Add `#![no_std]` support to the `wasmtime` crate

This commit enables compiling the `runtime`, `gc`, and `component-model`
features of the `wasmtime` crate on targets that do not have `std`. This
tags the crate as `#![no_std]` and then updates everything internally to
import from `core` or `alloc` and adapt for the various idioms. This
ended up requiring some relatively extensive changes, but nothing too
too bad in the grand scheme of things.

* Require `std` for the perfmap profiling agent

prtest:full

* Fix build on wasm

* Fix windows build

* Remove unused import

* Fix Windows/Unix build without `std` feature

* Fix some doc links

* Remove unused import

* Fix build of wasi-common in isolation

* Fix no_std build on macos

* Re-fix build

* Fix standalone build of wasmtime-cli-flags

* Resolve a merge conflict

* Review comments

* Remove unused import

show more ...

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime_runtime` to `crate::runtime::vm`

* Remove dep on `wasmtime-runtime` from `wasmtime-cli`

* Move the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module

* Update labeler for merged crates

* Fix `publish verify`

prtest:full

show more ...

Support compilation-only build by adding a `runtime` feature (#7766)

* Add `runtime` feature to `wasmtime` crate

This feature can be disabled to build `wasmtime` only for compilation.
This can be u

Support compilation-only build by adding a `runtime` feature (#7766)

* Add `runtime` feature to `wasmtime` crate

This feature can be disabled to build `wasmtime` only for compilation.
This can be useful when cross-compiling, especially on a target that
can't run wasmtime itself (e.g. `wasm32`).

* prtest:full

* don't round pages without runtime feature

* fix async assertions

* move profiling into runtime

* enable runtime for wasmtime-wasi

* enable runtime for c-api

* fix build_artifacts in non-cache case

* fix miri extensions

* enable runtime for wast

* enable runtime for explorer

* support cranelift all-arch on wasm32

* add doc links for `WeakEngine`

* simplify lib runtime cfgs

* move limits and resources to runtime

* move stack to runtime

* move coredump and debug to runtime

* add runtime to coredump and async features

* add wasm32 build job

* combine engine modules

* single compile mod

* remove allow for macro paths

* add comments

show more ...