Refactor `String` usage in `wasmtime_environ::Module` (#12565)

* Refactor `String` usage in `wasmtime_environ::Module`

Do not hold regular `String`s; instead use our own OOM-handling
`wasmtime_core

Refactor `String` usage in `wasmtime_environ::Module` (#12565)

* Refactor `String` usage in `wasmtime_environ::Module`

Do not hold regular `String`s; instead use our own OOM-handling
`wasmtime_core::alloc::String` or, even better, an interned `Atom` from the
`StringPool`.

Also avoid `IndexMap`, as it doesn't handle OOMs.

* Use OOM-handling `IndexMap` in `wasmtime_environ::Module`

* review feedback

show more ...

Do not re-export `anyhow!` in the `wasmtime::prelude` (#12298)

We are trying to move to `format_err!` instead.

Fix some panics compiling components and resources (#11798)

In working on bytecodealliance/wasm-tools#2335 I found that there's a
few test cases in wasm-tools which Wasmtime was panicking to compile

Fix some panics compiling components and resources (#11798)

In working on bytecodealliance/wasm-tools#2335 I found that there's a
few test cases in wasm-tools which Wasmtime was panicking to compile.
The issues were all related to resource types and how information wasn't
registered ahead of time before it was translated from wasmparser's
representation to Wasmtime's representation. The high-level cause for
this had to do with how component and instance types are handled, as
opposed to concrete components or instances themselves. This was
effectively a hole in Wasmtime's translation process for components
which has never been filled out since the original implementation of
resources. The reason that this never came up before is:

* Most components don't currently import or export a component itself.
* Most components don't currently import or export component or instance
types (as opposed to values).

One of these was required to trigger this issue. The solution
implemented in this commit is to plumb the concept of an "abstract
resource" which is part of a type but not actually ever used at runtime
except for type equality during type reflection. This is expected to
have little-to-no impact on real-world components given that these
situations are rarely occurring.

show more ...

update `component-model-async` plumbing (#11123)

* [DO NOT MERGE] update `component-model-async` plumbing

This pulls in the latest Component Model async ABI code from the
`wasip3-prototyping` repo,

update `component-model-async` plumbing (#11123)

* [DO NOT MERGE] update `component-model-async` plumbing

This pulls in the latest Component Model async ABI code from the
`wasip3-prototyping` repo, including various API refactors and spec updates.

This includes all the changes to the `wasmtime` crate from `wasip3-prototyping`
_except_ that the `concurrent` submodule and child submodules contain only
non-functional stubs. For that reason, and the fact that
e.g. `Func::call_async` is now implemented in terms of `Func::call_concurrent`,
most of the component model tests are failing. This commit is not meant to be
merged as-is; a follow-up commit (to be PR'd separately) will contain the real
`concurrent` implementation, at which point the tests will pass again. I'm
splitting these into separate PRs to make review easier.

Signed-off-by: Joel Dice <[email protected]>

* Undo wit-bindgen changes

No longer necessary after other refactors

* Move back to crates.io-based wit-bindgen

* Undo upgrade of http-body-util

(deferred for future PR)

* Add back in arbitrary use of async

Looks like it may have been lost by accident

* Make imports more conventional for Wasmtime

* Some minor changes

* Privatize a component field

* Cut down a bit on #[cfg]

* Undo a no-longer-necessary `pub`

* add doc comments for `{Future,Stream,ErrorContext}Any`

Signed-off-by: Joel Dice <[email protected]>

* rename `concurrent` stub module to `concurrent_disabled`

...and avoid panicking in the stubs.

Signed-off-by: Joel Dice <[email protected]>

* fix test regression

Signed-off-by: Joel Dice <[email protected]>

* revert `call_async` and `post_return_impl` changes

These will need to wait until the `component-model-async` feature is fully
implemented.

Signed-off-by: Joel Dice <[email protected]>

* remove unused struct

Signed-off-by: Joel Dice <[email protected]>

* add `Options::callback` field

This isn't used yet, but will be used when the real `component-model-async`
implementation is merged.

Signed-off-by: Joel Dice <[email protected]>

* Remove no-longer-needed feature

* Trim reexports from Wasmtime

Some of these are no longer needed or can be avoided with small changes.
Some deps are likely needed in the next commit but they'll be best added
there.

* Update test expectations

* More trimming of Cargo.toml

* Defer `*Buffer` traits to next PR

Not needed for this PR I believe.

* Use conventional Wasmtime imports + remove dummy_waker

* Reduce duplication in `*_disabled`

* Remove some unncessary bounds

* Remove some `for<'a>` bounds where unnecessary

* Remove another bound

* Defer more functions to the next PR

`drop_fibers` is different in the next PR, so defer it to then.

* Remove some reexports no longer necessary

Bindings generation changed awhile back so these aren't needed, defer
the implementations to the next PR.

* Remove unnecessary drop

This was already moved to `run_manual_drop_routines`

* Defer a `pub(crate)` to a future PR

* Expand comments in traphandlers

* Defer some types to the next PR

* Update linker documentation

* Add `Send`/`Sync` bounds to `ComponentType`

This commit is extracted from from review of #11123 and #11127. While
not literally present in those PRs it's my own personal conclusion that
it's best to just go ahead and add these bounds at the "base" of the
component trait hierarchy. The current implementation in #11123 adds
bounds in many locations and this would remove the need to add bounds
everywhere and instead have everything inherited through the `Lift` and
`Lower` traits.

This raises the question of: why? The main conclusion that I've reached
leading to this change is that Wasmtime currently will store `R`, a
return value, on the stack during the lowering process back into linear
memory. This might involve allocation, however, meaning that wasm can be
invoked and a context switch could happen. For Wasmtime's `unsafe impl`
of `Send` and `Sync` on fibers to be sound it requires that this
stack-local variable is also `Send` and `Sync` as it's an entirely
user-provided type. Thus I've concluded that for results it's always
required for these to be both `Send` and `Sync` (or at the very least,
`Send`).

Given that I've gone ahead and updated to require both `Send` and `Sync`
for both params and results. This is not expected to actually have any
impact in practice since all primitives are already `Send`/`Sync` (minus
`Rc` impls all removed here) and all `bindgen!`-generated types are
compositions of `Send`/`Sync` primitives meaning that they're also
`Send` and `Sync`.

* Remove some now-unnecessary bounds

* Fix build after #11160

* Remove some now-unnecessary duplicate bounds

* Uncomment test that now works

* Undo accidental doc wrap

* Clarify comment on `Value` types

Don't leave `TODO` in public-facing documentation ideally

* Actually resolve the conflict (forgot to commit)

* Avoid returning boxed futures in APIs

* Defer making constructors more public to a future PR

* Make a method name more conventional

* Refactor `linear_lift_into_from_memory`

* Drop the `max_count` parameter in favor of slicing the `WasmList`
itself. Avoids situations such as what happens if `max_count` is
larger than the length of the list.
* Don't have the default implementation collect to a vector and then
push all that onto a different vector. Instead push each item
individually through `extend`.

* De-indent a block of code added

* Remove unsafety from `prepare_call`

Mostly move the parameters themselves to the closure to avoid raw
pointers/drop/etc.

This will have the consequence of in the future `call_async` is going to
now require `Params: 'static` but that seems more-or-less inevitable at
this point.

* Go back to returning box, alas.

* Apply same treatment to lift function

Make it a closure and reduce some levels of indirection of the various
functions in play.

* Refactor `lower_params` to require less context.

Relax the bounds on the closure specified since it's immediately called
and then additionally take out parameters/captures that the closure can
carry itself.

* Don't pass extraneous `Instance` parameter

This can now be inferred from `Func`.

* Clean up some SAFETY comments

* Generalize the signature of `lift_results`

* Move `lift_results` function to `Func`

Also rename the lift/lower helpers to `with_{lift,lower}_context`

* Remove parameter from `with_lift_context`

Like `with_lower_context` this is fine to capture in the closure passed
in.

* Simplify the dynamic lifting logic

Don't call `with_lift_context` in two locations, only call it once with
a dynamic parameter.

* Refactor away the `Func::lift_results_sync` helper

* Use `with_lift_context` in `call_raw`

* Simplify a call to `Func::call_unchecked_raw`

* Ungate `with_lift_context` to fix non-cm-async build

* Fix compile (bad cherry-pick conflict resolution)

* Use `with_lower_context` in `call_raw`

Trying to unify the async/concurrent paths as much as possible.

* Move params out of `call_raw`

Let closures capture the params, no need to thread it through as an
unnecessary argument.

* Clean up unsafety in `Func::call_raw`

* Accurately mark `call_raw` itself as `unsafe`, then document why
callers should be safe.
* Don't have one large `unsafe` block in `call_raw`, instead split it up
with separate safety comments.

* Move a one-off type definition closer to its use

* Avoid intermediate allocations in dynamic calls

* Simplify a future-return site

* Deduplicate checking parameter count

* Simplify a future invocation with `?`

* Simplify a variable declaration

* Simplify some function signatures

* Remove outdated safety comment

* Refactor to not require `Params: 'static` on `call_async`

* Remove no-longer-necessary SAFETY comment

* Fix typos

* Add a fast-path with no `Box` for sync host functions

Speeds up host calls by ~20% and puts them back on parity with the
beforehand numbers Wasmtime has.

* Synchronize signatures of async/concurrent dynamic calls

Use slices for both instead of vecs for one and slices for the other.
Required some slight rejiggering. Apparently one can solve a closure
problem with another closure, then one surely has no more closure
problems.

* Fix non-cm-async build

---------

Signed-off-by: Joel Dice <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

show more ...

Remove `Stored` usage from `wasmtime::component::Instance` (#10913)

* Remove mutability required on a number of component methods

Powered by previous refactorings it's possible to just take
`&Store

Remove `Stored` usage from `wasmtime::component::Instance` (#10913)

* Remove mutability required on a number of component methods

Powered by previous refactorings it's possible to just take
`&StoreOpaque` in these locations.

* Move core instance map into `ComponentInstance`

This commit moves the mapping from a `RuntimeInstanceIndex` to a
`wasmtime::Instance` from `wasmtime::component::InstanceData` into the
internal `ComponentInstance` structure. This is done in preparation to
remove `InstanceData` eventually.

* Move `Component` to live in `ComponentInstance`

Previously this lived in `InstanceData` but that's just a historical
artifact of the old `wasmtime` and `wasmtime-runtime` split. Nowadays
it's possible to store `Component` directly to further move information
out of `InstanceData`.

* Move `InstanceData::imports` to `ComponentInstance`

More preparation for the removal of `InstanceData` entirely.

* Move lookup functions from `InstanceData` to `ComponentInstance`

More prep for the removal of `InstanceData`.

* Reduce some more reliance on `InstanceState`

Just removing it from some type signatures to make future removal
easier.

* Remove an unused result from a helper function

No caller needs it, so go ahead and remove it.

* Make `ComponentInstance::imports` private

This moves some methods around and documents preexisting `unsafe`
contracts to get this field private and prevent external access to it.

* Remove `InstanceData` from `Instantiator`.

Use `OwnedComponentInstance` instead.

* Remove `InstanceData`, refactor `Instance`

This commit refactors the representation of `component::Instance` to be
purely index-based to data already within a store. This makes creation
of an `Instance` free compared to before where auxiliary data needed to
be created. Note that this doesn't actually change storage within a
`Store<T>` as instances are still there, it's just that now the storage
lives in a non-`Stored`-related location.

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

wasmtime: remove Any-typing from the vm's representation of resource types table (#10607)

This any-typing is a left over escape hatch from the wasmtime /
wasmtime-runtime divide, but now that the ty

wasmtime: remove Any-typing from the vm's representation of resource types table (#10607)

This any-typing is a left over escape hatch from the wasmtime /
wasmtime-runtime divide, but now that the types all live in the same
crate we can eliminate the Any and just use the concrete type
everywhere.

show more ...

Wasm GC: Fix an incorrect assertion and canonicalize types for runtime usage in ExternType::from_wasmtime (#10223)

* Fix assertion in `PartialEq` for `RegisteredType` again

It is possible for two `

Wasm GC: Fix an incorrect assertion and canonicalize types for runtime usage in ExternType::from_wasmtime (#10223)

* Fix assertion in `PartialEq` for `RegisteredType` again

It is possible for two `WasmSubType`s to be equal to each other, as far as
`derive(PartialEq)` is concerned, but still different from each other if they
are in different rec groups or even if they are at different indices within the
same rec group. The assertion mistakenly did not permit either of these,
however.

Fixes #9714

* Canonicalize all types for runtime usage when creating `wasmtime::{Module,Component}`s

Rather than canonicalizing them on demand in functions like
`{Func,Global,Table}Type::from_wasmtime` and other places. Instead, we do it in
one place, up front, so that it is very unlikely we miss anything. Doing this
involves changing some things from `ModuleInternedTypeIndex`es to
`EngineOrModuleTypeIndex`es in `wasmtime_environ`, which means that a bunch of
uses of those things need to unwrap the appropriate kind of type index at usage
sites (e.g. compilation uses will unwrap `ModuleInternedTypeIndex`es, runtime
uses usage will unwrap `VMSharedTypeIndex`es). And it additionally required
implementing the `TypeTrace` trait for a handful of things to unlock the
provided `canonicalize_for_runtime_usage` trait method for those things.

All this machinery is required to avoid an assertion failure in the regression
test introduced in the previous commit, which was triggered because we were
failing to canonicalize type indices inside `ExternType`s for runtime usage on
some code paths. We shouldn't have to play that kind of whack-a-mole in the
future, thanks to this new approach.

* Fix a warning in no-default-features builds

* Fix another warning in weird cfg builds

show more ...

Implement semver compatibility for exports (#8830)

* Implement semver compatibility for exports

This commit is an implementation of component model semver compatibility
for export lookups. Previous

Implement semver compatibility for exports (#8830)

* Implement semver compatibility for exports

This commit is an implementation of component model semver compatibility
for export lookups. Previously in #7994 component imports were made
semver-aware to ensure that bumping version numbers would not be a
breaking change. This commit implements the same feature for component
exports. This required some refactoring to move the definition of semver
compat around and the previous refactoring in #8786 enables frontloading
this work to happen before instantiation.

Closes #8395

* Review comments

* Fix tests

show more ...

Add `anyhow` stuff to our internal `wasmtime` crate prelude (#8804)

* Add `anyhow` stuff to our internal `wasmtime` crate prelude

We use it basically everywhere and it is annoying to have to import

Add `anyhow` stuff to our internal `wasmtime` crate prelude (#8804)

* Add `anyhow` stuff to our internal `wasmtime` crate prelude

We use it basically everywhere and it is annoying to have to import.

I also did an audit of existing `use` statements and removed the now-redundant
ones and replaced one-off imports with usage of the prelude, so that the prelude
is available by default in more places.

* Fix `cargo doc`

show more ...

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

* crates/test-programs typos

* crates/c-api typos

* crates/cache typos

* crates other typos

* cranelift/codegen/src/isa typos

* cranelift/codegen/src other typos

* cranelift/codegen other typos

* cranelift other typos

* ci js typo

* .github workflows typo

* RELEASES typo

* Fix clang-format documentation line

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTF

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTFLAGS` `--cfg` flag to be specified. With `no_std` becoming
officially supported this should provide a better onboarding experience
where the fallback custom platform is used. This will cause linker
errors if the symbols aren't implemented and searching/googling should
lead back to our docs/repo (eventually, hopefully).

* Change Wasmtime's TLS state to a single pointer

This commit updates the management of TLS to rely on just a single
pointer rather than a pair of a pointer and a `bool`. Additionally
management of the TLS state is pushed into platform-specific modules to
enable different means of managing it, namely the "custom" platform now
has a C function required to implement TLS state for Wasmtime.

* Delay conversion to `Instant` in atomic intrinsics

The `Duration` type is available in `no_std` but the `Instant` type is
not. The intention is to only support the `threads` proposal if `std` is
active but to assist with this split push the `Duration` further into
Wasmtime to avoid using a type that can't be mentioned in `no_std`.

* Gate more parts of Wasmtime on the `profiling` feature

Move `serde_json` to an optional dependency and gate the guest profiler
entirely on the `profiling` feature.

* Refactor conversion to `anyhow::Error` in `wasmtime-environ`

Have a dedicated trait for consuming `self` in addition to a
`Result`-friendly trait.

* Gate `gimli` in Wasmtime on `addr2line`

Cut down the dependency list if `addr2line` isn't enabled since then
the dependency is not used. While here additionally lift the version
requirement for `addr2line` up to the workspace level.

* Update `bindgen!` to have `no_std`-compatible output

Pull most types from Wasmtime's `__internal` module as the source of
truth.

* Use an `Option` for `gc_store` instead of `OnceCell`

No need for synchronization here when mutability is already available in
the necessary contexts.

* Enable embedder-defined host feature detection

* Add `#![no_std]` support to the `wasmtime` crate

This commit enables compiling the `runtime`, `gc`, and `component-model`
features of the `wasmtime` crate on targets that do not have `std`. This
tags the crate as `#![no_std]` and then updates everything internally to
import from `core` or `alloc` and adapt for the various idioms. This
ended up requiring some relatively extensive changes, but nothing too
too bad in the grand scheme of things.

* Require `std` for the perfmap profiling agent

prtest:full

* Fix build on wasm

* Fix windows build

* Remove unused import

* Fix Windows/Unix build without `std` feature

* Fix some doc links

* Remove unused import

* Fix build of wasi-common in isolation

* Fix no_std build on macos

* Re-fix build

* Fix standalone build of wasmtime-cli-flags

* Resolve a merge conflict

* Review comments

* Remove unused import

show more ...

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime_runtime` to `crate::runtime::vm`

* Remove dep on `wasmtime-runtime` from `wasmtime-cli`

* Move the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module

* Update labeler for merged crates

* Fix `publish verify`

prtest:full

show more ...

Update nightly used in CI and fix warnings (#8416)

Fixes some warnings that nightly Rust has started emitting.

Add a component type accessor for `Component` (#8078)

* Add a component type accessor for `Component`

All the bits are already in place so all that's needed is to touch up
some docs and fixup a min

Add a component type accessor for `Component` (#8078)

* Add a component type accessor for `Component`

All the bits are already in place so all that's needed is to touch up
some docs and fixup a minor case uncovered through testing. Otherwise
this enables runtime reflection over the imports and exports of an
uninstantiated component.

* Run test through miri

show more ...

Better error message for wrong import implementation (#7645)

Signed-off-by: Ryan Levick <[email protected]>

Move dtor information from `ResourceAny` into a `Store` (#8061)

* Move dtor information from `ResourceAny` into a `Store`

This commit moves the internals of the `ResourceAny` type related to
destru

Move dtor information from `ResourceAny` into a `Store` (#8061)

* Move dtor information from `ResourceAny` into a `Store`

This commit moves the internals of the `ResourceAny` type related to
destructors into the auxiliary data within a `Store`. This avoids the
need to carry around pointers with `ResourceAny` and additionally makes
it easier to work with.

As part of this commit this also updates the behavior of
`ResourceAny::try_from_resource` to no longer need an `InstancePre` and
type information. This was required originally to get
`ResourceAny::resource_drop` working to drop the host resource. In
retrospect I'm not sure if this was the best goal to achieve because
`Resource<T>` already has no destructor support and one of the more
common use cases for the conversion is to simply turn around and give it
back to a component where a component already has enough destructor
information.

In the end this should make both `ResourceAny` and `Resource<T>` pretty
close to "just an index" which is easier to reason about when working
with resources than carrying additional pointers.

* Remove now-unneeded ResourceImportIndex machinery

show more ...

Make `component::Linker` semver-aware (#7994)

This commit is an implementation of #7860 for Wasmtime where
`wasmtime::component::Linker` is now "semver aware". This means that it
assumes that hosts

Make `component::Linker` semver-aware (#7994)

This commit is an implementation of #7860 for Wasmtime where
`wasmtime::component::Linker` is now "semver aware". This means that it
assumes that hosts are always managing WIT interfaces in a
semver-aare fashion meaning that semver-compatible upgrade exclusively
adds functionality. This neatly fits into the idea of subtyping at the
instance-level where if a binary built against 0.2.0 only requests a
subset of functionality from a runtime that provides 0.2.1, that should
work just fine.

Specifically what this change does is:

* For all names inserted into a `Linker` there might also be a "semver
compatible name" which is registered as well. For example `[email protected]`
is also registered as `..@1`.

* Semver-compatible names are only provided for versions without a
prerelease and with either a nonzero major or minor version number.

* When looking up an item in the linker if no exact match is found then
if a semver-compatible-name is available for the lookup key then
that's consulted as well.

This semantically means that if a components imports WASI 0.2.0 then a
runtime which only provides WASI 0.2.1 will be able to instantiate the
component. Furthermore if a component imports WASI 0.2.1 but only
imports the subset of WASI that was available in 0.2.0 then it will be
instantiable in a runtime that only supports 0.2.0.

This implementation is intended to be a crucial part of the evolution to
WASI to make it more seamless to upgrade WASI from both a host and guest
perspective. This no longer requires everyone to upgrade to the same
version all at the same time but instead decouples the upgrade
schedules.

Closes #7860

show more ...

Support compilation-only build by adding a `runtime` feature (#7766)

* Add `runtime` feature to `wasmtime` crate

This feature can be disabled to build `wasmtime` only for compilation.
This can be u

Support compilation-only build by adding a `runtime` feature (#7766)

* Add `runtime` feature to `wasmtime` crate

This feature can be disabled to build `wasmtime` only for compilation.
This can be useful when cross-compiling, especially on a target that
can't run wasmtime itself (e.g. `wasm32`).

* prtest:full

* don't round pages without runtime feature

* fix async assertions

* move profiling into runtime

* enable runtime for wasmtime-wasi

* enable runtime for c-api

* fix build_artifacts in non-cache case

* fix miri extensions

* enable runtime for wast

* enable runtime for explorer

* support cranelift all-arch on wasm32

* add doc links for `WeakEngine`

* simplify lib runtime cfgs

* move limits and resources to runtime

* move stack to runtime

* move coredump and debug to runtime

* add runtime to coredump and async features

* add wasm32 build job

* combine engine modules

* single compile mod

* remove allow for macro paths

* add comments

show more ...