Add a new `ResourceDynamic` type (#11885)

* Move `ResourceAny` to its own submodule

In preparation for upcoming other refactorings...

* Move `Resource<T>` to its own submodule

* Move `ResourceTy

Add a new `ResourceDynamic` type (#11885)

* Move `ResourceAny` to its own submodule

In preparation for upcoming other refactorings...

* Move `Resource<T>` to its own submodule

* Move `ResourceType` to its own submodule

* Move host resource table infrastructure to its own module

* Add module documentation to resource-related modules

* Add a new `ResourceDynamic` type

This commit adds a new type `wasmtime::component::ResourceDynamic` to
the embedding API which internally is more-or-less the same as
`Resource<T>` but with different type information. This
`ResourceDynamic` specifically enables storing runtime information for
managing the type of the resource as opposed to `Resource<T>` which
requires static information.

The main goal of this type is to enable exposing resources in the C API
of Wasmtime. Currently there is no way to define different resource
types in the C API because the only option is `Resource<T>`. With this
type it will be possible to create distinct resource types in the C API
for embedders to use.

cc #11437

* Apply suggestions from code review

Co-authored-by: Joel Dice <[email protected]>

---------

Co-authored-by: Joel Dice <[email protected]>

show more ...

Fix some panics compiling components and resources (#11798)

In working on bytecodealliance/wasm-tools#2335 I found that there's a
few test cases in wasm-tools which Wasmtime was panicking to compile

Fix some panics compiling components and resources (#11798)

In working on bytecodealliance/wasm-tools#2335 I found that there's a
few test cases in wasm-tools which Wasmtime was panicking to compile.
The issues were all related to resource types and how information wasn't
registered ahead of time before it was translated from wasmparser's
representation to Wasmtime's representation. The high-level cause for
this had to do with how component and instance types are handled, as
opposed to concrete components or instances themselves. This was
effectively a hole in Wasmtime's translation process for components
which has never been filled out since the original implementation of
resources. The reason that this never came up before is:

* Most components don't currently import or export a component itself.
* Most components don't currently import or export component or instance
types (as opposed to values).

One of these was required to trigger this issue. The solution
implemented in this commit is to plumb the concept of an "abstract
resource" which is part of a type but not actually ever used at runtime
except for type equality during type reflection. This is expected to
have little-to-no impact on real-world components given that these
situations are rarely occurring.

show more ...

Rename `Lift` and `Lower` trait methods (#11070)

* `Lower::lower` becomes `Lower::linear_lower_to_flat`
* `Lower::store` becomes `Lower::linear_lower_to_memory`
* `Lift::lift` becomes `Lift::linear_

Rename `Lift` and `Lower` trait methods (#11070)

* `Lower::lower` becomes `Lower::linear_lower_to_flat`
* `Lower::store` becomes `Lower::linear_lower_to_memory`
* `Lift::lift` becomes `Lift::linear_lift_from_flat`
* `Lift::load` becomes `Lift::linear_lift_from_memory`

This renaming is to distinguish these linear-memory methods from the GC versions
that will be added in follow up commits.

No functional changes here, just renaming.

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Add `T: 'static` to `Store<T>` (#10760)

* Add `T: 'static` to `Store<T>

Since the beginning the `T` type parameter on `Store<T>` has had no
bounds on it. This was intended for maximal flexibility i

Add `T: 'static` to `Store<T>` (#10760)

* Add `T: 'static` to `Store<T>

Since the beginning the `T` type parameter on `Store<T>` has had no
bounds on it. This was intended for maximal flexibility in terms of what
embedders place within a `Store<T>` and I've personally advocated that
we need to keep it this way. In the development of the WASIp3 work,
however, I've at least personally reached the conclusion that this is no
longer tenable and proceeding will require adding a `'static` bound to
data within a store.

Wasmtime today [already] carries unsafe `transmute`s to work around this
lack of `'static` bound, and while the number of `unsafe` parts is
relatively small right now we're still fundamentally lying to the
compiler about lifetime bounds internally. With the WASIp3 async work
this degree of "lying" has become even worse. Joel has written up some
examples [on Zulip] about how the Rust compiler is requiring `'static`
bounds in surprising ways. These patterns are cropping up quite
frequently in the WASIp3 work and it's becoming particularly onerous
maintaining all of the `unsafe` and ensuring that everything is in sync.

In the WASIp3 repository I've additionally [prototyped a change] which
would additionally practically require `T: 'static` in more locations.
This change is one I plan on landing in Wasmtime in the near future and
while its main motivations are for enabling WASIp3 work it is also a
much nicer system than what we have today, in my opinion.

Overall the cost of not having `T: 'static` on `Store<T>` is effectively
becoming quite costly, in particular with respect to WASIp3 work. This
is coupled with all known embedders already using `T: 'static` data
within a `Store<T>` so the expectation of the impact of this change is
not large. The main downside of this change as a result is that when and
where to place `'static` bounds is sort of a game of whack-a-mole with
the compiler. For example I changed `Store<T>` to require `'static`
here, but the rest of the change is basically "hit compile until rustc
says it's ok". There's not necessarily a huge amount of rhyme-or-reason
to where `'static` bounds crop up, which can be surprising or difficult
to work with for users.

In the end I feel that this change is necessary and one we can't shy
away from. If problems crop up we'll need to figure out how to thread
that needle at that time, but I'm coming around to thinking that
`T: 'static` is just a fundamental constraint we'll have to take on at
this time. Maybe a future version of Rust that fixes some of Joel's
examples (if they can be fixed, we're not sure of that) we could
consider relaxing this but that's left for future work.

[already]: https://github.com/bytecodealliance/wasmtime/blob/35053d6d8d1a5d4692cf636cba0c920b4a79a44b/crates/wasmtime/src/runtime/store.rs#L602-L611
[on Zulip]: https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/.22type.20may.20not.20live.20long.20enough.22.20for.20generic.20closure/near/473862072
[prototyped a change]: https://github.com/bytecodealliance/wasip3-prototyping/pull/158

* Remove a no-longer-necessary `unsafe` block

* Update test expectations

* Fix gc-disabled builds

show more ...

Collapse resource tables to be per-instance, not per-resource (#10701)

This commit is an update to how Wasmtime represents and processes
resource handles for components. The upstream specification c

Collapse resource tables to be per-instance, not per-resource (#10701)

This commit is an update to how Wasmtime represents and processes
resource handles for components. The upstream specification changed in
WebAssembly/component-model#427 and while it's been awhile since that
change this is Wasmtime catching up to that change. It's expected that
these paths are going to get stressed more with component model async
and more kinds of handles so this additionally refactors things to leave
sort of "holes" where async resources are going to go.

This notably will result in different behavior for guest programs.
Previously if there were two resources imported into a guest they could
have overlapping indices and now they're going to all have disjoint
indices since they're all in the same index space. It's expected that
this is will have a minimal, if any, impact on component guests as in
theory they're all mostly treating handles as opaque indexes today
anyway.

show more ...

Support platforms without 64-bit atomics (#10134)

* Support platforms without 64-bit atomics

This commit enables Wasmtime to build on platforms without 64-bit atomic
instructions, such as Rust's `r

Support platforms without 64-bit atomics (#10134)

* Support platforms without 64-bit atomics

This commit enables Wasmtime to build on platforms without 64-bit atomic
instructions, such as Rust's `riscv32imac-unknown-none-elf` target.
There are only two users of 64-bit atomics right now which are epochs
and allocation of `StoreId`. This commit adds `#[cfg]` to epoch-related
infrastructure in the runtime to turn that all of if 64-bit atomics
aren't available. The thinking is that epochs more-or-less don't work
without 64-bit atomics so it's easier to just remove them entirely.

Allocation of `StoreId` is trickier though because it's so core to
Wasmtime and it basically can't be removed. I've opted to change the
allocator to 32-bit indices instead of 64-bit indices. Note that
`StoreId` requires unique IDs to be allocated for safety which means
that while a 64-bit integer won't overflow for a few thousand years a
32-bit integer will overflow in a few hours from quickly creating
stores. The rough hope though is that embeddings on platforms like this
aren't churning through stores. Regardless if this condition is
triggered it'll result in a panic rather than unsoundness, so we
hopefully have at least that going for us.

Closes #8768

* Update component resources to not use `AtomicU64`

These aren't intended to be used in threaded contexts anyway and the use
of `AtomicXXX` is just to have interior mutability while still being
`Send` and `Sync`. Switch to using `AtomicU32` which is more portable.

* Use `RwLock<u64>` for `StoreId` instead.

* Fix compile

* Fix imports

show more ...

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today th

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today this is not possible because
Pulley does not properly respect either strict or permissive provenance
models. The goal of this refactoring is to enable fixing this in a
future commit that doesn't touch everything in the codebase. Instead
everything is touched here in this commit.

The basic problem with Pulley is that it is incompatible with the strict
provenance model of Rust which means that we'll be using "exposed
provenance" APIs to satisfy Rust's soundness requirements. In this model
we must explicitly call `ptr.expose_provenance()` on any pointers
which are exposed to compiled code. Arguably we should also be already
doing this for natively-compiled code but I am not certain about how
strictly this is required.

Currently in Wasmtime today we call `ptr.expose_provenance()` nowhere.
It also turns out, though, that we share quite a few pointers in quite a
few places with compiled code. This creates a bit of a problem! The
solution settled on in this commit looks like:

* A new marker trait, `VmSafe`, is introduced. This trait is used to
represent "safe to share with compiled code" types and enumerates some
properties such as defined ABIs, primitives wrappers match primitive
ABIs, and notably "does not contain a pointer".

* A new type, `VmPtr<T>`, is added to represent pointers shared with
compiled code. Internally for now this is just `SendSyncPtr<T>` but in
the future it will be `usize`. By using `SendSyncPtr<T>` it shouldn't
actually really change anything today other than requiring a lot of
refactoring to get the types to line up.

* The core `vmctx_plus_offset*` methods are updated to require
`T: VmSafe`. Previously they allowed any `T` which is relatively
dangerous to store any possible Rust type in Cranelift-accessible
areas.

These three fundamental changes were introduced in this commit. All
further changes were refactoring necessary to get everything working
after these changes. For example many types in `vmcontext.rs`, such as
`VMFuncRef`, have changed to using `VmPtr<T>` instead of `NonNull<T>` or
`*mut T`. This is a pretty expansive change which resulted in touching a
lot of places.

One premise of `VmPtr<T>` is that it's non-null. This was an
additional refactoring that updated a lot of places where previously
`*mut T` was used and now either `VmPtr<T>` or `NonNull<T>` is used.

In the end the intention is that `VmPtr<T>` is used whenever pointers
are store in memory that can be accessed from Cranelift. When operating
inside of the runtime `NonNull<T>` or `SendSyncPtr<T>` is preferred
instead.

As a final note, no provenance changes have actually happened yet. For
example this doesn't fix Pulley in MIRI. What it does enable, though, is
that the future commit to fix Pulley in MIRI will be much smaller with
this having already landed.

* Run the full test suite in PR CI

prtest:full

* Minor CI issues

* Fix no_std build

* Fix miri build

* Don't use `VmPtr` in FFI function signatures

Use `NonNull` or `*mut u8` as appropriate for function signatures
instead. It shouldn't be required to use `VmPtr` during the handoff to
compiled code as we've already annotated the pointer going out.

* Fix rebase conflict

* Review comments

show more ...

Remove usage of `gen` as an identifier (#9958)

This commit is a start to some preparation for the Rust 2024 edition for
Wasmtime and this workspace. The `rust-2024-compatibility` lint group in
rustc

Remove usage of `gen` as an identifier (#9958)

This commit is a start to some preparation for the Rust 2024 edition for
Wasmtime and this workspace. The `rust-2024-compatibility` lint group in
rustc, currently off-by-default, is intended to assist with migrating
code to prepare for the 2024 edition. Some of those lints though are, in
my opinion, far too noisy to be turned on so this PR doesn't turn on the
whole group. Instead though I plan on enabling individual lints over
time in our `Cargo.toml` before the 2024 edition is enabled. This should
hopefully provide a relatively smooth and less churn-y path to enabling
the 2024 edition in the future.

The first lint enabled here in this commit is the `keyword_idents_2024`
lint which warns against usage of identifiers that will become keywords
in the 2024 edition. The only one affecting Wasmtime is the `gen`
identifier (soon to be keyword) and we had quite a few instances of it.
Where possible I've renamed to `generator` or `generate` or `generated`
but when used as methods from upstream crate traits (e.g.
`RngCore::gen`) we're forced to use `r#gen`. The `rand` crate will be
updated in 0.9.0 to avoid this keyword so this shouldn't be permanent.

show more ...

Use a helper method to invoke `VMFuncRef::array_call` (#9630)

* Use a helper method to invoke `VMFuncRef::array_call`

This is intended to encapsulate the usage of a native raw pointer and in
the fu

Use a helper method to invoke `VMFuncRef::array_call` (#9630)

* Use a helper method to invoke `VMFuncRef::array_call`

This is intended to encapsulate the usage of a native raw pointer and in
the future act as a dispatch point for invoking Pulley instead of native
code.

* Fix some callers

* More fixese

show more ...

Update nightly in CI to latest (#9195)

Additionally address some new warnings that are cropping up throughout
the codebase.

Add `anyhow` stuff to our internal `wasmtime` crate prelude (#8804)

* Add `anyhow` stuff to our internal `wasmtime` crate prelude

We use it basically everywhere and it is annoying to have to import

Add `anyhow` stuff to our internal `wasmtime` crate prelude (#8804)

* Add `anyhow` stuff to our internal `wasmtime` crate prelude

We use it basically everywhere and it is annoying to have to import.

I also did an audit of existing `use` statements and removed the now-redundant
ones and replaced one-off imports with usage of the prelude, so that the prelude
is available by default in more places.

* Fix `cargo doc`

show more ...

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTF

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTFLAGS` `--cfg` flag to be specified. With `no_std` becoming
officially supported this should provide a better onboarding experience
where the fallback custom platform is used. This will cause linker
errors if the symbols aren't implemented and searching/googling should
lead back to our docs/repo (eventually, hopefully).

* Change Wasmtime's TLS state to a single pointer

This commit updates the management of TLS to rely on just a single
pointer rather than a pair of a pointer and a `bool`. Additionally
management of the TLS state is pushed into platform-specific modules to
enable different means of managing it, namely the "custom" platform now
has a C function required to implement TLS state for Wasmtime.

* Delay conversion to `Instant` in atomic intrinsics

The `Duration` type is available in `no_std` but the `Instant` type is
not. The intention is to only support the `threads` proposal if `std` is
active but to assist with this split push the `Duration` further into
Wasmtime to avoid using a type that can't be mentioned in `no_std`.

* Gate more parts of Wasmtime on the `profiling` feature

Move `serde_json` to an optional dependency and gate the guest profiler
entirely on the `profiling` feature.

* Refactor conversion to `anyhow::Error` in `wasmtime-environ`

Have a dedicated trait for consuming `self` in addition to a
`Result`-friendly trait.

* Gate `gimli` in Wasmtime on `addr2line`

Cut down the dependency list if `addr2line` isn't enabled since then
the dependency is not used. While here additionally lift the version
requirement for `addr2line` up to the workspace level.

* Update `bindgen!` to have `no_std`-compatible output

Pull most types from Wasmtime's `__internal` module as the source of
truth.

* Use an `Option` for `gc_store` instead of `OnceCell`

No need for synchronization here when mutability is already available in
the necessary contexts.

* Enable embedder-defined host feature detection

* Add `#![no_std]` support to the `wasmtime` crate

This commit enables compiling the `runtime`, `gc`, and `component-model`
features of the `wasmtime` crate on targets that do not have `std`. This
tags the crate as `#![no_std]` and then updates everything internally to
import from `core` or `alloc` and adapt for the various idioms. This
ended up requiring some relatively extensive changes, but nothing too
too bad in the grand scheme of things.

* Require `std` for the perfmap profiling agent

prtest:full

* Fix build on wasm

* Fix windows build

* Remove unused import

* Fix Windows/Unix build without `std` feature

* Fix some doc links

* Remove unused import

* Fix build of wasi-common in isolation

* Fix no_std build on macos

* Re-fix build

* Fix standalone build of wasmtime-cli-flags

* Resolve a merge conflict

* Review comments

* Remove unused import

show more ...

Use rustdoc's `doc_auto_cfg` feature instead of `doc_cfg` (#8532)

This commit removes all our `#[cfg_attr(..., doc(cfg(...)))]`
annotations throughout Wasmtime and `wasmtime-wasi`. These are all
rep

Use rustdoc's `doc_auto_cfg` feature instead of `doc_cfg` (#8532)

This commit removes all our `#[cfg_attr(..., doc(cfg(...)))]`
annotations throughout Wasmtime and `wasmtime-wasi`. These are all
replaced with `feature(doc_auto_cfg)` which automatically infers the
attribute to show rather than requiring us to duplicate it.
Spot-checking the docs this looks just-as-readable while being much
easier to maintain over time.

show more ...

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime_runtime` to `crate::runtime::vm`

* Remove dep on `wasmtime-runtime` from `wasmtime-cli`

* Move the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module

* Update labeler for merged crates

* Fix `publish verify`

prtest:full

show more ...

Clear borrow tracking converting `ResourceAny` to `Resource<T>` (#8071)

This commit clears out the dynamic tracking of borrows within a
`ResourceAny` when it's converted to a `Resource<T>`. The `Res

Clear borrow tracking converting `ResourceAny` to `Resource<T>` (#8071)

This commit clears out the dynamic tracking of borrows within a
`ResourceAny` when it's converted to a `Resource<T>`. The `Resource<T>`
type doesn't participate in this borrow tracking in the same way as
`ResourceAny` because it's assumed that hosts are well-behaved in this
regard. This means that the conversion here should respect these
guarantees of `Resource<T>` and auto-clear the borrow tracking.

show more ...

Move dtor information from `ResourceAny` into a `Store` (#8061)

* Move dtor information from `ResourceAny` into a `Store`

This commit moves the internals of the `ResourceAny` type related to
destru

Move dtor information from `ResourceAny` into a `Store` (#8061)

* Move dtor information from `ResourceAny` into a `Store`

This commit moves the internals of the `ResourceAny` type related to
destructors into the auxiliary data within a `Store`. This avoids the
need to carry around pointers with `ResourceAny` and additionally makes
it easier to work with.

As part of this commit this also updates the behavior of
`ResourceAny::try_from_resource` to no longer need an `InstancePre` and
type information. This was required originally to get
`ResourceAny::resource_drop` working to drop the host resource. In
retrospect I'm not sure if this was the best goal to achieve because
`Resource<T>` already has no destructor support and one of the more
common use cases for the conversion is to simply turn around and give it
back to a component where a component already has enough destructor
information.

In the end this should make both `ResourceAny` and `Resource<T>` pretty
close to "just an index" which is easier to reason about when working
with resources than carrying additional pointers.

* Remove now-unneeded ResourceImportIndex machinery

show more ...

Fix a panic reflecting the types of a component (#8031)

This commit fixes an issue when inspecting the types of a component
where if a resource type wasn't substituted through an import it would
end

Fix a panic reflecting the types of a component (#8031)

This commit fixes an issue when inspecting the types of a component
where if a resource type wasn't substituted through an import it would
end up panicking. The fix here is to add a third kind of resource type
which represents an uninstantiated component as opposed to an
instantiated component or host type.

Closes #8003

show more ...

Reserve handle index 0 in the component model (#7661)

This commit updates the allocation scheme for resources in the component
model to start at 1 instead of 0 when communicating with components.
Th

Reserve handle index 0 in the component model (#7661)

This commit updates the allocation scheme for resources in the component
model to start at 1 instead of 0 when communicating with components.
This is an implementation of WebAssembly/component-model#284.

While this broke a number of tests we have this shouldn't actually break
any components in practice. The broken tests were all overly-precise in
their assertions and error messages and this shouldn't idiomatically
come up in any guest language, so this should not be a practically
breaking change.

This change additionally places an upper limit on the maximum
allocatable index at `1 << 30` which is also specified in the above PR.

show more ...

Perform stronger typechecks of host-owned resources (#7902)

* Perform stronger typechecks of host-owned resources

Right now the abstraction for host-owned resources in Wasmtime is quite
simple, it'

Perform stronger typechecks of host-owned resources (#7902)

* Perform stronger typechecks of host-owned resources

Right now the abstraction for host-owned resources in Wasmtime is quite
simple, it's "just an index". This can be error prone because the host
can suffer both from use-after-free and ABA-style problems. While
there's not much that can be done about use-after-free the previous
implementation would accidentally enable "AB" style problems where if a
host-owned resource index was deallocated and then reallocated with
another type the original handle would still work. While not a major bug
this can be confusing and additionally violate's our guarantees as a
component runtime to guests to ensure that resources always have valid
`rep`s going into components.

This commit adds a new layer of storage which keeps track of the
`ResourceType` for all host-owned resources. This means that resources
going into a host-owned table now record their type and coming out of a
host-owned table a typecheck is always performed. Note that guests can
continue to skip this logic as they already have per-type tables and so
won't suffer the same issue.

This change is inspired by my taking a look at #7883. The crux of the
issue was a typo where a resource was reused by accident which led to
confusing behavior due to the reuse. This change instead makes it return
an error more quickly and doesn't allow the component to see any invalid
state.

Closes #7883

* Fix test

* Use generation counters, not types

* Review comments

show more ...

Fix more nightly warnings (#7907)

Additionally give `-Zcheck-cfg` a spin with Cargo and fix some mistakes
with our `--cfg` and `#[cfg]` directives.

Support compilation-only build by adding a `runtime` feature (#7766)

* Add `runtime` feature to `wasmtime` crate

This feature can be disabled to build `wasmtime` only for compilation.
This can be u

Support compilation-only build by adding a `runtime` feature (#7766)

* Add `runtime` feature to `wasmtime` crate

This feature can be disabled to build `wasmtime` only for compilation.
This can be useful when cross-compiling, especially on a target that
can't run wasmtime itself (e.g. `wasm32`).

* prtest:full

* don't round pages without runtime feature

* fix async assertions

* move profiling into runtime

* enable runtime for wasmtime-wasi

* enable runtime for c-api

* fix build_artifacts in non-cache case

* fix miri extensions

* enable runtime for wast

* enable runtime for explorer

* support cranelift all-arch on wasm32

* add doc links for `WeakEngine`

* simplify lib runtime cfgs

* move limits and resources to runtime

* move stack to runtime

* move coredump and debug to runtime

* add runtime to coredump and async features

* add wasm32 build job

* combine engine modules

* single compile mod

* remove allow for macro paths

* add comments

show more ...