Debugging: provide access to private (non-exported) entities. (#12367)

* Debugging: provide access to private (non-exported) entities.

A debugger will need to access all entities (globals, tables,

Debugging: provide access to private (non-exported) entities. (#12367)

* Debugging: provide access to private (non-exported) entities.

A debugger will need to access all entities (globals, tables, memories),
even those that are not exported, in order to provide a full debugging
experience: for example, a developer who has a debugger attached to a
Wasm component will expect to be able to see data in its memory.

Historically we have been very careful in Wasmtime to provide access to
Wasm instances' entities only as the Wasm type system allows -- that is,
only if they are exported. However, debugging is privileged -- in the
same way that a native host debugger has `ptrace` and can view
everything about the debuggee, we need to provide APIs for seeing
through the encapsulation boundary.

To ensure that this "violation of encapsulation" is scoped only to the
extent needed for the legitimate need (debugging), this API is
dynamically available only when `guest_debug` is configured true for a
given engine. Otherwise, the accessor returns `None`.

I opted not to provide a full introspection API that enumerates all of
the entities as the debugger should already have access to the debuggee
module and be able to enumerate the entities. Thus, the API only
provides a host-API handle when asking for an entity by index in a given
instance's index space.

* Review feedback.

* Fix tests on 32-bit build (where threads and thus shared memory are not supported)

show more ...

Prevent using shared memories with `Memory` (#12022)

* Prevent using shared memories with `Memory`

This commit fixes a few issues where it was possible to represent a wasm
shared linear memory with

Prevent using shared memories with `Memory` (#12022)

* Prevent using shared memories with `Memory`

This commit fixes a few issues where it was possible to represent a wasm
shared linear memory with the `wasmtime::Memory` type. This is not sound
because `wasmtime::Memory` provides safe Rust access to the bytes where
that is not possible with wasm shared memories. Shared memories in Rust
must be represented by `SharedMemory`, not `wasmtime::Memory`.

Specifically this commit prevents two vectors of this happening:

1. `Memory::new` now requires that the memory type specified is
non-shared. Instead `SharedMemory::new` must be used instead.

2. Core dumps now skip over shared memories when iterating over all
memories in the store. Supporting shared memories is left to a future
feature request for now.

* CI fixes

show more ...

Gut `vm::Export` to mostly be `crate::Extern` (#11229)

* Remove `Table::from_wasmtime_table`

This commit removes the unsafe function `Table::from_wasmtime_table`.
This goes a bit further and remove

Gut `vm::Export` to mostly be `crate::Extern` (#11229)

* Remove `Table::from_wasmtime_table`

This commit removes the unsafe function `Table::from_wasmtime_table`.
This goes a bit further and removes `ExportTable` entirely as well which
means that table lookup on a `vm::Instance` directly returns a
`wasmtime::Table` without any need to translate back-and-forth.

* Remove `Tag::from_wasmtime_tag`

Like the previous commit, but for tags.

* Remove `Global::from_wasmtime_global`

Like the previous commit, but for globals.

* Remove `Memory::from_wasmtime_memory`

Like the previous commit, but for memories.

* Remove `Func::from_wasmtime_function`

Like previous commits, but for functions.

* Fix lints

* Fill out missing safety comment

* Review comments

show more ...

Refactor globals to no longer use `Stored` (#10902)

This commit refactors the `wasmtime::Global` to avoid the usage of
`Stored<T>` internally. This makes conversion from internal global state
to ext

Refactor globals to no longer use `Stored` (#10902)

This commit refactors the `wasmtime::Global` to avoid the usage of
`Stored<T>` internally. This makes conversion from internal global state
to external global state a noop along the lines of previous commits. The
end goal is to remove `Stored` entirely and enable more pervasively using
external types internally within Wasmtime as well.

Globals were different than the prior iterations of memories, tags, and
tables. Globals have three different ways of defining them: wasm
instances, the host embedder, and component flags. Representing these
in all the various locations required a bit of finesse in how everything
is represented and stored at rest and such. In the end there's a small
amount of "type punning" in a few instance/vmctx fields related to
globals now since everything is squeezed into one slot. This is required
because the `VMGlobalImport` structure must have a size known to
wasm-compiled code and `wasmtime::Global` must have a known layout for C
code.

In the end while this is more code to manage globals my hope is that the
end result will be a net negative in terms of complexity by ensuring
that the embedder API is additionally suitable for use internally within
Wasmtime as well.

show more ...

Add a `DefinedTableIndex` to table imports/exports (#10879)

This commit extends `ExportTable` and `VMTableImport` (renamed from
`VMTable`) to include a `DefinedTableIndex` in the same manner that
me

Add a `DefinedTableIndex` to table imports/exports (#10879)

This commit extends `ExportTable` and `VMTableImport` (renamed from
`VMTable`) to include a `DefinedTableIndex` in the same manner that
memories carry their index they're defined at as well. The main goal of
this change is to power the next change which will update how tables are
stored in a store.

show more ...

Add `VMContext` and `DefinedTagIndex` fields to tag-related imports/exports (#10882)

* Add `VMContext` and `DefinedTagIndex` fields to tag-related imports/exports

Similar to memories/tables, will b

Add `VMContext` and `DefinedTagIndex` fields to tag-related imports/exports (#10882)

* Add `VMContext` and `DefinedTagIndex` fields to tag-related imports/exports

Similar to memories/tables, will be used in the next commit.

* Fix VMOffsets size of tag

show more ...

Exception and control tags (#10251)

* Tags

* Tag tests

* Tests

* Refer to tags issue

* Engine index

* Simplify

* Fix clippy warnings

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today th

Provenance preparation for Pulley (#10043)

* Provenance preparation for Pulley

This commit is an internal refactoring of Wasmtime's runtime to prepare
to execute Pulley in MIRI. Currently today this is not possible because
Pulley does not properly respect either strict or permissive provenance
models. The goal of this refactoring is to enable fixing this in a
future commit that doesn't touch everything in the codebase. Instead
everything is touched here in this commit.

The basic problem with Pulley is that it is incompatible with the strict
provenance model of Rust which means that we'll be using "exposed
provenance" APIs to satisfy Rust's soundness requirements. In this model
we must explicitly call `ptr.expose_provenance()` on any pointers
which are exposed to compiled code. Arguably we should also be already
doing this for natively-compiled code but I am not certain about how
strictly this is required.

Currently in Wasmtime today we call `ptr.expose_provenance()` nowhere.
It also turns out, though, that we share quite a few pointers in quite a
few places with compiled code. This creates a bit of a problem! The
solution settled on in this commit looks like:

* A new marker trait, `VmSafe`, is introduced. This trait is used to
represent "safe to share with compiled code" types and enumerates some
properties such as defined ABIs, primitives wrappers match primitive
ABIs, and notably "does not contain a pointer".

* A new type, `VmPtr<T>`, is added to represent pointers shared with
compiled code. Internally for now this is just `SendSyncPtr<T>` but in
the future it will be `usize`. By using `SendSyncPtr<T>` it shouldn't
actually really change anything today other than requiring a lot of
refactoring to get the types to line up.

* The core `vmctx_plus_offset*` methods are updated to require
`T: VmSafe`. Previously they allowed any `T` which is relatively
dangerous to store any possible Rust type in Cranelift-accessible
areas.

These three fundamental changes were introduced in this commit. All
further changes were refactoring necessary to get everything working
after these changes. For example many types in `vmcontext.rs`, such as
`VMFuncRef`, have changed to using `VmPtr<T>` instead of `NonNull<T>` or
`*mut T`. This is a pretty expansive change which resulted in touching a
lot of places.

One premise of `VmPtr<T>` is that it's non-null. This was an
additional refactoring that updated a lot of places where previously
`*mut T` was used and now either `VmPtr<T>` or `NonNull<T>` is used.

In the end the intention is that `VmPtr<T>` is used whenever pointers
are store in memory that can be accessed from Cranelift. When operating
inside of the runtime `NonNull<T>` or `SendSyncPtr<T>` is preferred
instead.

As a final note, no provenance changes have actually happened yet. For
example this doesn't fix Pulley in MIRI. What it does enable, though, is
that the future commit to fix Pulley in MIRI will be much smaller with
this having already landed.

* Run the full test suite in PR CI

prtest:full

* Minor CI issues

* Fix no_std build

* Fix miri build

* Don't use `VmPtr` in FFI function signatures

Use `NonNull` or `*mut u8` as appropriate for function signatures
instead. It shouldn't be required to use `VmPtr` during the handoff to
compiled code as we've already annotated the pointer going out.

* Fix rebase conflict

* Review comments

show more ...

Remove the `wasmtime_environ::MemoryPlan` type (#9532)

* Remove the `wasmtime_environ::MemoryPlan` type

This is the equivalent of #9530 for memories. The goal of this commit is
to eventually remove

Remove the `wasmtime_environ::MemoryPlan` type (#9532)

* Remove the `wasmtime_environ::MemoryPlan` type

This is the equivalent of #9530 for memories. The goal of this commit is
to eventually remove the abstraction layer of `MemoryPlan` and
`MemoryStyle` in favor of directly reading the configuration of
`Tunables`. The prediction is that it will be simpler to work directly
with configured values instead of a layer of abstraction between the
configuration and the runtime which needs to be evolved independently to
capture how to interpret the configuration.

Like with #9530 my plan is to eventually remove the `MemoryStyle` type
itself, but that'll be a larger change, so it's deferred to a future
PR.

* Fix shared memory disabled build

show more ...

Remove the `wasmtime_environ::TablePlan` type (#9530)

* Remove the `wasmtime_environ::TablePlan` type

In my quest to simplify memory configuration and how things work
internally in Wasmtime one thi

Remove the `wasmtime_environ::TablePlan` type (#9530)

* Remove the `wasmtime_environ::TablePlan` type

In my quest to simplify memory configuration and how things work
internally in Wasmtime one thing I've identified to accomplish is the
removal of the `TablePlan` and `MemoryPlan` types. These introduce an
abstraction layer between table/memory implementations and `Tunables`
and personally I find it simpler to directly reference `Tunables` and
use that instead. The goal of this commit is to plumb `Tunables` closer
to where it's directly read by removing the "indirection" through the
`*Plan` types.

The `TablePlan` and `MemoryPlan` types are pervasively used throughout
Wasmtime so instead of having one large commit delete everything this is
instead a piecemeal approach to incrementally get towards the goal of
removal. Here just `TablePlan` is removed and `Tunables` is plumbed in a
few more places. I plan to also in the future remove `TableStyle` and
`MemoryStyle` in favor of directly reading `Tunables` but that's left
for a future commit. For now `TableStyle` persists and its usage is a
bit odd in isolation but I plan to follow this up with the removal of
`TableStyle`.

* Fix non-component-model build

show more ...

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

* crates/test-programs typos

* crates/c-api typos

* crates/cache typos

* crates other typos

* cranelift/codegen/src/isa typos

* cranelift/codegen/src other typos

* cranelift/codegen other typos

* cranelift other typos

* ci js typo

* .github workflows typo

* RELEASES typo

* Fix clang-format documentation line

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTF

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTFLAGS` `--cfg` flag to be specified. With `no_std` becoming
officially supported this should provide a better onboarding experience
where the fallback custom platform is used. This will cause linker
errors if the symbols aren't implemented and searching/googling should
lead back to our docs/repo (eventually, hopefully).

* Change Wasmtime's TLS state to a single pointer

This commit updates the management of TLS to rely on just a single
pointer rather than a pair of a pointer and a `bool`. Additionally
management of the TLS state is pushed into platform-specific modules to
enable different means of managing it, namely the "custom" platform now
has a C function required to implement TLS state for Wasmtime.

* Delay conversion to `Instant` in atomic intrinsics

The `Duration` type is available in `no_std` but the `Instant` type is
not. The intention is to only support the `threads` proposal if `std` is
active but to assist with this split push the `Duration` further into
Wasmtime to avoid using a type that can't be mentioned in `no_std`.

* Gate more parts of Wasmtime on the `profiling` feature

Move `serde_json` to an optional dependency and gate the guest profiler
entirely on the `profiling` feature.

* Refactor conversion to `anyhow::Error` in `wasmtime-environ`

Have a dedicated trait for consuming `self` in addition to a
`Result`-friendly trait.

* Gate `gimli` in Wasmtime on `addr2line`

Cut down the dependency list if `addr2line` isn't enabled since then
the dependency is not used. While here additionally lift the version
requirement for `addr2line` up to the workspace level.

* Update `bindgen!` to have `no_std`-compatible output

Pull most types from Wasmtime's `__internal` module as the source of
truth.

* Use an `Option` for `gc_store` instead of `OnceCell`

No need for synchronization here when mutability is already available in
the necessary contexts.

* Enable embedder-defined host feature detection

* Add `#![no_std]` support to the `wasmtime` crate

This commit enables compiling the `runtime`, `gc`, and `component-model`
features of the `wasmtime` crate on targets that do not have `std`. This
tags the crate as `#![no_std]` and then updates everything internally to
import from `core` or `alloc` and adapt for the various idioms. This
ended up requiring some relatively extensive changes, but nothing too
too bad in the grand scheme of things.

* Require `std` for the perfmap profiling agent

prtest:full

* Fix build on wasm

* Fix windows build

* Remove unused import

* Fix Windows/Unix build without `std` feature

* Fix some doc links

* Remove unused import

* Fix build of wasi-common in isolation

* Fix no_std build on macos

* Re-fix build

* Fix standalone build of wasmtime-cli-flags

* Resolve a merge conflict

* Review comments

* Remove unused import

show more ...

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime_runtime` to `crate::runtime::vm`

* Remove dep on `wasmtime-runtime` from `wasmtime-cli`

* Move the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module

* Update labeler for merged crates

* Fix `publish verify`

prtest:full

show more ...