[44.0.0] Merged backports for security advisories (#13007)

* fix(environ): repair unsound StringPool::try_clone()

The 43.0 release introduced a soundness bug in StringPool::try_clone(): the
cloned

[44.0.0] Merged backports for security advisories (#13007)

* fix(environ): repair unsound StringPool::try_clone()

The 43.0 release introduced a soundness bug in StringPool::try_clone(): the
cloned map retains &'static str keys pointing into the original pool's
strings storage. Once the original Linker is dropped those keys dangle.

Cloning a Linker, then dropping the original one, leaves a linker whose
registered imports could no longer be found, causing instantiation to
fail with "unknown import".

Signed-off-by: Flavio Castelli <[email protected]>

* Fix pooling allocator predicate to reset VM permissions

This commit fixes a mistake that was introduced in #9583 where the logic
to reset a linear memory slot in the pooling allocator used the wrong
predicate. Specifically VM permissions must be reset if virtual memory
can be relied on at all, and the preexisting predicate of
`can_elide_bounds_check` was an inaccurate representation of this. The
correct predicate to check is `can_use_virtual_memory`.

* winch: Fix the type of the `table.size` output register

This commit corrects the tagged size of the output of the `table.size`
instruction. Previously this was hardcoded as a 32-bit integer instead
of consulting the table's index type to use the
index-type-sized-register instead.

* winch: Fix a host panic when executing `table.fill`

This commit fixes a possible panic when a Winch-compiled module executes
the `table.fill` instruction. Refactoring in #11254 updated Cranelift
but forgot to update Winch meaning that Winch's indices were still using
the module-level indices instead of the `DefinedTableIndex` space. This
adds some tests and updates Winch's translation to use preexisting
helpers.

* x64: Fix `f64x2.splat` without SSE3

Don't sink a load into `pshufd` which loads 16 bytes, instead force
`put_in_xmm` to ensure only 8 bytes are loaded.

* Properly verify alignment in string transcoding

This commit updates string transcoding between guest modules to properly
verify alignment. Previously alignment was only verified on the first
allocation, not reallocations, which is not spec-compliant. This
additionally fixes a possible host panic when dealing with unaligned
pointers.

* Fix type confusion in AArch64 amode RegScaled folding

* winch: Add add_uextend to perform explicit extension when needed.

This commit fixes an out-of-bounds access caused by the lack zero
extension in the code responsible for calculating the heap address for
loads/stores.

This issue manifests in aarch64 (unlike x64) given that no automatic
extension is performed, resulting in an out-of-bounds access.

An alternative approach is to emit an extend for the index, however
this approach is preferred given that it gives the MacroAssembler
layer better control of how to lower addition, e.g., in aarch64 we can
inline the desired extension in a single instruction.

* winch: Correctly type the result of table.grow

This commit fixes an out-of-bounds access caused by the lack of type
narrowing from the `table.grow` builtin. Without explicit narrowing,
the type is treated as 64-bit value, which could cause issues when
paired with loads/stores.

* Review comments

* Properly handle table index types

Only narrow when dealing with the 64-bit pointer/32-bit tables

* Fix panic with out-of-bounds flags in `Value`

This commit fixes a panic when a component model `Value` is lifted from
a flags value which specifies out-of-bounds bits as 1. This is specified
in the component model to ignore the out-of-bounds bits, which `flags!`
correctly did (and thus `bindgen!`), but `Value` treated out-of-bounds
bits as a panic due to indexing an array.

* Fix bounds checks in FACT's `string_to_compact` method

We need to bounds check the source byte length, not the number of code units.

* Add missing realloc validation in string transcoding

This commit adds a missing validation that a return value of `realloc`
is inbounds during string transcoding. This was accidentally missing on
the transcoding path from `utf8` to `latin1+utf16` which meant that a
nearly-raw pointer could get passed to the host to perform the
transcode.

* winch: Refine zero extension heuristic

This commit refines the zero extension heuristic such that it
unconditionally emits a zero extension when dealing with 32-bit
heaps. This eliminates any ambiguity related to the value of the
memory indices across ISAs.

* Fix failure on 32-bit

* Fix miri test

---------

Signed-off-by: Flavio Castelli <[email protected]>
Co-authored-by: Flavio Castelli <[email protected]>
Co-authored-by: Shun Kashiwa <[email protected]>
Co-authored-by: Saúl Cabrera <[email protected]>
Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Add support for map type (#12216)

* Add support for map type

Signed-off-by: Yordis Prieto <[email protected]>

* Add Map and MapEntry classes to support key/value pairs in component model

Th

Add support for map type (#12216)

* Add support for map type

Signed-off-by: Yordis Prieto <[email protected]>

* Add Map and MapEntry classes to support key/value pairs in component model

This commit introduces the Map and MapEntry classes, enabling the representation of map values in the component model. The Map class allows for the creation and iteration of key/value pairs, enhancing the functionality of the wasmtime component API. Additionally, the .gitignore file is updated to exclude build artifacts from the crates/c-api directory.

* Add wasm_component_model_map configuration support

* Format code

* Format C code

* Enhance component model to support HashMap<K, V> type

This commit introduces support for HashMap<K, V> in the component model, allowing maps to be represented as list<tuple<K, V>> in the canonical ABI. It includes implementations for the ComponentType, Lower, and Lift traits for HashMap, enabling type checking, lowering to flat representations, and lifting from memory. Additionally, the maximum depth for type generation in the fuzzing utility is updated to accommodate the new map type.

* Refactor component configuration to introduce map support

This commit removes the previous wasm features configuration and adds new functions for creating a map-configured engine. The `map_config` and `map_engine` functions are introduced to facilitate the use of the component model with maps in tests, ensuring that the engine is properly configured for map types in the component model.

* Add new WAST test for map types and remove map type definitions from existing tests

This commit introduces a new WAST test file specifically for testing various map types in the component model. Additionally, it removes the redundant map type definitions from the existing types.wast file to streamline the test suite.

* Update component fuzzing and dynamic tests to replace call_and_post_return with call

* Format code

* Refactor HashMap usage in typed.rs to use wasmtime_environ collections

* Fix HashMap initialization and insertion to handle potential errors in typed.rs

* Refactor HashMap handling in typed.rs to use lower_map_iter for improved iteration and memory management. Introduce new implementations for ComponentType, Lower, and Lift traits for std::collections::HashMap, enhancing support for map types in the component model.

* Fix map adapter trampoline compilation and alignment bugs

The translate_map function had two categories of bugs preventing map
adapter trampolines from working:

1. Wasm stack discipline: local_set_new_tmp emits LocalSet which pops
from the stack, but was called when the stack was empty (to
"pre-allocate" locals). Fixed by computing values first, then
calling local_set_new_tmp to consume them—matching translate_list's
pattern. Also removed an erroneous LocalTee that left an orphan
value on the stack. Affected: src_byte_len, dst_byte_len,
cur_src_ptr, cur_dst_ptr.

2. Pointer advancement: after value translation, the pointer still
points at the value start. The code only advanced by trailing
padding instead of value_size + trailing_padding, causing every
loop iteration to re-read the same memory.

Also fixes entry layout to use proper record alignment rules (entry
align = max(key_align, value_align), value at aligned offset).

* Refactor map entry layout calculations to use canonical ABI

* Remove unnecessary clone of map pairs during lowering

Val::Map already holds Vec<(Val, Val)> which derefs to &[(Val, Val)],
matching lower_map's signature directly. The intermediate Vec allocation
and deep clone of every key/value pair was redundant.

* Deduplicate map lift logic between HashMap implementations

* Deduplicate list and map sequence translation scaffolding

* Fix cargo fmt formatting issues

* Deduplicate map typecheck logic

* Deduplicate map lowering with linear_lower_map_to_flat and linear_lower_map_to_memory helpers

* Clean up lift_try_map: use drop, move TryHashMap import to module scope

* Fix CI: arbtest overflow and no-std HashMap lift_map

- component_fuzz: use saturating_sub in generate_hashable_key to prevent
underflow when fuel is 0 and Enum variant is chosen
- typed: remove incorrect ? operators in lift_map for hashbrown::HashMap
(with_capacity and insert don't return Result)

* Store map tuple layout in TypeMap

Compute map entry ABI and value offsets once during type building, and reuse that metadata in runtime map lift/lower paths instead of recalculating tuple layout at each call site.

* Refactor map ABI argument passing

Bundle map lift/lower layout and type metadata into a small MapAbi32 helper so map helper calls stay concise without changing behavior.

* Fix CI: enable component_model_map in fuzzing and handle map in arbitrary_val

The fuzzer's component_api oracle was generating map types but the engine
didn't have the map feature enabled, and arbitrary_val had no arm for
Type::Map. Enable component_model_map in the store helper (matching how
component_model_async is forced on) and implement arbitrary value generation
for map types.

---------

Signed-off-by: Yordis Prieto <[email protected]>

show more ...

deprecate `[Typed]Func::post_return[_async]` and make them no-ops (#12498)

* deprecate `[Typed]Func::post_return[_async]` and make them no-ops

With the advent of the Component Model concurrency ABI

deprecate `[Typed]Func::post_return[_async]` and make them no-ops (#12498)

* deprecate `[Typed]Func::post_return[_async]` and make them no-ops

With the advent of the Component Model concurrency ABI and it's `task.return`
intrinsic, post-return functions have been informally deprecated and are
expected to be removed for WASI 1.0 and the corresponding stable edition of the
Component Model. Consequently, it does not make sense anymore to require
embedders to explicitly call the post-return function after using
`[Typed]Func::call[_async]`.

As of this commit, `[Typed]Func::post_return[_async]` are no-ops. Instead, the
post-return function is called automatically as part of
`[Typed]Func::call[_async]` if present, which is how
`[Typed]Func::call_concurrent` has worked all along. In addition, this commit
fixes and tests a couple of cases where the task and/or thread was being
disposed of before the post-return function was called.

* address review feedback

* test post-return function in more scenarios

Specifically, I've split the `invoke_post_return` test into multiple tests:

- using `TypedFunc::call`
- using `TypedFunc::call_async` with concurrency support enabled
- using `TypedFunc::call_async` with concurrency support disabled
- using `Func::call_async` with concurrency support disabled
- using `TypedFunc::call_concurrent`

* remove GCC/clang-specific deprecation attribute

This broke the MSVC build.

* bless bindgen output

* remove obsolete post-return functions and fields

Now that post-return calls are handled internally without requiring explicit
action by the embedder, we can avoid unnecessary bookkeeping.

show more ...

Migrate the Wasmtime CLI to `wasmtime::error` (#12295)

* Migrate wasmtime-cli to `wasmtime::error`

* migrate benches to `wasmtime::error` as well

* Remove new usage of anyhow that snuck in

update `component-model-async` plumbing (#11123)

* [DO NOT MERGE] update `component-model-async` plumbing

This pulls in the latest Component Model async ABI code from the
`wasip3-prototyping` repo,

update `component-model-async` plumbing (#11123)

* [DO NOT MERGE] update `component-model-async` plumbing

This pulls in the latest Component Model async ABI code from the
`wasip3-prototyping` repo, including various API refactors and spec updates.

This includes all the changes to the `wasmtime` crate from `wasip3-prototyping`
_except_ that the `concurrent` submodule and child submodules contain only
non-functional stubs. For that reason, and the fact that
e.g. `Func::call_async` is now implemented in terms of `Func::call_concurrent`,
most of the component model tests are failing. This commit is not meant to be
merged as-is; a follow-up commit (to be PR'd separately) will contain the real
`concurrent` implementation, at which point the tests will pass again. I'm
splitting these into separate PRs to make review easier.

Signed-off-by: Joel Dice <[email protected]>

* Undo wit-bindgen changes

No longer necessary after other refactors

* Move back to crates.io-based wit-bindgen

* Undo upgrade of http-body-util

(deferred for future PR)

* Add back in arbitrary use of async

Looks like it may have been lost by accident

* Make imports more conventional for Wasmtime

* Some minor changes

* Privatize a component field

* Cut down a bit on #[cfg]

* Undo a no-longer-necessary `pub`

* add doc comments for `{Future,Stream,ErrorContext}Any`

Signed-off-by: Joel Dice <[email protected]>

* rename `concurrent` stub module to `concurrent_disabled`

...and avoid panicking in the stubs.

Signed-off-by: Joel Dice <[email protected]>

* fix test regression

Signed-off-by: Joel Dice <[email protected]>

* revert `call_async` and `post_return_impl` changes

These will need to wait until the `component-model-async` feature is fully
implemented.

Signed-off-by: Joel Dice <[email protected]>

* remove unused struct

Signed-off-by: Joel Dice <[email protected]>

* add `Options::callback` field

This isn't used yet, but will be used when the real `component-model-async`
implementation is merged.

Signed-off-by: Joel Dice <[email protected]>

* Remove no-longer-needed feature

* Trim reexports from Wasmtime

Some of these are no longer needed or can be avoided with small changes.
Some deps are likely needed in the next commit but they'll be best added
there.

* Update test expectations

* More trimming of Cargo.toml

* Defer `*Buffer` traits to next PR

Not needed for this PR I believe.

* Use conventional Wasmtime imports + remove dummy_waker

* Reduce duplication in `*_disabled`

* Remove some unncessary bounds

* Remove some `for<'a>` bounds where unnecessary

* Remove another bound

* Defer more functions to the next PR

`drop_fibers` is different in the next PR, so defer it to then.

* Remove some reexports no longer necessary

Bindings generation changed awhile back so these aren't needed, defer
the implementations to the next PR.

* Remove unnecessary drop

This was already moved to `run_manual_drop_routines`

* Defer a `pub(crate)` to a future PR

* Expand comments in traphandlers

* Defer some types to the next PR

* Update linker documentation

* Add `Send`/`Sync` bounds to `ComponentType`

This commit is extracted from from review of #11123 and #11127. While
not literally present in those PRs it's my own personal conclusion that
it's best to just go ahead and add these bounds at the "base" of the
component trait hierarchy. The current implementation in #11123 adds
bounds in many locations and this would remove the need to add bounds
everywhere and instead have everything inherited through the `Lift` and
`Lower` traits.

This raises the question of: why? The main conclusion that I've reached
leading to this change is that Wasmtime currently will store `R`, a
return value, on the stack during the lowering process back into linear
memory. This might involve allocation, however, meaning that wasm can be
invoked and a context switch could happen. For Wasmtime's `unsafe impl`
of `Send` and `Sync` on fibers to be sound it requires that this
stack-local variable is also `Send` and `Sync` as it's an entirely
user-provided type. Thus I've concluded that for results it's always
required for these to be both `Send` and `Sync` (or at the very least,
`Send`).

Given that I've gone ahead and updated to require both `Send` and `Sync`
for both params and results. This is not expected to actually have any
impact in practice since all primitives are already `Send`/`Sync` (minus
`Rc` impls all removed here) and all `bindgen!`-generated types are
compositions of `Send`/`Sync` primitives meaning that they're also
`Send` and `Sync`.

* Remove some now-unnecessary bounds

* Fix build after #11160

* Remove some now-unnecessary duplicate bounds

* Uncomment test that now works

* Undo accidental doc wrap

* Clarify comment on `Value` types

Don't leave `TODO` in public-facing documentation ideally

* Actually resolve the conflict (forgot to commit)

* Avoid returning boxed futures in APIs

* Defer making constructors more public to a future PR

* Make a method name more conventional

* Refactor `linear_lift_into_from_memory`

* Drop the `max_count` parameter in favor of slicing the `WasmList`
itself. Avoids situations such as what happens if `max_count` is
larger than the length of the list.
* Don't have the default implementation collect to a vector and then
push all that onto a different vector. Instead push each item
individually through `extend`.

* De-indent a block of code added

* Remove unsafety from `prepare_call`

Mostly move the parameters themselves to the closure to avoid raw
pointers/drop/etc.

This will have the consequence of in the future `call_async` is going to
now require `Params: 'static` but that seems more-or-less inevitable at
this point.

* Go back to returning box, alas.

* Apply same treatment to lift function

Make it a closure and reduce some levels of indirection of the various
functions in play.

* Refactor `lower_params` to require less context.

Relax the bounds on the closure specified since it's immediately called
and then additionally take out parameters/captures that the closure can
carry itself.

* Don't pass extraneous `Instance` parameter

This can now be inferred from `Func`.

* Clean up some SAFETY comments

* Generalize the signature of `lift_results`

* Move `lift_results` function to `Func`

Also rename the lift/lower helpers to `with_{lift,lower}_context`

* Remove parameter from `with_lift_context`

Like `with_lower_context` this is fine to capture in the closure passed
in.

* Simplify the dynamic lifting logic

Don't call `with_lift_context` in two locations, only call it once with
a dynamic parameter.

* Refactor away the `Func::lift_results_sync` helper

* Use `with_lift_context` in `call_raw`

* Simplify a call to `Func::call_unchecked_raw`

* Ungate `with_lift_context` to fix non-cm-async build

* Fix compile (bad cherry-pick conflict resolution)

* Use `with_lower_context` in `call_raw`

Trying to unify the async/concurrent paths as much as possible.

* Move params out of `call_raw`

Let closures capture the params, no need to thread it through as an
unnecessary argument.

* Clean up unsafety in `Func::call_raw`

* Accurately mark `call_raw` itself as `unsafe`, then document why
callers should be safe.
* Don't have one large `unsafe` block in `call_raw`, instead split it up
with separate safety comments.

* Move a one-off type definition closer to its use

* Avoid intermediate allocations in dynamic calls

* Simplify a future-return site

* Deduplicate checking parameter count

* Simplify a future invocation with `?`

* Simplify a variable declaration

* Simplify some function signatures

* Remove outdated safety comment

* Refactor to not require `Params: 'static` on `call_async`

* Remove no-longer-necessary SAFETY comment

* Fix typos

* Add a fast-path with no `Box` for sync host functions

Speeds up host calls by ~20% and puts them back on parity with the
beforehand numbers Wasmtime has.

* Synchronize signatures of async/concurrent dynamic calls

Use slices for both instead of vecs for one and slices for the other.
Required some slight rejiggering. Apparently one can solve a closure
problem with another closure, then one surely has no more closure
problems.

* Fix non-cm-async build

---------

Signed-off-by: Joel Dice <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Consolidate "util" crates for testing (#10423)

* Consolidate "util" crates for testing

This commit consolidates all of the existing crates we have for testing
into a smaller set of crates. Specific

Consolidate "util" crates for testing (#10423)

* Consolidate "util" crates for testing

This commit consolidates all of the existing crates we have for testing
into a smaller set of crates. Specifically:

* `crates/misc/component-fuzz-util` => `wasmtime-test-util` + `component-fuzz`
feature
* `crates/misc/component-test-util` => `wasmtime-test-util` + `component`
feature
* `crates/wast-util` => `wasmtime-test-util` + `wast` feature
* `crates/misc/component-macro-test` => `wasmtime-test-macros`

The goal is to have one location we put various test helpers/macros
rather than our current organically-grown many locations. This is
inspired by the test failure on #10405 where I'd like to refactor more
infrastructure to a "test util" location but it wasn't clear where to
put it so I wanted to do this refactoring first.

* Remove unused file

show more ...

Expose component parameter names in types (#9618)

* Expose component parameter names in types

This commit plumbs through the parameter names since they're available
in the component for consumers o

Expose component parameter names in types (#9618)

* Expose component parameter names in types

This commit plumbs through the parameter names since they're available
in the component for consumers of the type information exposed by
Wasmtime.

Closes #9595

* Fix fuzzer build

show more ...

Remove type information from `wasmtime::component::Val` (#8062)

This commit is a large refactor of the `Val` type as used with
components to remove inherent type information present currently. The
`

Remove type information from `wasmtime::component::Val` (#8062)

This commit is a large refactor of the `Val` type as used with
components to remove inherent type information present currently. The
`Val` type is now only an AST of what a component model value looks like
and cannot fully describe the type that it is without further context.
For example enums only store the case that's being used, not the full
set of cases.

The motivation for this commit is to make it simpler to use and
construct `Val`, especially in the face of resources. Some problems
solved here are:

* With resources in play managing type information is not trivial and
can often be surprising. For example if you learn the type of a
function from a component and the instantiate the component twice the
type information is not suitable to use with either function due to
exported resources acquiring unique types on all instantiations.

* Functionally it's much easier to construct values when type
information is not required as it no longer requires probing various
pieces for type information here and there.

* API-wise there's far less for us to maintain as there's no need for a
type-per-variant of component model types. Pieces now fit much more
naturally into a `Val` shape without extra types.

* Functionally when working with `Val` there's now only one typecheck
instead of two. Previously a typecheck was performed first when a
`Val` was created and then again later when it was passed to wasm. Now
the typecheck only happens when passed to wasm.

It's worth pointing out that `Val` as-is is a pretty inefficient
representation of component model values, for example flags are stored
as a list of strings. While semantically correct this is quite
inefficient for most purposes other than "get something working". To
that extent my goal is to, in the future, add traits that enable
building a custom user-defined `Val` (of sorts), but still dynamically.
This should enable embedders to opt-in to a more efficient
representation that relies on contextual knowledge.

show more ...

Refactor `wasmtime::FuncType` to hold a handle to its registered type (#7892)

* Refactor `wasmtime::FuncType` to hold a handle to its registered type

Rather than holding a copy of the type directly

Refactor `wasmtime::FuncType` to hold a handle to its registered type (#7892)

* Refactor `wasmtime::FuncType` to hold a handle to its registered type

Rather than holding a copy of the type directly, it now holds a `RegisteredType`
which internally is

* A `VMSharedTypeIndex` pointing into the engine's types registry.
* An `Arc` handle to the engine's type registry.
* An `Arc` handle to the actual type.

The last exists only to keep it so that accessing a `wasmtime::FuncType`'s
parameters and results fast, avoiding any new locking on call hot paths.

This is helping set the stage for further types and `TypeRegistry` refactors
needed for Wasm GC.

* Update the C API for the function types refactor

prtest:full

* rustfmt

* Fix benches build

show more ...

feat: support component type introspection (#7804)

* feat: support component instance import type introspection

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: implement `impo

feat: support component type introspection (#7804)

* feat: support component instance import type introspection

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: implement `import_types` on `Linker`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: flatten `types::ComponentItem` members

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: remove `types` re-exports

Signed-off-by: Roman Volosatovs <[email protected]>

* feat: implement component type introspection

Signed-off-by: Roman Volosatovs <[email protected]>

* chore: derive `Debug` for `Field`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: return `ExactSizeIterator` for component function parameters/results

Signed-off-by: Roman Volosatovs <[email protected]>

* tests: add `introspection` test

Signed-off-by: Roman Volosatovs <[email protected]>

* chore: derive `Clone` on `ComponentItem` types

Signed-off-by: Roman Volosatovs <[email protected]>

* feat: support module and instance type introspection

Signed-off-by: Roman Volosatovs <[email protected]>

---------

Signed-off-by: Roman Volosatovs <[email protected]>

show more ...

Remove component union types (#6913)

- Bump wasm-tools deps
- Use new TypeSectionReader::into_iter_err_on_gc_types method

Update to latest wasm-tools crates (#6378)

* Update to latest wasm-tools crates

This commit pushes through the full update of the wasm-tools crates
through Wasmtime. There are two major features wh

Update to latest wasm-tools crates (#6378)

* Update to latest wasm-tools crates

This commit pushes through the full update of the wasm-tools crates
through Wasmtime. There are two major features which changed, both
related to components, which required updates in Wasmtime:

* Resource types are now implemented in wasm-tools and they're not yet
implemented in Wasmtime so I've stubbed out the integration point with
panics as reminders to come back and implement them.

* There are new validation rules about how aggregate types must be
named. This doesn't affect runtime internals at all but was done on
behalf of code generators. This did however affect a number of tests
which have to ensure that types are exported.

* Fix more tests

* Add vet entries

show more ...

Run some tests in MIRI on CI (#6332)

* Run some tests in MIRI on CI

This commit is an implementation of getting at least chunks of Wasmtime
to run in MIRI on CI. The full test suite is not possible

Run some tests in MIRI on CI (#6332)

* Run some tests in MIRI on CI

This commit is an implementation of getting at least chunks of Wasmtime
to run in MIRI on CI. The full test suite is not possible to run in MIRI
because MIRI cannot run Cranelift-produced code at runtime (aka it
doesn't support JITs). Running MIRI, however, is still quite valuable if
we can manage it because it would have trivially detected
GHSA-ch89-5g45-qwc7, our most recent security advisory. The goal of this
PR is to select a subset of the test suite to execute in CI under MIRI
and boost our confidence in the copious amount of `unsafe` code in
Wasmtime's runtime.

Under MIRI's default settings, which is to use the [Stacked
Borrows][stacked] model, much of the code in `Instance` and `VMContext`
is considered invalid. Under the optional [Tree Borrows][tree] model,
however, this same code is accepted. After some [extremely helpful
discussion][discuss] on the Rust Zulip my current conclusion is that
what we're doing is not fundamentally un-sound but we need to model it
in a different way. This PR, however, uses the Tree Borrows model for
MIRI to get something onto CI sooner rather than later, and I hope to
follow this up with something that passed Stacked Borrows. Additionally
that'll hopefully make this diff smaller and easier to digest.

Given all that, the end result of this PR is to get 131 separate unit
tests executing on CI. These unit tests largely exercise the embedding
API where wasm function compilation is not involved. Some tests compile
wasm functions but don't run them, but compiling wasm through Cranelift
in MIRI is so slow that it doesn't seem worth it at this time. This does
mean that there's a pretty big hole in MIRI's test coverage, but that's
to be expected as we're a JIT compiler after all.

To get tests working in MIRI this PR uses a number of strategies:

* When platform-specific code is involved there's now `#[cfg(miri)]` for
MIRI's version. For example there's a custom-built "mmap" just for
MIRI now. Many of these are simple noops, some are `unimplemented!()`
as they shouldn't be reached, and some are slightly nontrivial
implementations such as mmaps and trap handling (for native-to-native
function calls).

* Many test modules are simply excluded via `#![cfg(not(miri))]` at the
top of the file. This excludes the entire module's worth of tests from
MIRI. Other modules have `#[cfg_attr(miri, ignore)]` annotations to
ignore tests by default on MIRI. The latter form is used in modules
where some tests work and some don't. This means that all future test
additions will need to be effectively annotated whether they work in
MIRI or not. My hope though is that there's enough precedent in the
test suite of what to do to not cause too much burden.

* A number of locations are fixed with respect to MIRI's analysis. For
example `ComponentInstance`, the component equivalent of
`wasmtime_runtime::Instance`, was actually left out from the fix for
the CVE by accident. MIRI dutifully highlighted the issues here and
I've fixed them locally. Some locations fixed for MIRI are changed to
something that looks similar but is subtly different. For example
retaining items in a `Store<T>` is now done with a Wasmtime-specific
`StoreBox<T>` type. This is because, with MIRI's analyses, moving a
`Box<T>` invalidates all pointers derived from this `Box<T>`. We don't
want these semantics, so we effectively have a custom `Box<T>` to suit
our needs in this regard.

* Some default configuration is different under MIRI. For example most
linear memories are dynamic with no guards and no space reserved for
growth. Settings such as parallel compilation are disabled. These are
applied to make MIRI "work by default" in more places ideally. Some
tests which perform N iterations of something perform fewer iterations
on MIRI to not take quite so long.

This PR is not intended to be a one-and-done-we-never-look-at-it-again
kind of thing. Instead this is intended to lay the groundwork to
continuously run MIRI in CI to catch any soundness issues. This feels,
to me, overdue given the amount of `unsafe` code inside of Wasmtime. My
hope is that over time we can figure out how to run Wasm in MIRI but
that may take quite some time. Regardless this will be adding nontrivial
maintenance work to contributors to Wasmtime. MIRI will be run on CI for
merges, MIRI will have test failures when everything else passes,
MIRI's errors will need to be deciphered by those who have probably
never run MIRI before, things like that. Despite all this to me it seems
worth the cost at this time. Just getting this running caught two
possible soundness bugs in the component implementation that could have
had a real-world impact in the future!

[stacked]: https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md
[tree]: https://perso.crans.org/vanille/treebor/
[discuss]: https://rust-lang.zulipchat.com/#narrow/stream/269128-miri/topic/Tree.20vs.20Stacked.20Borrows.20.26.20a.20debugging.20question

* Update alignment comment

show more ...

Use floats for `wasmtime::component::Val::Float*` (#5510)

The definitions of `wasmtime::component::Val::Float{32,64}` mirrored
`wasmtime::Val::F{32,64}` by using integers as their wrapped types,
s

Use floats for `wasmtime::component::Val::Float*` (#5510)

The definitions of `wasmtime::component::Val::Float{32,64}` mirrored
`wasmtime::Val::F{32,64}` by using integers as their wrapped types,
storing the bit representation of their floating point values.
This was necessary for the core Wasm `f32`/`f64` types because Rust
floats don't have guaranteed NaN bit representations.

The component model `float32`/`float64` types require NaN
canonicalization, so we can use normal Rust `f{32,64}` instead.

Closes #5480

show more ...

Update the wasm-tools family of crates (#5310)

Most of the changes here are the updates to the component model which
includes optional URL fields in imports/exports.

Upgrade wasm-tools crates, namely the component model (#4715)

* Upgrade wasm-tools crates, namely the component model

This commit pulls in the latest versions of all of the `wasm-tools`
family

Upgrade wasm-tools crates, namely the component model (#4715)

* Upgrade wasm-tools crates, namely the component model

This commit pulls in the latest versions of all of the `wasm-tools`
family of crates. There were two major changes that happened in
`wasm-tools` in the meantime:

* bytecodealliance/wasm-tools#697 - this commit introduced a new API for
more efficiently reading binary operators from a wasm binary. The old
`Operator`-based reading was left in place, however, and continues to
be what Wasmtime uses. I hope to update Wasmtime in a future PR to use
this new API, but for now the biggest change is...

* bytecodealliance/wasm-tools#703 - this commit was a major update to
the component model AST. This commit almost entirely deals with the
fallout of this change.

The changes made to the component model were:

1. The `unit` type no longer exists. This was generally a simple change
where the `Unit` case in a few different locations were all removed.
2. The `expected` type was renamed to `result`. This similarly was
relatively lightweight and mostly just a renaming on the surface. I
took this opportunity to rename `val::Result` to `val::ResultVal` and
`types::Result` to `types::ResultType` to avoid clashing with the
standard library types. The `Option`-based types were handled with
this as well.
3. The payload type of `variant` and `result` types are now optional.
This affected many locations that calculate flat type
representations, ABI information, etc. The `#[derive(ComponentType)]`
macro now specifically handles Rust-defined `enum` types which have
no payload to the equivalent in the component model.
4. Functions can now return multiple parameters. This changed the
signature of invoking component functions because the return value is
now bound by `ComponentNamedList` (renamed from `ComponentParams`).
This had a large effect in the tests, fuzz test case generation, etc.
5. Function types with 2-or-more parameters/results must uniquely name
all parameters/results. This mostly affected the text format used
throughout the tests.

I haven't added specifically new tests for multi-return but I changed a
number of tests to use it. Additionally I've updated the fuzzers to all
exercise multi-return as well so I think we should get some good
coverage with that.

* Update version numbers

* Use crates.io

show more ...

implement fuzzing for component types (#4537)

This addresses #4307.

For the static API we generate 100 arbitrary test cases at build time, each of
which includes 0-5 parameter types, a result ty

implement fuzzing for component types (#4537)

This addresses #4307.

For the static API we generate 100 arbitrary test cases at build time, each of
which includes 0-5 parameter types, a result type, and a WAT fragment containing
an imported function and an exported function. The exported function calls the
imported function, which is implemented by the host. At runtime, the fuzz test
selects a test case at random and feeds it zero or more sets of arbitrary
parameters and results, checking that values which flow host-to-guest and
guest-to-host make the transition unchanged.

The fuzz test for the dynamic API follows a similar pattern, the only difference
being that test cases are generated at runtime.

Signed-off-by: Joel Dice <[email protected]>

show more ...

support dynamic function calls in component model (#4442)

* support dynamic function calls in component model

This addresses #4310, introducing a new `component::values::Val` type for
representi

support dynamic function calls in component model (#4442)

* support dynamic function calls in component model

This addresses #4310, introducing a new `component::values::Val` type for
representing component values dynamically, as well as `component::types::Type`
for representing the corresponding interface types. It also adds a `call` method
to `component::func::Func`, which takes a slice of `Val`s as parameters and
returns a `Result<Val>` representing the result.

Note that I've moved `post_return` and `call_raw` from `TypedFunc` to `Func`
since there was nothing specific to `TypedFunc` about them, and I wanted to
reuse them. The code in both is unchanged beyond the trivial tweaks to make
them fit in their new home.

Signed-off-by: Joel Dice <[email protected]>

* order variants and match cases more consistently

Signed-off-by: Joel Dice <[email protected]>

* implement lift for String, Box<str>, etc.

This also removes the redundant `store` parameter from `Type::load`.

Signed-off-by: Joel Dice <[email protected]>

* implement code review feedback

This fixes a few issues:

- Bad offset calculation when lowering
- Missing variant padding
- Style issues regarding `types::Handle`
- Missed opportunities to reuse `Lift` and `Lower` impls

It also adds forwarding `Lift` impls for `Box<[T]>`, `Vec<T>`, etc.

Signed-off-by: Joel Dice <[email protected]>

* move `new_*` methods to specific `types` structs

Per review feedback, I've moved `Type::new_record` to `Record::new_val` and
added a `Type::unwrap_record` method; likewise for the other kinds of types.

Signed-off-by: Joel Dice <[email protected]>

* make tuple, option, and expected type comparisons recursive

These types should compare as equal across component boundaries as long as their
type parameters are equal.

Signed-off-by: Joel Dice <[email protected]>

* improve error diagnostic in `Type::check`

We now distinguish between more failure cases to provide an informative error
message.

Signed-off-by: Joel Dice <[email protected]>

* address review feedback

- Remove `WasmStr::to_str_from_memory` and `WasmList::get_from_memory`
- add `try_new` methods to various `values` types
- avoid using `ExactSizeIterator::len` where we can't trust it
- fix over-constrained bounds on forwarded `ComponentType` impls

Signed-off-by: Joel Dice <[email protected]>

* rearrange code per review feedback

- Move functions from `types` to `values` module so we can make certain struct fields private
- Rename `try_new` to just `new`

Signed-off-by: Joel Dice <[email protected]>

* remove special-case equality test for tuples, options, and expecteds

Instead, I've added a FIXME comment and will open an issue to do recursive
structural equality testing.

Signed-off-by: Joel Dice <[email protected]>

show more ...