Add support for map type (#12216)

* Add support for map type

Signed-off-by: Yordis Prieto <[email protected]>

* Add Map and MapEntry classes to support key/value pairs in component model

Th

Add support for map type (#12216)

* Add support for map type

Signed-off-by: Yordis Prieto <[email protected]>

* Add Map and MapEntry classes to support key/value pairs in component model

This commit introduces the Map and MapEntry classes, enabling the representation of map values in the component model. The Map class allows for the creation and iteration of key/value pairs, enhancing the functionality of the wasmtime component API. Additionally, the .gitignore file is updated to exclude build artifacts from the crates/c-api directory.

* Add wasm_component_model_map configuration support

* Format code

* Format C code

* Enhance component model to support HashMap<K, V> type

This commit introduces support for HashMap<K, V> in the component model, allowing maps to be represented as list<tuple<K, V>> in the canonical ABI. It includes implementations for the ComponentType, Lower, and Lift traits for HashMap, enabling type checking, lowering to flat representations, and lifting from memory. Additionally, the maximum depth for type generation in the fuzzing utility is updated to accommodate the new map type.

* Refactor component configuration to introduce map support

This commit removes the previous wasm features configuration and adds new functions for creating a map-configured engine. The `map_config` and `map_engine` functions are introduced to facilitate the use of the component model with maps in tests, ensuring that the engine is properly configured for map types in the component model.

* Add new WAST test for map types and remove map type definitions from existing tests

This commit introduces a new WAST test file specifically for testing various map types in the component model. Additionally, it removes the redundant map type definitions from the existing types.wast file to streamline the test suite.

* Update component fuzzing and dynamic tests to replace call_and_post_return with call

* Format code

* Refactor HashMap usage in typed.rs to use wasmtime_environ collections

* Fix HashMap initialization and insertion to handle potential errors in typed.rs

* Refactor HashMap handling in typed.rs to use lower_map_iter for improved iteration and memory management. Introduce new implementations for ComponentType, Lower, and Lift traits for std::collections::HashMap, enhancing support for map types in the component model.

* Fix map adapter trampoline compilation and alignment bugs

The translate_map function had two categories of bugs preventing map
adapter trampolines from working:

1. Wasm stack discipline: local_set_new_tmp emits LocalSet which pops
from the stack, but was called when the stack was empty (to
"pre-allocate" locals). Fixed by computing values first, then
calling local_set_new_tmp to consume them—matching translate_list's
pattern. Also removed an erroneous LocalTee that left an orphan
value on the stack. Affected: src_byte_len, dst_byte_len,
cur_src_ptr, cur_dst_ptr.

2. Pointer advancement: after value translation, the pointer still
points at the value start. The code only advanced by trailing
padding instead of value_size + trailing_padding, causing every
loop iteration to re-read the same memory.

Also fixes entry layout to use proper record alignment rules (entry
align = max(key_align, value_align), value at aligned offset).

* Refactor map entry layout calculations to use canonical ABI

* Remove unnecessary clone of map pairs during lowering

Val::Map already holds Vec<(Val, Val)> which derefs to &[(Val, Val)],
matching lower_map's signature directly. The intermediate Vec allocation
and deep clone of every key/value pair was redundant.

* Deduplicate map lift logic between HashMap implementations

* Deduplicate list and map sequence translation scaffolding

* Fix cargo fmt formatting issues

* Deduplicate map typecheck logic

* Deduplicate map lowering with linear_lower_map_to_flat and linear_lower_map_to_memory helpers

* Clean up lift_try_map: use drop, move TryHashMap import to module scope

* Fix CI: arbtest overflow and no-std HashMap lift_map

- component_fuzz: use saturating_sub in generate_hashable_key to prevent
underflow when fuel is 0 and Enum variant is chosen
- typed: remove incorrect ? operators in lift_map for hashbrown::HashMap
(with_capacity and insert don't return Result)

* Store map tuple layout in TypeMap

Compute map entry ABI and value offsets once during type building, and reuse that metadata in runtime map lift/lower paths instead of recalculating tuple layout at each call site.

* Refactor map ABI argument passing

Bundle map lift/lower layout and type metadata into a small MapAbi32 helper so map helper calls stay concise without changing behavior.

* Fix CI: enable component_model_map in fuzzing and handle map in arbitrary_val

The fuzzer's component_api oracle was generating map types but the engine
didn't have the map feature enabled, and arbitrary_val had no arm for
Type::Map. Enable component_model_map in the store helper (matching how
component_model_async is forced on) and implement arbitrary value generation
for map types.

---------

Signed-off-by: Yordis Prieto <[email protected]>

show more ...

Minimal implementation of fixed-length lists to enable wit-bindgen runtime tests (#10619)

* necessary parts to make the wit-bindgen test pass

* post-merge fixes

* fix clippy

* inline type computa

Minimal implementation of fixed-length lists to enable wit-bindgen runtime tests (#10619)

* necessary parts to make the wit-bindgen test pass

* post-merge fixes

* fix clippy

* inline type computations

* link missing functionality to an issue

* undo unintended removal of blank line

* fix logic mistake, enable feature in wasmtime

* optimized conversion

* add offset

* unify on name and implement wast option

* cargo fmt and beautify fixed_size_list_type

* add multi memory as it is needed by the test runner

* standardize on the official name (component model standard) "fixed-length lists"

changing wasm-tools (parser etc. is a different task)

* More instances of fixed size corrected to fixed length

show more ...

"Downgrade" threads support to tier 2, disable fuzzing (#12036)

* "Downgrade" threads support to tier 2, disable fuzzing

This commit is borne out of a fuzz bug that was opened recently. The
fuzz bu

"Downgrade" threads support to tier 2, disable fuzzing (#12036)

* "Downgrade" threads support to tier 2, disable fuzzing

This commit is borne out of a fuzz bug that was opened recently. The
fuzz bug specifically has to do with fallout from #12022, specifically
`SharedMemory` being used to allocated instead of `Memory`. In this
situation the resource limiter is no longer consulted meaning that
shared memories bypass this and aren't caught by OOM checks. This is
currently by design because `SharedMemory` instances don't know which
resource limiter to hook into per-store.

More generally though the implementation of wasm threads, while workable
in Wasmtime, has a number of known relatively large deficiencies. These
were not resolved prior to ungating the wasm proposal (that's on me) but
nevertheless the quality of implementation is not quite up to "tier 1
par" with the rest of what Wasmtime offers. Given this the threads
proposal is now downgraded to tier 2. To help minimize the impact of
this the wasm proposal is left enabled-by-default, but creation of a
`SharedMemory` in the Rust API requires opting-in via a new
`Config::shared_memory` method.

This commit shuffles around some documentation of wasm proposals to
split it into tier 1/2/3 instead of on/off-by-default and then adds a
column for whether the proposal is on-by-default.

* clangformat

* Fix tests

* Add tests for failed creation

Fix an issue where defined shared memories weren't gated

* Sync disabled threads stub

* Fix another test

prtest:full

* Fix fuzzing tests

* Fix dwarf tests

show more ...

Cooperative Multithreading (#11751)

* Initial work

* Almost compiling

* Partially working

* Cleanup

* Fix merge

* Cancellation and suspension refactoring

* Remove printlns

* Test with several

Cooperative Multithreading (#11751)

* Initial work

* Almost compiling

* Partially working

* Cleanup

* Fix merge

* Cancellation and suspension refactoring

* Remove printlns

* Test with several threads

* More testing

* Cancellation

* Fix cancellation for explicit suspends

* Finish cancellation test

* Store threads in the instance table

* Deletion almost there

* Tests all pass

* Tighten up task deletion

* Set thread state correctly

* Store pairs of thread and task ids

* Remove lift abi members

* Cleanup unnecessary change

* More cleanup

* Cleanup

* Revert cargo changes

* Revert cargo changes

* Comments

* Comments on test

* Update comments

* Update comments

* Add space that was removed in an earlier commit

* Cleanup

* Delete threads from the instance table

* Revert cargo file

* Remove dead code

* Clippy changes

* Clippy

* Revert unnecessary changes

* Revert unnecessary changes

* Revert unnecessary changes

* Revert unnecessary changes

* Revert unnecessary changes

* Review comments

* Review feedback

* Make thread IDs per-component-instance

* Fix config

* Tighten up completion

* Clippy

* Trigger full PR test

* Move funcref table reading

* Remove unused import

* Formatting

* Rename RemoveOnDrop

* Review feedback

* Review feedback

* Move start thread closure

* Review feedback

* Review feedback

* Correct feature for import

* Review feedback

* Disable failing tests

* Enable fixed tests

* Review feedback

* Readd tests

* Ignore task deletion test with Miri

show more ...

Exceptions: add basic support for fuzzing. (#11511)

* Enable use of exceptions in our fuzz module generator.

This patch was provided by Alex in #11505; it is thus

Co-authored-by: Alex Crichton <al

Exceptions: add basic support for fuzzing. (#11511)

* Enable use of exceptions in our fuzz module generator.

This patch was provided by Alex in #11505; it is thus

Co-authored-by: Alex Crichton <[email protected]>

* Add ExnRef comparison to differential result checking.

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Initial component model and GC support in fused adapters (#11020)

* Initial component model and GC support in fused adapters

This lays down the initial infrastructure for support for GC in our fuse

Initial component model and GC support in fused adapters (#11020)

* Initial component model and GC support in fused adapters

This lays down the initial infrastructure for support for GC in our fused
adapters for the component model. We keep track of whether each lifted/lowered
function wants args/results as GC values or in linear memory. We additionally
plumb through the core function types of the functions being lifted and lowered
for (eventual) use with GC adapters.

Ultimately, this commit is enough to fuse together lifted and lowered functions
where one or both are using the GC variant of the canonical ABI. Attempting to
actually pass arguments will hit `todo!()`s. The work of implementing those
`todo!()`s is left to future commits.

* Address review feedback

* More review feedback

* Fix wasmtime-fuzzing build

* Fix the `factc` example program

* cargo fmt

* Fix fact-valid-module fuzzer

show more ...

Add plumbing for `error-context` component model feature (#10807)

This commit adds plumbing in locations for the `error-context` feature
of the component model. This is porting some minor changes fr

Add plumbing for `error-context` component model feature (#10807)

This commit adds plumbing in locations for the `error-context` feature
of the component model. This is porting some minor changes from the
wasip3-prototyping repository back to the main repo.

show more ...

Log module "DNA" during fuzzing (#10612)

This commit updates the fuzzing infrastructure of the `wasmtime-fuzzing`
crate to record the "DNA string" of a module used to generate a module
in a `*.dna`

Log module "DNA" during fuzzing (#10612)

This commit updates the fuzzing infrastructure of the `wasmtime-fuzzing`
crate to record the "DNA string" of a module used to generate a module
in a `*.dna` file. This is accompanied with a `*.json` file to pass to
`wasm-tools smith --config`. The end result is that it should be
possible now to more easily reproduce a module generation outside of
Wasmtime itself when reproducing bugs and such.

Creation of these files is gated on the debug log level in a similar
manner to creation of normal wasm files is gated on the debug log level
too.

show more ...

Update wasm spec test suite, add exception feature flags (#10553)

* Update wasm spec test suite, add exception feature flags

This commit performs an update of the spec test suite submodule to the
n

Update wasm spec test suite, add exception feature flags (#10553)

* Update wasm spec test suite, add exception feature flags

This commit performs an update of the spec test suite submodule to the
next-to-latest commit. The latest commit will require updating the
`wast` dependency which isn't published yet.

This update brings in the `wasm-3.0` folder of tests since it's been
awhile since the last update. That update notably means that the
exception-handling proposal is mixed in with all the others with various
tests. Getting tests as flagged as passing or failing as a result was
unexpectedly difficult.

The solution I ended up settling on was to preemptively implement some
infrastructure for the exceptions proposal:

* `wasmtime::Config` methods
* `wasmtime` CLI flags
* integration with wast testing
* various updates to `should_fail`

It turns out we can run a few tests with the exception proposal, notably
due to tags being implemented for stack switching. That meant that this
couldn't blanket ignore the exceptions proposal and say it's expected to
fail. Instead the proposal is said "this passes!" and tests are
individually listed as "this is expected to fail".

This then required changing an `unsupported!` panic to plumbing errors
around to avoid actually implementing the exceptions proposal here.

* Review comments

show more ...

Update wasm-tools crates to 1.228.0 (#10501)

* Update wasm-tools crates to 1.228.0

* Add missing canonical builtins

* Add new component model feature flags

* Vet wasm-tool dependencies using wild

Update wasm-tools crates to 1.228.0 (#10501)

* Update wasm-tools crates to 1.228.0

* Add missing canonical builtins

* Add new component model feature flags

* Vet wasm-tool dependencies using wildcard

* Fix tests hidden behind new CM features

Several CM-async tests were failing due to upstream changes to the
canonical builtins. Here we add some new Wasmtime flags for enabling
these new features, `wasm_component_model_async_builtins` and
`wasm_component_model_async_stackful`, and propagate that far enough to
enable the features for these tests. This also involved removing all the
error context details from these builtins.

Co-authored-by: Alex Crichton <[email protected]>

* Add some additional plumbing for fuzz configuration

prtest:full

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Enable the GC proposal during general fuzzing (#10332)

* Enable the GC proposal during general fuzzing

This allows us to fuzz Wasm GC in our fuzz targets that use the common
config-generation infra

Enable the GC proposal during general fuzzing (#10332)

* Enable the GC proposal during general fuzzing

This allows us to fuzz Wasm GC in our fuzz targets that use the common
config-generation infrastructure, such as the differential fuzz target.

Fixes #10328

* Make handling of non-deterministic errors more robust in differential fuzzer

* remove logging from functions that can be called from signal handlers

show more ...

chore: fix some typos in comments (#10309)

Signed-off-by: shenpengfeng <[email protected]>

Update wasm-tools crates to 226 (#10235)

* Update wasm-tools crates to 226

This commit updates the wasm-tools crates to the 226 track of versions.
The most notable change here is the removal and re

Update wasm-tools crates to 226 (#10235)

* Update wasm-tools crates to 226

This commit updates the wasm-tools crates to the 226 track of versions.
The most notable change here is the removal and refactoring within
wasm-tools of support for multiple and named returns in the component
model along with the support for 33+ flags in the `flags` type. This has
been gated for some time now and it's just now being removed now that
it's been awhile an no issues have been raised.

prtest:full

* Fix some issues from CI

* Fix benchmarks

* Switch to crates.io-based crates

show more ...

add component-model-async/lift.wast test (#10083)

This is another piece of #9582 which I'm splitting out to make review easier.

This test includes two components: one which exports a function using

add component-model-async/lift.wast test (#10083)

This is another piece of #9582 which I'm splitting out to make review easier.

This test includes two components: one which exports a function using the
async-with-callback ABI, and another which uses the async-without-callback ABI.
It doesn't actually instantiate or run either component yet.

The rest of the changes fill in some TODOs to make the test pass.

Signed-off-by: Joel Dice <[email protected]>

show more ...

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint levels. This new lint annotation will
silence a lint but will itself cause a lint if it doesn't actually
silence anything. This is quite useful to ensure that annotations don't
get stale over time.

Another feature is the ability to use a `reason` directive on the
attribute with a string explaining why the attribute is there. This
string is then rendered in compiler messages if a warning or error
happens.

This commit migrates applies a few changes across the workspace:

* Some `#[allow]` are changed to `#[expect]` with a `reason`.
* Some `#[allow]` have a `reason` added if the lint conditionally fires
(mostly related to macros).
* Some `#[allow]` are removed since the lint doesn't actually fire.
* The workspace configures `clippy::allow_attributes_without_reason = 'warn'`
as a "ratchet" to prevent future regressions.
* Many crates are annotated to allow `allow_attributes_without_reason`
during this transitionary period.

The end-state is that all crates should use
`#[expect(..., reason = "...")]` for any lint that unconditionally fires
but is expected. The `#[allow(..., reason = "...")]` lint should be used
for conditionally firing lints, primarily in macro-related code.
The `allow_attributes_without_reason = 'warn'` level is intended to be
permanent but the transitionary
`#[expect(clippy::allow_attributes_without_reason)]` crate annotations
to go away over time.

* Fix adapter build

prtest:full

* Fix one-core build of icache coherence

* Use `allow` for missing_docs

Work around rust-lang/rust#130021 which was fixed in Rust 1.83 and isn't
fixed for our MSRV at this time.

* More MSRV compat

show more ...

Update extended-const fuzzing (#9605)

* Remove the one-off configuration option for this which now duplicates
what's in `wasm-smith`.
* Reach whether or not the feature is enabled from `wasm-smith

Update extended-const fuzzing (#9605)

* Remove the one-off configuration option for this which now duplicates
what's in `wasm-smith`.
* Reach whether or not the feature is enabled from `wasm-smith`.
* Update docs of wasm proposals for some recent changes (e.g.
`extended-const` is fuzzed.

show more ...

Improve fuzzing of `*.wast` tests (#9587)

* Improve fuzzing of `*.wast` tests

Currently we have a fuzzer which is tasked with running `*.wast` tests
with fuzz-generated configurations. This asserts

Improve fuzzing of `*.wast` tests (#9587)

* Improve fuzzing of `*.wast` tests

Currently we have a fuzzer which is tasked with running `*.wast` tests
with fuzz-generated configurations. This asserts that we at least
satisfy all basic wasm semantics regardless of how various knobs in
`Config` are turned (modulo limits to resources). The current fuzzing
though is not comprehensive in that it doesn't include all the spec
tests that we pass from all proposals. This runs the risk of we don't
actually fuzz anything until the spec tests are merged upstream, which
can take a significant amount of time.

This commit refactors the `*.wast`-management infrastructure to share
test discovery and feature calculation between `tests/wast.rs` and
fuzzing. This new support crate centralizes limits and discovery for
both to use. Additionally fuzzing is updated to no longer throw out test
cases if configuration isn't applicable but instead clamp configuration
to the minimum required values (e.g. features + resource limits). This
means that we should now be fuzzing all spec tests that pass in all
configurations.

This new fuzzer discovered a few minor issues with the GC proposal
implementation, for example, such as:

* Some instructions were translated using trapping methods directly on
`FunctionBuilder` rather than `FuncEnvironment` meaning they didn't
properly handle `signals-based-traps` configuration.

* Fuel handling for `return_call_ref` wasn't correct because it was
accidentally omitted from the list of return-call instructions that
need special treatment.

* Add some manifest metadata

show more ...

Don't use `MemoryStyle` for heap base pointer relocations (#9569)

* Don't use `MemoryStyle` for heap base pointer relocations

Instead add a helper method to `Memory` which indicates whether the bas

Don't use `MemoryStyle` for heap base pointer relocations (#9569)

* Don't use `MemoryStyle` for heap base pointer relocations

Instead add a helper method to `Memory` which indicates whether the base
pointer of memory can move. Use this and plumb the result around to the
various locations that require it. This improves the `readonly`
application of the base pointer in Cranelift by having the optimization
kick in in more scenarios. It additionally fixes combining shared linear
memories with 64-bit addressing or a page size of 1 by adding a runtime
check before relocation of a linear memory that it's allowed to do so.

A few codegen tests are added to ensure that `readonly` is applied where
it wasn't previously and additionally a new `*.wast` test was added with
the cross product of various features of linear memories and some basic
tests to ensure that the memories all work as expected.

This refactoring fixes two preexisting issues about `shared` memories
requiring a "static" memory style. The restriction is now based on
whether the pointer can relocate or not and that's upheld via the new
trait method added here.

To account for these bug fixes the fuzzers have been updated to allow
mixing the `threads` proposal with `memory64` or `custom-page-sizes`.

Closes #4267
Closes #9523

* Optionally increase the allocation size for dynamic memories

This code will be short-lived due to scheduled future refactorings but
the idea is that when a "dynamic" memory is chosen the minimum size of
the allocation needs to be at least `tunables.memory_reservation` to fit
the constraints of the rest of the system, so be sure to factor that
in.

show more ...

Disable fuzzing both custom-page-sizes and threads (#9524)

This is an attempt to avoid #9523 in the fuzzers while that issue is
fixed independently. Fuzzing is pretty limited right now so the hope i

Disable fuzzing both custom-page-sizes and threads (#9524)

This is an attempt to avoid #9523 in the fuzzers while that issue is
fixed independently. Fuzzing is pretty limited right now so the hope is
to let fuzzers find other issues while the refactoring here progresses.

show more ...

Add general fuzzing support for custom page sizes (#9462)

* Add general fuzzing support for custom page sizes

* Add custom-page-sizes as an expected feature for module generation

Centralize wasm-features-in-fuzzing a bit more (#9450)

Move everything into the same location about what features are enabled,
which are disabled, and such. Also statically enumerate all the feature

Centralize wasm-features-in-fuzzing a bit more (#9450)

Move everything into the same location about what features are enabled,
which are disabled, and such. Also statically enumerate all the features
that wasm-smith enables for us and acknowledge that here too.

show more ...

Update the wasm-tools family of crates (#7921)

* Update the wasm-tools family of crates

Pulling in some updates to improve how WIT is managed in this
repository. No changes just yet, however, just

Update the wasm-tools family of crates (#7921)

* Update the wasm-tools family of crates

Pulling in some updates to improve how WIT is managed in this
repository. No changes just yet, however, just pulling in the updates
first.

* Fix tests

* Fix fuzzer build

show more ...

Update the wasm-tools family of crates (#7764)

* Update the wasm-tools family of crates

Brings in support for validating gc instructions, but they're all left
disabled for now.

* Update fuzz test

Update the wasm-tools family of crates (#7764)

* Update the wasm-tools family of crates

Brings in support for validating gc instructions, but they're all left
disabled for now.

* Update fuzz test case generation

* More test fixes, remove stray files

* More test fixes

* Rebase

show more ...

Run differential fuzzing in non-trapping mode 90% of the time (#5385)

Various improvements to differential fuzzing (#4845)

* Improve wasmi differential fuzzer

* Support modules with a `start` function
* Implement trap-matching to ensure that wasmi and Wasmtime bot

Various improvements to differential fuzzing (#4845)

* Improve wasmi differential fuzzer

* Support modules with a `start` function
* Implement trap-matching to ensure that wasmi and Wasmtime both report
the same flavor of trap.

* Support differential fuzzing where no engines match

Locally I was attempting to run against just one wasm engine with
`ALLOWED_ENGINES=wasmi` but the fuzzer quickly panicked because the
generated test case didn't match wasmi's configuration. This commit
updates engine-selection in the differential fuzzer to return `None` if
no engine is applicable, throwing out the test case. This won't be hit
at all with oss-fuzz-based runs but for local runs it'll be useful to
have.

* Improve proposal support in differential fuzzer

* De-prioritize unstable wasm proposals such as multi-memory and
memory64 by making them more unlikely with `Unstructured::ratio`.
* Allow fuzzing multi-table (reference types) and multi-memory by
avoiding setting their maximums to 1 in `set_differential_config`.
* Update selection of the pooling strategy to unconditionally support
the selected module config rather than the other way around.

* Improve handling of traps in differential fuzzing

This commit fixes an issue found via local fuzzing where engines were
reporting different results but the underlying reason for this was that
one engine was hitting stack overflow before the other. To fix the
underlying issue I updated the execution to check for stack overflow
and, if hit, it discards the entire fuzz test case from then on.

The rationale behind this is that each engine can have unique limits for
stack overflow. One test case I was looking at for example would stack
overflow at less than 1000 frames with epoch interruption enabled but
would stack overflow at more than 1000 frames with it disabled. This
means that the state after the trap started to diverge and it looked
like the engines produced different results.

While I was at it I also improved the "function call returned a trap"
case to compare traps to make sure the same trap reason popped out.

* Fix fuzzer tests

show more ...