"Downgrade" threads support to tier 2, disable fuzzing (#12036)

* "Downgrade" threads support to tier 2, disable fuzzing

This commit is borne out of a fuzz bug that was opened recently. The
fuzz bu

"Downgrade" threads support to tier 2, disable fuzzing (#12036)

* "Downgrade" threads support to tier 2, disable fuzzing

This commit is borne out of a fuzz bug that was opened recently. The
fuzz bug specifically has to do with fallout from #12022, specifically
`SharedMemory` being used to allocated instead of `Memory`. In this
situation the resource limiter is no longer consulted meaning that
shared memories bypass this and aren't caught by OOM checks. This is
currently by design because `SharedMemory` instances don't know which
resource limiter to hook into per-store.

More generally though the implementation of wasm threads, while workable
in Wasmtime, has a number of known relatively large deficiencies. These
were not resolved prior to ungating the wasm proposal (that's on me) but
nevertheless the quality of implementation is not quite up to "tier 1
par" with the rest of what Wasmtime offers. Given this the threads
proposal is now downgraded to tier 2. To help minimize the impact of
this the wasm proposal is left enabled-by-default, but creation of a
`SharedMemory` in the Rust API requires opting-in via a new
`Config::shared_memory` method.

This commit shuffles around some documentation of wasm proposals to
split it into tier 1/2/3 instead of on/off-by-default and then adds a
column for whether the proposal is on-by-default.

* clangformat

* Fix tests

* Add tests for failed creation

Fix an issue where defined shared memories weren't gated

* Sync disabled threads stub

* Fix another test

prtest:full

* Fix fuzzing tests

* Fix dwarf tests

show more ...

Add a configuration knob for `PAGEMAP_SCAN` (#11433)

* Add a configuration knob for `PAGEMAP_SCAN`

This commit adds `PoolingAlloationConfig::pagemap_scan` and additionally
adds `-Opooling-pagemap-s

Add a configuration knob for `PAGEMAP_SCAN` (#11433)

* Add a configuration knob for `PAGEMAP_SCAN`

This commit adds `PoolingAlloationConfig::pagemap_scan` and additionally
adds `-Opooling-pagemap-scan` to configure on the CLI. This is the same
tri-state configuration option as `MpkEnable` so that enum was renamed
to just `Enabled` and repurposed for both options.

This then additionally turns the option off-by-default instead of the
previous on-by-default-if-able-to to enable more slowly rolling out this
feature.

* Fix broken test

show more ...

Simplify some conditional testing logic for Winch (#11166)

* Simplify some conditional testing logic for Winch

Run all tests on all platforms in `#[wasmtime_test]` for Winch without
`#[cfg]` and u

Simplify some conditional testing logic for Winch (#11166)

* Simplify some conditional testing logic for Winch

Run all tests on all platforms in `#[wasmtime_test]` for Winch without
`#[cfg]` and use the result of `should_fail` to determine whether it
should fail or not (e.g. still expect failure on riscv64/s390x/etc).

* Ignore non-Pulley tests on Miri

show more ...

winch(aarch64) Run integration tests in winch-aarch64 (#11031)

* winch(aarch64) Run integration tests in winch-aarch64

This commit closes
https://github.com/bytecodealliance/wasmtime/issues/8321

W

winch(aarch64) Run integration tests in winch-aarch64 (#11031)

* winch(aarch64) Run integration tests in winch-aarch64

This commit closes
https://github.com/bytecodealliance/wasmtime/issues/8321

With https://github.com/bytecodealliance/wasmtime/pull/11013, Winch
passes all spec tests for Core Wasm on Aarch64.

* Run `cargo fmt`

* Add an `integration` option to `TestConfig`

`wast::TestConfig` is shared across spec tests and integration tests,
this commits adds a flag to identify if the config is for integration
test purposes making it easier to contextually decide if tests are
expected to fail or pass depending on the Wasm configuration options.

* Use `spec_test` instead of `integration`

To better differentiate spec tests from integration tests

* Ensure only aarch64 / x64 when testing Winch

* Revert directive for typed_v128 test

Put back `#[wasmtime_test(wasm_features(simd),
strategies(not(Winch)))]`; the reason why this tests shouldn't run
with Winch is because it requires AVX, but we can have a generic rule
at the test config level since not all SIMD based tests require AVX,
it really depends on which SIMD instructions are used.

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Run CI tests through AddressSanitizer (#10537)

This is similar to running tests in Valgrind (which we should perhaps
also do...) but can be useful for catching use-after-free style bugs
faster than

Run CI tests through AddressSanitizer (#10537)

This is similar to running tests in Valgrind (which we should perhaps
also do...) but can be useful for catching use-after-free style bugs
faster than when a process crashes. Given the unsafe nature of Wasmtime
this is something we should have probably enabled awhile back but
otherwise so long as it doesn't take too long to run on CI seems like an
easy win of a boost-of-confidence.

prtest:asan

show more ...

Enable the `unsafe-op-in-unsafe-fn` lint (#10559)

* Enable the `unsafe-op-in-unsafe-fn` lint

This commit enables the `unsafe-op-in-unsafe-fn` lint in rustc for the
entire workspace. This lint will

Enable the `unsafe-op-in-unsafe-fn` lint (#10559)

* Enable the `unsafe-op-in-unsafe-fn` lint

This commit enables the `unsafe-op-in-unsafe-fn` lint in rustc for the
entire workspace. This lint will be warn-by-default in the 2024 edition
so this is intended to smooth the future migration to the new edition.

Many `unsafe` blocks were added in places the lint warned about, with
two major exceptions. The `wasmtime` and `wasmtime-c-api` crates simply
expect this lint to fire and effectively disable the lint. They're too
big at this time to do through this PR. My hope is that one day in the
future they'll be migrated, but more realistically that probably won't
happen so these crates just won't benefit from this lint.

* Fix nostd fiber build

prtest:full

* Fix build on Windows

* Fix asan build

show more ...

s390x: Skip memory allocation failure tests (#10530)

These tests assume that a very large allocation will fail due to
exceeding address space limits. This is not the case on s390x,
where the full 6

s390x: Skip memory allocation failure tests (#10530)

These tests assume that a very large allocation will fail due to
exceeding address space limits. This is not the case on s390x,
where the full 64-bit address space is available for allocation.
(Of course, actually using that much memory would still fail at
some point, but the mere allocation may not, depending on the
kernel's memory overcommit handling configuration.)

Note that running the tests under qemu will cause the expected
failure, which is why this problem is not noticed by the CI.

Simply disable these test on s390x.

show more ...

Run the full test suite on 32-bit platforms (#9837)

* Run the full test suite on 32-bit platforms

This commit switches to running the full test suite in its entirety
(`./ci/run-tests.sh`) on 32-bit

Run the full test suite on 32-bit platforms (#9837)

* Run the full test suite on 32-bit platforms

This commit switches to running the full test suite in its entirety
(`./ci/run-tests.sh`) on 32-bit platforms in CI in addition to 64-bit
platforms. This notably adds i686 and armv7 as architectures that are
tested in CI.

Lots of little fixes here and there were applied to a number of tests.
Many tests just don't run on 32-bit platforms or a platform without
Cranelift support, and they've been annotated as such where necessary.
Other tests were adjusted to run on all platforms a few minor bug fixes
are here as well.

prtest:full

* Fix clippy warning

* Get wasm code working by default on 32-bit

Don't require the `pulley` feature opt-in on 32-bit platforms to get
wasm code running.

* Fix dead code warning

* Fix build on armv7

* Fix test assertion on armv7

* Review comments

* Update how tests are skipped

* Change how Pulley is defaulted

Default to pulley in `build.rs` rather than in `Cargo.toml` to make it
easier to write down the condition and comment what's happening. This
means that the `pulley-interpreter` crate and pulley support in
Cranelift is always compiled in now and cannot be removed. This should
hopefully be ok though as the `pulley-interpreter` crate is still
conditionally used (meaning it can get GC'd) and the code-size of
Cranelift is not as important as the runtime itself.

* pulley: Save/restore callee-save state on traps

* Fewer clippy warnings about casts

* Use wrapping_add in `g32_addr`, fixing arm test

show more ...

Fix some off-by-one comparisons in assertions (#9763)

* Fix some off-by-one comparisons in assertions

Fuzzing found a small issue with #9687 and this commit relaxes a few
off-by-one checks to allow

Fix some off-by-one comparisons in assertions (#9763)

* Fix some off-by-one comparisons in assertions

Fuzzing found a small issue with #9687 and this commit relaxes a few
off-by-one checks to allow addressing one-byte-beyond-the-end of a
linear memory.

* Review comments

show more ...

Remove static/dynamic memories from public docs (#9545)

* Remove static/dynamic memories from public docs

This commit removes the terminology of "static" and "dynamic" memories
from the public-faci

Remove static/dynamic memories from public docs (#9545)

* Remove static/dynamic memories from public docs

This commit removes the terminology of "static" and "dynamic" memories
from the public-facing documentation of Wasmtime, notably on the
`Config` structure and its various configuration settings. The goal of
this commit is in the same vein as #9543 which is to simplify the memory
settings of Wasmtime for users in this case.

This change doesn't actually have any code changes beyond renames (and
handling now-deprecated CLI options). The goal of this commit is to
basically rewrite how we document the effect of various settings of
Wasmtime. Notably:

* `Config::static_memory_maximum_size` is now `memory_reservation`.
* `Config::static_memory_forced` is now `memory_reservation_is_maximum`.
* `Config::dynamic_memory_reserved_for_growth` is now
`memory_reservation_for_growth`.

Documentation for all of these options has been rewritten and updated to
take into account the removal of "dynamic" and "static" terminology.
Additionally more words have been written about the various effects of
each setting and how things related to wasm features such as index type
sizes and custom page sizes.

The rewritten documentation is intended to basically already match what
Wasmtime does today. I believe that all of these settings are useful in
one form or another so none have been dropped but the updated
documentation is intended to help simplify the mental model for how
they're processed internally and how they affect allocations and such.

* Fix how the setting is flipped

* Review comments

show more ...

Merge static/dynamic guard size options (#9528)

* Merge static/dynamic guard size options

This commit is the first of what will likely be a few to refactor the
memory-related configuration options

Merge static/dynamic guard size options (#9528)

* Merge static/dynamic guard size options

This commit is the first of what will likely be a few to refactor the
memory-related configuration options in Wasmtime. The end goal of these
refactorings is to fix some preexisting issues and additionally make the
configuration easier to understand for both users and implementors
alike. First on the chopping block here is to merge the
`dynamic_memory_guard_size` and `static_memory_guard_size` options into
one option. AFAIK there's not a strong reason to have separate
configuration options for these so it's hopefully simpler to have a
single `memory_guard_size` option which applies to all linear memories
equally.

I'll note that the old CLI options are preserved but are documented as
deprecated. We don't currently warn on using "deprecated options" so for
now the old options are just documented as deprecated and are otherwise
silently accepted.

* Fix compilation of C API

* Fix build of fuzzers

show more ...

Implement the table64 extension to the memory64 proposal (#9206)

This commit implements the table64 extention in both Wasmtime and
Cranelift.

Most of the work was changing a bunch of u32 values to

Implement the table64 extension to the memory64 proposal (#9206)

This commit implements the table64 extention in both Wasmtime and
Cranelift.

Most of the work was changing a bunch of u32 values to u64/usize.
The decisions were made in align with the PR #3153 which
implemented the memory64 propsal itself.

One significant change was the introduction of `IndexType`
and `Limits` which streamline and unify the handling of limits
for both memories and tables.

The spec and fuzzing tests related to table64 are re-enabled which
provides a good coverage of the feature.

show more ...

style: simplify string formatting (#9047)

* style: simplify string formatting

* fix: formatting in `benches/call.rs`

winch: Consolidate memory tests in `test/all/memory.rs` (#8936)

* winch: Consolidate memory tests in `test/all/memory.rs`

This commit updates the memory tests to use
`wasmtime_test_macros::wasmtime

winch: Consolidate memory tests in `test/all/memory.rs` (#8936)

* winch: Consolidate memory tests in `test/all/memory.rs`

This commit updates the memory tests to use
`wasmtime_test_macros::wasmtime_test` where applicable. This change
consolidates Winch's integration test suite into Wasmtime's general
test suite.

Note that this change doesn't update `wasmtime_test` to account for
pooling allocation configuration. I can either update the macro in this
PR or do it as a follow-up change once we decide to migrate
`tests/all/pooling_allocator.rs`.

* Remove `winch` mod from `tests/main.rs`

show more ...

Wasmtime: Implement the custom-page-sizes proposal (#8763)

* Wasmtime: Implement the custom-page-sizes proposal

This commit adds support for the custom-page-sizes proposal to Wasmtime:
https://gith

Wasmtime: Implement the custom-page-sizes proposal (#8763)

* Wasmtime: Implement the custom-page-sizes proposal

This commit adds support for the custom-page-sizes proposal to Wasmtime:
https://github.com/WebAssembly/custom-page-sizes

I've migrated, fixed some bugs within, and extended the `*.wast` tests for this
proposal from the `wasm-tools` repository. I intend to upstream them into the
proposal shortly.

There is a new `wasmtime::Config::wasm_custom_page_sizes_proposal` method to
enable or disable the proposal. It is disabled by default.

Our fuzzing config has been updated to turn this feature on/off as dictated by
the arbitrary input given to us from the fuzzer.

Additionally, there were getting to be so many constructors for
`wasmtime::MemoryType` that I added a builder rather than add yet another
constructor.

In general, we store the `log2(page_size)` rather than the page size
directly. This helps cut down on invalid states and properties we need to
assert.

I've also intentionally written this code such that supporting any power of two
page size (rather than just the exact values `1` and `65536` that are currently
valid) will essentially just involve updating `wasmparser`'s validation and
removing some debug asserts in Wasmtime.

* Update error string expectation

* Remove debug logging

* Use a right shift instead of a division

* fix error message expectation again

* remove page size from VMMemoryDefinition

* fix size of VMMemoryDefinition again

* Only dynamically check for `-1` sentinel for 1-byte page sizes

* Import functions that are used a few times

* Better handle overflows when rounding up to the host page size

Propagate errors instead of returning a value that is not actually a rounded up
version of the input.

Delay rounding up various config sizes until runtime instead of eagerly doing it
at config time (which isn't even guaranteed to work, so we already had to have a
backup plan to round up at runtime, since we might be cross-compiling wasm or
not have the runtime feature enabled).

* Fix some anyhow and nostd errors

* Add missing rounding up to host page size at runtime

* Add validate feature to wasmparser dep

* Add some new rounding in a few places, due to no longer rounding in config methods

* Avoid actually trying to allocate the whole address space in the `massive_64_bit_still_limited` test

The point of the test is to ensure that we hit the limiter, so just cancel the
allocation from the limiter, and otherwise avoid MIRI attempting to allocate a
bunch of memory after we hit the limiter.

* prtest:full

* Revert "Avoid actually trying to allocate the whole address space in the `massive_64_bit_still_limited` test"

This reverts commit ccfa34a78dd3d53e49a6158ca03077d42ce8bcd7.

* miri: don't attempt to allocate more than 4GiB of memory

It seems that rather than returning a null pointer from `std::alloc::alloc`,
miri will sometimes choose to simply crash the whole program.

* remove duplicate prelude import after rebasing

show more ...

Use bytes for maximum size of linear memory with pooling (#8628)

* Use bytes for maximum size of linear memory with pooling

This commit changes configuration of the pooling allocator to use a
byte-

Use bytes for maximum size of linear memory with pooling (#8628)

* Use bytes for maximum size of linear memory with pooling

This commit changes configuration of the pooling allocator to use a
byte-based unit rather than a page based unit. The previous
`PoolingAllocatorConfig::memory_pages` configuration option configures
the maximum size that a linear memory may grow to at runtime. This is an
important factor in calculation of stripes for MPK and is also a
coarse-grained knob apart from `StoreLimiter` to limit memory
consumption. This configuration option has been renamed to
`max_memory_size` and documented that it's in terms of bytes rather than
pages as before.

Additionally the documented constraint of `max_memory_size` must be
smaller than `static_memory_bound` is now additionally enforced as a
minor clean-up as part of this PR as well.

* Review comments

* Fix benchmark build

show more ...

Page align the static memory maximum size (#8630)

This fixes an accidental regression from #8616 where page alignment was
implicitly happening due to how configuration was processed but it
wasn't re

Page align the static memory maximum size (#8630)

This fixes an accidental regression from #8616 where page alignment was
implicitly happening due to how configuration was processed but it
wasn't re-added in the refactoring.

show more ...

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTF

Add support for `#![no_std]` to the `wasmtime` crate (#8533)

* Always fall back to custom platform for Wasmtime

This commit updates Wasmtime's platform support to no longer require an
opt-in `RUSTFLAGS` `--cfg` flag to be specified. With `no_std` becoming
officially supported this should provide a better onboarding experience
where the fallback custom platform is used. This will cause linker
errors if the symbols aren't implemented and searching/googling should
lead back to our docs/repo (eventually, hopefully).

* Change Wasmtime's TLS state to a single pointer

This commit updates the management of TLS to rely on just a single
pointer rather than a pair of a pointer and a `bool`. Additionally
management of the TLS state is pushed into platform-specific modules to
enable different means of managing it, namely the "custom" platform now
has a C function required to implement TLS state for Wasmtime.

* Delay conversion to `Instant` in atomic intrinsics

The `Duration` type is available in `no_std` but the `Instant` type is
not. The intention is to only support the `threads` proposal if `std` is
active but to assist with this split push the `Duration` further into
Wasmtime to avoid using a type that can't be mentioned in `no_std`.

* Gate more parts of Wasmtime on the `profiling` feature

Move `serde_json` to an optional dependency and gate the guest profiler
entirely on the `profiling` feature.

* Refactor conversion to `anyhow::Error` in `wasmtime-environ`

Have a dedicated trait for consuming `self` in addition to a
`Result`-friendly trait.

* Gate `gimli` in Wasmtime on `addr2line`

Cut down the dependency list if `addr2line` isn't enabled since then
the dependency is not used. While here additionally lift the version
requirement for `addr2line` up to the workspace level.

* Update `bindgen!` to have `no_std`-compatible output

Pull most types from Wasmtime's `__internal` module as the source of
truth.

* Use an `Option` for `gc_store` instead of `OnceCell`

No need for synchronization here when mutability is already available in
the necessary contexts.

* Enable embedder-defined host feature detection

* Add `#![no_std]` support to the `wasmtime` crate

This commit enables compiling the `runtime`, `gc`, and `component-model`
features of the `wasmtime` crate on targets that do not have `std`. This
tags the crate as `#![no_std]` and then updates everything internally to
import from `core` or `alloc` and adapt for the various idioms. This
ended up requiring some relatively extensive changes, but nothing too
too bad in the grand scheme of things.

* Require `std` for the perfmap profiling agent

prtest:full

* Fix build on wasm

* Fix windows build

* Remove unused import

* Fix Windows/Unix build without `std` feature

* Fix some doc links

* Remove unused import

* Fix build of wasi-common in isolation

* Fix no_std build on macos

* Re-fix build

* Fix standalone build of wasmtime-cli-flags

* Resolve a merge conflict

* Review comments

* Remove unused import

show more ...

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime_runtime` to `crate::runtime::vm`

* Remove dep on `wasmtime-runtime` from `wasmtime-cli`

* Move the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module

* Update labeler for merged crates

* Fix `publish verify`

prtest:full

show more ...

Use some new standard library methods from Rust 1.74, 1.75 (#8209)

* Use some `byte_add` methods from Rust 1.75

Now that we're able to, use some convenience methods from the 1.75
release of Rust.

Use some new standard library methods from Rust 1.74, 1.75 (#8209)

* Use some `byte_add` methods from Rust 1.75

Now that we're able to, use some convenience methods from the 1.75
release of Rust.

* Use `Atomic*::from_ptr` from Rust 1.75

Helps clean up some casts and clarify local intent.

show more ...

Update some CI dependencies (#7983)

* Update some CI dependencies

* Update to the latest nightly toolchain
* Update mdbook
* Update QEMU for cross-compiled testing
* Update `cargo nextest` for usag

Update some CI dependencies (#7983)

* Update some CI dependencies

* Update to the latest nightly toolchain
* Update mdbook
* Update QEMU for cross-compiled testing
* Update `cargo nextest` for usage with MIRI

prtest:full

* Remove lots of unnecessary imports

* Downgrade qemu as 8.2.1 seems to segfault

* Remove more imports

* Remove unused winch trait method

* Fix warnings about unused trait methods

* More unused imports

* More unused imports

show more ...

Fix memory initialization when offset is negative (#7559)

This commit fixes a bug in initializing memory segments of 32-bit
memories where if the offset was negative when viewed as a signed
integer

Fix memory initialization when offset is negative (#7559)

This commit fixes a bug in initializing memory segments of 32-bit
memories where if the offset was negative when viewed as a signed
integer the offset was incorrectly sign-extended to a 64-bit value
instead of zero-extended. This commit replaces an `i32`-to-`u64` cast
with an `i32`-to-`u32` cast followed by a `u32`-to-`u64` cast which
performs the zero extend.

Closes #7558

show more ...

mpk: enable more tests (#7376)

* mpk: add Wasm test

This adds a Wasmtime-level test checking that, when MPK is available on
the system, an MPK-configured memory pool obeys the same invariants as
ot

mpk: enable more tests (#7376)

* mpk: add Wasm test

This adds a Wasmtime-level test checking that, when MPK is available on
the system, an MPK-configured memory pool obeys the same invariants as
other memory pool configurations (e.g., `guards_present` for normal
allocation, `guards_present_pooling` for non-MPK pooling allocation).

It also fixes a bug. A previous commit had updated the validation logic
of `MemoryPool` to check that the memory plan's bound would be reached
before the next slot of the same stripe. But `MemoryPool::allocate` has
some "double-check" logic that was triggered by this test. It now
matches the logic in `MemoryPool::validate`.

* mpk: explicitly disable MPK on certain tests

Some Wasmtime tests rely on specific limits to the memory pool. When MPK
is enabled, these tests fail because MPK splits access to the pool
slices among different stores. This change does not yet enable MPK,
though locally I run with MPK enabled. With this commit and MPK enabled
locally, all Wasmtime tests now pass.

show more ...

Bump wasm-tools crates (#7094)

* Bump wasm-tools crates

Two major changes/reasons for this update:

* Primarily pulling in support for semicolons-in-WIT files. Semicolons are
not currently requir

Bump wasm-tools crates (#7094)

* Bump wasm-tools crates

Two major changes/reasons for this update:

* Primarily pulling in support for semicolons-in-WIT files. Semicolons are
not currently required, though, so I'll follow-up later with actual
semicolons.

* The syntax for parsing `(if ...)` was fixed in `wast`. Previously it
did not require `(then ...)` but this is required by the spec. New
spec tests require this as well. This breaks existing text format
tests which don't use `(then ...)` inside of an `(if ...)`. Most tests
were updated by hand but `embenchen_*` tests were updated by running
through the old parser to produce non-s-expression using code.

* Fix an example `*.wat`

show more ...