Update fibers to avoid no-return functions (#12928)

* Update fibers to avoid no-return functions

This commit is aimed at fixing the ASAN false positives in #12899.
Initially the fix there was to in

Update fibers to avoid no-return functions (#12928)

* Update fibers to avoid no-return functions

This commit is aimed at fixing the ASAN false positives in #12899.
Initially the fix there was to invoke some `__asan_*` intrinsics, and
I ended up finding a sort of smaller set of `__asan_*` intrinsics to
call as well. In the end what's happening though is that fibers, upon
terminating, have a few frames of Rust code on the stack before
switching off. To ASAN these frames never returned so when a stack is
subsequently reused ASAN is tricked into thinking this is buffer
overflow or use-after-free since it's stomping on frames that haven't
returned.

The fix in this commit is to avoid this style of function which doesn't
returns. Functions which don't return in Rust are easy to leak memory
from and are a hazard from a safety perspective as well (e.g. it's
unsafe to skip running destructors of stack variables). I feel we've had
better success over time with "all Rust functions always return" and so
what's what was applied here. Unlike #12899 or my thoughts on that PR
this does not have any new `__asan_*` intrinsic calls. Instead what this
does is it shuffles around responsibility for what exact piece of the
infrastructure is responsible for what. Specifically `fiber_start`
functions now actually return, meaning the `wasmtime_fiber_start` naked
function actually resumes execution, unlike before. The
`wasmtime_fiber_start` then delegates to `wasmtime_fiber_switch`
immediately to perform the final switch.

Effectively there's now only two function frames that never return, and
both of these frames are handwritten inline assembly. This means that
ASAN gets to see that all normal functions return and updates all of its
metadata accordingly. The end result is that the original issue from #12899
is fixed and this I feel is in general more robust as well.

One caveat is that the handwritten `wasmtime_fiber_start` assembly needs
to invoke a sibling `wasmtime_fiber_switch_` function. In lieu of trying
to figure out how to get PIC-vs-not calls working (e.g. static calls)
I've opted to use indirect function calls and pointers instead. This
mirrors historical changes in our fiber implementation too.

* Fix CI builds

* Fix miri

show more ...

Fix some overflows when allocating a max-size fiber stack (#12868)

* Fix some overflows when allocating a max-size fiber stack

These are some minor issues that won't actually surface in practice bu

Fix some overflows when allocating a max-size fiber stack (#12868)

* Fix some overflows when allocating a max-size fiber stack

These are some minor issues that won't actually surface in practice but
seem good to fix nonetheless. Allocating a max-size fiber stack should
fail, and it shouldn't fail with a panic or a debug assert.

* Fix CI

show more ...

wasmtime-fiber: use `wasmtime_environ::error` instead of `anyhow` (#12206)

* wasmtime-fiber: use `wasmtime_environ::error` instead of `anyhow`

* Fix publish script topological sorting

Use `naked_asm!`, delete `asm_func!` (#11405)

This deletes our home-grown `asm_func!` macro in favor of using
`#[unsafe(naked)]` functions within Wasmtime. This is needed for
fiber-related bits righ

Use `naked_asm!`, delete `asm_func!` (#11405)

This deletes our home-grown `asm_func!` macro in favor of using
`#[unsafe(naked)]` functions within Wasmtime. This is needed for
fiber-related bits right now where we need tight control over the exact
assembly of some functions. This additionally migrates s390x fiber bits
to Rust as inline assembly is now stable for s390x.

prtest:full

show more ...

Add a dummy impl of fibers for Miri (#11009)

* Add a dummy impl of fibers for Miri

This commit extends the `wasmtime-internal-fiber` crate with an
implementation for Miri. Previously this was entir

Add a dummy impl of fibers for Miri (#11009)

* Add a dummy impl of fibers for Miri

This commit extends the `wasmtime-internal-fiber` crate with an
implementation for Miri. Previously this was entirely unsupported
because fibers use inline assembly. The implementation with Miri spawns
a separate thread and keeps it in a suspended state with locks to model
a suspended stack. This technically isn't correct because TLS variables
will be wrong, but it's "correct enough" for our usage in Wasmtime. In
the end this enables running more tests in Miri which is always a good
thing, and a number of loose odds and ends were cleaned up relate to our
unsafe management of async state.

* Apply suggestions from code review

Co-authored-by: Pat Hickey <[email protected]>

---------

Co-authored-by: Pat Hickey <[email protected]>

show more ...

More clearly flag internal crates as such (#10963)

* More clearly flag internal crates as such

This commit is an attempt to more clearly flag internal crates in this
project as internal and not int

More clearly flag internal crates as such (#10963)

* More clearly flag internal crates as such

This commit is an attempt to more clearly flag internal crates in this
project as internal and not intended for external use. Specifically:

* Many crates are renamed from `wasmtime-foo` to
`wasmtime-internal-foo`.
* All of these crates now have `INTERNAL: ...` in their crates.io
description.
* All of these crates now have a warning at the top of their
documentation discouraging use.

This change is a result of rustsec/advisory-db#1999 where the goal is to
be crystal clear from a project perspective that usage of these crates
are highly discouraged and not supported. We'll still probably get such
advisories but we won't be considering them CVEs from the project itself
due to the internal nature of these crates and the discouraging
warnings.

Some concrete changes used here are:

* Inter-crate dependencies still use `wasmtime_foo` for naming and do
so with Cargo's package-renaming features.
* Crate renames are specified at the workspace level so the rename is
only in one locations and all other inherit it.
* Contribution documentation now has some brief guidelines about crate
organization.

* Update vet config

* Update checks for wasmtime-fiber

prtest:full

* Update publish script

* Another fiber rename

* Fix some doc tests

show more ...

Duplicate page size determination in `wasmtime-fiber` (#10803)

* Duplicate page size determination in `wasmtime-fiber`

Currently Wasmtime has a function `crate::runtime::vm::host_page_size`
but thi

Duplicate page size determination in `wasmtime-fiber` (#10803)

* Duplicate page size determination in `wasmtime-fiber`

Currently Wasmtime has a function `crate::runtime::vm::host_page_size`
but this isn't reachable from the `wasmtime-fiber` crate and instead tha
crate uses `rustix::param::page_size` to determine the host page size.
It looks like this usage of `rustix` is causing a panic in #10802.
Ideally `wasmtime-fiber` would be able to use the same function but the
crate separation does not currently make that feasible. For now
duplicate the logic of `wasmtime` into `wasmtime-fiber` as it's modest
enough to ensure that this does not panic.

Closes #10802

* Run full test suite in CI

prtest:full

show more ...

Remove some `#![expect(clippy::allow_attributes_without_reason)]` (#10661)

Clean up some crates by migrating from `#[allow]` to `#[expect]`
(ideally) or `#[allow]`-with-reason

Run CI tests through AddressSanitizer (#10537)

This is similar to running tests in Valgrind (which we should perhaps
also do...) but can be useful for catching use-after-free style bugs
faster than

Run CI tests through AddressSanitizer (#10537)

This is similar to running tests in Valgrind (which we should perhaps
also do...) but can be useful for catching use-after-free style bugs
faster than when a process crashes. Given the unsafe nature of Wasmtime
this is something we should have probably enabled awhile back but
otherwise so long as it doesn't take too long to run on CI seems like an
easy win of a boost-of-confidence.

prtest:asan

show more ...

Enable the `unsafe-op-in-unsafe-fn` lint (#10559)

* Enable the `unsafe-op-in-unsafe-fn` lint

This commit enables the `unsafe-op-in-unsafe-fn` lint in rustc for the
entire workspace. This lint will

Enable the `unsafe-op-in-unsafe-fn` lint (#10559)

* Enable the `unsafe-op-in-unsafe-fn` lint

This commit enables the `unsafe-op-in-unsafe-fn` lint in rustc for the
entire workspace. This lint will be warn-by-default in the 2024 edition
so this is intended to smooth the future migration to the new edition.

Many `unsafe` blocks were added in places the lint warned about, with
two major exceptions. The `wasmtime` and `wasmtime-c-api` crates simply
expect this lint to fire and effectively disable the lint. They're too
big at this time to do through this PR. My hope is that one day in the
future they'll be migrated, but more realistically that probably won't
happen so these crates just won't benefit from this lint.

* Fix nostd fiber build

prtest:full

* Fix build on Windows

* Fix asan build

show more ...

Wasmtime: generalize `async_stack_zeroing` knob to cover initialization (#10027)

* Wasmtime: generalize `async_stack_zeroing` knob to cover initialization

This commit moves the knob from the `Pooli

Wasmtime: generalize `async_stack_zeroing` knob to cover initialization (#10027)

* Wasmtime: generalize `async_stack_zeroing` knob to cover initialization

This commit moves the knob from the `PoolingInstanceAllocatorConfig` to the
regular `Config` and now controls both whether stacks are zeroed before reuse
and whether they are zeroed before the initial use. The latter doesn't matter
usually, since anonymous mmaps are already zeroed so we don't have to do
anything there, but for no-std environments it is the difference between
manually zeroing the stack or simply using unininitialized memory.

* Fix CLI and test builds

* fix default config value

* fix some more tests

show more ...

Port wasmtime-fiber to `no_std` and allow `async` feature in `no_std` Wasmtime. (#9689)

This PR allows a `no_std` Wasmtime build to be configured with the
`async` feature. (Previously, a minimal `no

Port wasmtime-fiber to `no_std` and allow `async` feature in `no_std` Wasmtime. (#9689)

This PR allows a `no_std` Wasmtime build to be configured with the
`async` feature. (Previously, a minimal `no_std` configuration could
only run with sync entry points, without suspending of stacks.)

The main hurdle to this support was the `wasmtime-fiber` crate.
Fortunately, the "unix" variant of fibers was almost entirely portable
to a `no_std` environment, owing to the fact that it implements
stack-switching manually in assembly itself. I moved the per-ISA
implementations to a shared submodule and built the nostd platform
backend for `wasmtime-fiber` with a stripped-down version of the unix
backend.

The nostd backend does not support mmap'd stacks, does not support
custom stack allocators, and does not propagate panics.

prtest:full

show more ...

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint levels. This new lint annotation will
silence a lint but will itself cause a lint if it doesn't actually
silence anything. This is quite useful to ensure that annotations don't
get stale over time.

Another feature is the ability to use a `reason` directive on the
attribute with a string explaining why the attribute is there. This
string is then rendered in compiler messages if a warning or error
happens.

This commit migrates applies a few changes across the workspace:

* Some `#[allow]` are changed to `#[expect]` with a `reason`.
* Some `#[allow]` have a `reason` added if the lint conditionally fires
(mostly related to macros).
* Some `#[allow]` are removed since the lint doesn't actually fire.
* The workspace configures `clippy::allow_attributes_without_reason = 'warn'`
as a "ratchet" to prevent future regressions.
* Many crates are annotated to allow `allow_attributes_without_reason`
during this transitionary period.

The end-state is that all crates should use
`#[expect(..., reason = "...")]` for any lint that unconditionally fires
but is expected. The `#[allow(..., reason = "...")]` lint should be used
for conditionally firing lints, primarily in macro-related code.
The `allow_attributes_without_reason = 'warn'` level is intended to be
permanent but the transitionary
`#[expect(clippy::allow_attributes_without_reason)]` crate annotations
to go away over time.

* Fix adapter build

prtest:full

* Fix one-core build of icache coherence

* Use `allow` for missing_docs

Work around rust-lang/rust#130021 which was fixed in Rust 1.83 and isn't
fixed for our MSRV at this time.

* More MSRV compat

show more ...

Store one fiber stack in a `Store<T>` (#9604)

* Store one fiber stack in a `Store<T>`

This commit stores a single fiber stack in `Store<T>` as a cache to be
used throughout the lifetime of the `Sto

Store one fiber stack in a `Store<T>` (#9604)

* Store one fiber stack in a `Store<T>`

This commit stores a single fiber stack in `Store<T>` as a cache to be
used throughout the lifetime of the `Store`. This should help amortize
the cost of allocating a stack for use in a store because the same stack
can be used continuously throughout the lifetime of the `Store<T>`. This
notably reduces contention on the lock used to manage the pooling
allocator when possible.

* Fix non-async build

show more ...

Print an error on async stack overflow (#9304)

* Print an error on async stack overflow

This commit updates Wasmtime's handling of traps on Unix platforms to
print an error message on stack overflo

Print an error on async stack overflow (#9304)

* Print an error on async stack overflow

This commit updates Wasmtime's handling of traps on Unix platforms to
print an error message on stack overflow when the guard page is hit.
This is distinct from stack overflow in WebAssembly which raises a
normal trap and can be caught. This is instead to be used on
misconfigured hosts where the async stack is too small or wasm was
allowed to take up too much of the async stack. Currently no error
message is printed and the program simply aborts with a core dump which
can be difficult to debug.

This instead registers the range of the async guard page with the trap
handling infrastructure to test the faulting address and if it lies
within this range. If so then a small message is printed and then the
program is aborted with `libc::abort()`.

This does not impact the safety of any prior embedding or fix any
issues. It's instead intended purely as a diagnostic tool to help users
more quickly understand that stack size configuration settings are the
likely culprit.

* Fix build of c-api and tests

prtest:full

* Fix build on Windows

* Fix a warning on Windows

* Fix dead code on miri

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

Build and test wasmtime-fiber on i686/arm (#9035)

Now that we've got CI for Pulley throw in `wasmtime-fiber` as well with
a few extra fixes. This should also help improve the compilation
experience

Build and test wasmtime-fiber on i686/arm (#9035)

Now that we've got CI for Pulley throw in `wasmtime-fiber` as well with
a few extra fixes. This should also help improve the compilation
experience of Wasmtime to 32-bit platforms to hit a more first-class
error about unsupported platforms.

show more ...

Wasmtime(pooling allocator): Batch decommits (#8590)

This introduces a `DecommitQueue` for batching decommits together in the pooling
allocator:

* Deallocating a memory/table/stack enqueues their a

Wasmtime(pooling allocator): Batch decommits (#8590)

This introduces a `DecommitQueue` for batching decommits together in the pooling
allocator:

* Deallocating a memory/table/stack enqueues their associated regions of memory
for decommit; it no longer immediately returns the associated slot to the
pool's free list. If the queue's length has reached the configured batch size,
then we flush the queue by running all the decommits, and finally returning
the memory/table/stack slots to their respective pools and free lists.

* Additionally, if allocating a new memory/table/stack fails because the free
list is empty (aka we've reached the max concurrently-allocated limit for this
entity) then we fall back to a slow path before propagating the error. This
slow path flushes the decommit queue and then retries allocation, hoping that
the queue flush reclaimed slots and made them available for this fallback
allocation attempt. This involved defining a new `PoolConcurrencyLimitError`
to match on, which is also exposed in the public embedder API.

It is also worth noting that we *always* use this new decommit queue now. To
keep the existing behavior, where e.g. a memory's decommits happen immediately
on deallocation, you can use a batch size of one. This effectively disables
queueing, forcing all decommits to be flushed immediately.

The default decommit batch size is one.

This commit, with batch size of one, consistently gives me an increase on
`wasmtime serve`'s requests-per-second versus its parent commit, as measured by
`benches/wasmtime-serve-rps.sh`. I get ~39K RPS on this commit compared to ~35K
RPS on the parent commit. This is quite puzzling to me. I was expecting no
change, and hoping there wouldn't be a regression. I was not expecting a speed
up. I cannot explain this result at this time.

prtest:full

Co-authored-by: Jamey Sharp <[email protected]>

show more ...

Add a fuzzer for async wasm (#8440)

* Add a fuzzer for async wasm

This commit revives a very old branch of mine to add a fuzzer for
Wasmtime in async mode. This work was originally blocked on
llvm/

Add a fuzzer for async wasm (#8440)

* Add a fuzzer for async wasm

This commit revives a very old branch of mine to add a fuzzer for
Wasmtime in async mode. This work was originally blocked on
llvm/llvm-project#53891 and while that's still an issue it now contains
a workaround for that issue. Support for async fuzzing required a good
deal of refactorings and changes, and the highlights are:

* The main part is that new intrinsics,
`__sanitizer_{start,finish}_fiber_switch` are now invoked around the
stack-switching routines of fibers. This only works on Unix and is set
to only compile when ASAN is enabled (otherwise everything is a noop).
This required refactoring of things to get it all in just the right
way for ASAN since it appears that these functions not only need to be
called but more-or-less need to be adjacent to each other in the code.
My guess is that while we're switching ASAN is in a "weird state" and
it's not ready to run arbitrary code.

* Stacks are a problem. The above issue in LLVM outlines how stacks
cannot be deallocated at this time because if the deallocated virtual
memory is later used for the heap then ASAN will have a false positive
about stack overflow. To handle this stacks are specially handled in
asan mode by using a special allocation path that never deallocates
stacks. This logic additionally applies to the pooling allocator which
uses a different stack allocation strategy with ASAN.

With all of the above a new fuzzer is added. This fuzzer generates an
arbitrary module, selects an arbitrary means of async (e.g.
epochs/fuel), and then tries to execute the exports of the module with
various values. In general the fuzzer is looking for crashes/panics as
opposed to correct answers as there's no oracle here. This is also
intended to stress the code used to switch on and off stacks.

* Fix non-async build

* Remove unused import

* Review comments

* Fix compile on MIRI

* Fix Windows build

show more ...

clarify custom memory needs to be zero filled (#7565)

Signed-off-by: Tyler Rockwood <[email protected]>

Introduce API for custom stack memory (#7209)

* Introduce StackMemory and StackMemoryCreator

This allows custom implementations of stack memory to be plugged into
the async functionality for wasmti

Introduce API for custom stack memory (#7209)

* Introduce StackMemory and StackMemoryCreator

This allows custom implementations of stack memory to be plugged into
the async functionality for wasmtime. Currently, stacks are always
mmapped, and this custom allocator allows embedders to use any memory
they would like.

The new APIs are also exposed in the C api.

This has no effect on windows, as our hands are tied with fibers there.

Signed-off-by: Tyler Rockwood <[email protected]>

* Add test for custom host memory

Signed-off-by: Tyler Rockwood <[email protected]>

* fix allocator test

Signed-off-by: Tyler Rockwood <[email protected]>

* Address review comments

Signed-off-by: Tyler Rockwood <[email protected]>

* Fix lint warnings

prtest:full

Signed-off-by: Tyler Rockwood <[email protected]>

* fix windows lint warning

prtest:full

Signed-off-by: Tyler Rockwood <[email protected]>

---------

Signed-off-by: Tyler Rockwood <[email protected]>

show more ...

Fix a soundness issue with the component model and async (#6509)

* Force `execute_across_threads` to use multiple threads

Currently this uses tokio's `spawn_blocking` but that will reuse threads
in

Fix a soundness issue with the component model and async (#6509)

* Force `execute_across_threads` to use multiple threads

Currently this uses tokio's `spawn_blocking` but that will reuse threads
in its thread pool. Instead spawn a thread and perform a single poll on
that thread to force lots of fresh threads to be used and ideally stress
TLS management further.

* Add a guard against using stale stacks

This commit adds a guard to Wasmtime's async support to double-check
that when a call to `poll` is finished that the currently active TLS
activation pointer does not point to the stack that is being switched
off of. This is attempting to be a bit of a defense-in-depth measure to
prevent stale pointers from sticking around in TLS. This is currently
happening and causing #6493 which can result in unsoundness but
currently is manifesting as a crash.

* Fix a soundness issue with the component model and async

This commit addresses #6493 by fixing a soundness issue with the async
implementation of the component model. This issue has been presence
since the inception of the addition of async support to the component
model and doesn't represent a recent regression. The underlying problem
is that one of the base assumptions of the trap handling code is that
there's only one single activation in TLS that needs to be pushed/popped
when a stack is switched (e.g. a fiber is switched to or from). In the
case of the component model there might be two activations: one for an
invocation of a component function and then a second for an invocation
of a `realloc` function to return results back to wasm (e.g. in the case
an imported function returns a list).

This problem is fixed by changing how TLS is managed in the presence of
fibers. Previously when a fiber was suspended it would pop a single
activation from the top of the stack and save that to get pushed when
the fiber was resumed. This has the benefit of maintaining an entire
linked list of activations for the current thread but has the problem
above where it doesn't handle a fiber with multiple activations on it.
Instead now TLS management is done when a fiber is resumed instead of
suspended. Instead of pushing/popping a single activation the entire
linked list of activations is tracked for a particular fiber and stored
within the fiber itself. In this manner resuming a fiber will push
all activations onto the current thread and suspending a fiber will pop
all activations for the fiber (and store them as a new linked list in
the fiber's state itself).

This end result is that all activations on a fiber should now be managed
correctly, regardless of how many there are. The main downside of this
commit is that fiber suspension and resumption is more complicated, but
the hope there is that fiber suspension typically corresponds with I/O
not being ready or similar so the order of magnitude of TLS operations
isn't too significant compared to the I/O overhead.

Closes #6493

* Review comments

* Fix restoration during panic

show more ...

Fix some warnings on nightly Rust (#6388)

Upstream rust has decided that ignoring a value is not spelled
`drop(foo)` but instead it's spelled `let _ = foo`

Enable back-edge CFI by default on macOS (#4720)

Also, adjust the tests that are executed on that platform. Finally,
fix a bug with obtaining backtraces when back-edge CFI is enabled.

Copyright

Enable back-edge CFI by default on macOS (#4720)

Also, adjust the tests that are executed on that platform. Finally,
fix a bug with obtaining backtraces when back-edge CFI is enabled.

Copyright (c) 2022, Arm Limited.

show more ...

Use `global_asm!` instead of external assembly files (#4306)

* Use `global_asm!` instead of external assembly files

This commit moves the external assembly files of the `wasmtime-fiber`
crate in

Use `global_asm!` instead of external assembly files (#4306)

* Use `global_asm!` instead of external assembly files

This commit moves the external assembly files of the `wasmtime-fiber`
crate into `global_asm!` blocks defined in Rust. The motivation for
doing this is not very strong at this time, but the points in favor of
this are:

* One less tool needed to cross-compile Wasmtime. A linker is still
needed but perhaps one day that will improve as well.
* A "modern" assembler, built-in to LLVM, is used instead of whatever
appears on the system.

The first point hasn't really cropped up that much and typically getting
an assembler is just as hard as getting a linker nowadays. The second
point though has us using `hint #xx` in aarch64 assembly instead of the
actual instructions for assembler compatibility, and I believe that's no
longer necessary because the LLVM assembler supports the modern
instruction names.

The translation of the x86/x86_64 assembly has been done to Intel
syntax as well as opposed to the old AT&T syntax since that's Rust's
default. Additionally s390x still remains in an external assembler file
because `global_asm!` is still unstable in Rust on that platform.

* Simplify alignment specification

* Temporarily disable fail-fast

* Add `.cfi_def_cfa_offset 0` to fix CI

* Turn off fail-fast

* Review comments

show more ...