Handle more failures deallocating pooled memroy (#12888)

* Handle more failures deallocating pooled memroy

This commit replaces a few panics in the pooling allocator with
error-handling of what hap

Handle more failures deallocating pooled memroy (#12888)

* Handle more failures deallocating pooled memroy

This commit replaces a few panics in the pooling allocator with
error-handling of what happens at runtime. This is a defense-in-depth
measure to ensure that the pooling allocator doesn't panic at runtime
and instead handles errors where possible.

The first path fixed is in `deallocate_memory` where resetting a slot
could result in an error being returned on non-Linux platforms, and if
this happened it would cause a panic. The error is instead gracefully
handled by continuing slot deallocation but avoiding putting the image
itself back into memory. This leaves the slot in an `Unknown` state
which is already handled by resetting the state upon reuse. The main
consequence here is that future statistics about resident bytes won't be
accurate, but these are already inaccurate on non-Linux platforms
anyway, so there's no loss.

The second path fixes is in flushing a `DecommitQueue` where
`decommit_pages` was asserted to succeed. Instead now the error is
handled by dropping all images and leaving slots in an `Unknown` state,
similar to `deallocate_memory`.

* Review comments

show more ...

Reduce type complexity of `InstanceAllocator` async functions (#12887)

This is a follow-on to #12849 to try to simplify some of the resulting
signatures a bit. Notably the `Result<..., OutOfMemory>`

Reduce type complexity of `InstanceAllocator` async functions (#12887)

This is a follow-on to #12849 to try to simplify some of the resulting
signatures a bit. Notably the `Result<..., OutOfMemory>` is now packaged
up directly into the output future, so the functions still retain a sort
of "async trait" feel even though they're still incompatible with
`#[async_trait]` (and can't be defined with that anyway).

show more ...

Use explicit boxing for InstanceAllocator async methods (#12849)

Change `allocate_memory` and `allocate_table` in the `InstanceAllocator` trait
from `async fn`s to regular `fn`s that return `Result<

Use explicit boxing for InstanceAllocator async methods (#12849)

Change `allocate_memory` and `allocate_table` in the `InstanceAllocator` trait
from `async fn`s to regular `fn`s that return `Result<Pin<Box<dyn Future<...>>>,
OutOfMemory>`.

This avoids the implicit `Box::new` allocation that `#[async_trait]` generates
when calling these methods through `dyn InstanceAllocator`, which would panic on
OOM instead of returning an error. Now the boxing is done explicitly via
`try_new::<Box<_>>` which returns `Err(OutOfMemory)` on allocation failure.

show more ...

Include core instance sizes in component_instance_size limit (#12772)

* Include core instance sizes in component_instance_size limit

There exist several knobs for limiting the memory that might be

Include core instance sizes in component_instance_size limit (#12772)

* Include core instance sizes in component_instance_size limit

There exist several knobs for limiting the memory that might be consumed
for metadata for components. For core module instances within a
component, the two that previously existed to control metadata
allocations have been:
- A: max_core_instances_per_component
- B: component_instance_size

These allow for an embedder to set an upper bound on memory used by a
component's instances to A * B. This value could be quite large for
some systems and it would be nice to be able to set a cap on the total
memory that might be used for metadata across all instances while still
allowing for a greater number of instances with the potential for a
subset of those instances to be relatively large.

To allow for aggregate control over memory used within the runtime
for componenets, the existing `max_component_instance_size` limit
is extended to consider both the `VMComponentCtx` size as well as
the aggregate size of all core instances in the component.

* Fix err msg checks for component_instance_size_limit test

* Miri ignore component_core_instances_aggregate_size

show more ...

consistently create thread and task when entering component instance (#12379)

* consistently create thread and task when entering component instance

Previously, we weren't creating a new thread or

consistently create thread and task when entering component instance (#12379)

* consistently create thread and task when entering component instance

Previously, we weren't creating a new thread or task in all cases when entering
a component instance, even when component model async features were enabled. In
particular, entering an instance via a sync-to-sync, guest-to-guest adapter, via
`Linker::instantiate[_async]`, or via `[Typed]Func::call` all skipped creating a
thread or task, creating panics and/or instance mismatches in certain cases.

This commit addresses all those cases and also adds assertions to all CM async
intrinsics to verify that the caller instance matches the most-recently-pushed
task. Note that we still skip pushing and popping threads and tasks if no CM
async features are enabled in the `Config`.

In order to populate the `GuestTask::instance` field for tasks created as part
of `Linker::instantiate[_async]` calls, I had to add a
`RuntimeComponentInstanceIndex` field to `GlobalInitializer::InstantiateModule`
and friends so it would be available when needed.

While testing this, I uncovered and fixed a couple of related issues:

- We weren't checking the `may_leave` flag when guest-to-guest calling a resource destructor
- We weren't checking whether a subtask was ready to delete (e.g. that no threads were still running) before attempting to delete it while deleting its supertask

* fix warnings when component-model-async feature disabled

* address review feedback

* push a task when calling a resource dtor host-to-guest

* add tests which call `thread.index` from realloc and post-return functions

...and assert that the indexes match as expected.

Getting the post-return test to pass required moving the call to
`StoreOpaque::exit_sync_call` to after the post-return function is called.

show more ...

Do not re-export `anyhow!` in the `wasmtime::prelude` (#12298)

We are trying to move to `format_err!` instead.

Another batch of dependency updates (#11832)

* Another batch of dependency updates

Bringing some deps in `Cargo.toml` up-to-date with their latest versions
along the same lines as #11820 to avoid d

Another batch of dependency updates (#11832)

* Another batch of dependency updates

Bringing some deps in `Cargo.toml` up-to-date with their latest versions
along the same lines as #11820 to avoid deps getting too stale/old.

Code-wise this updates `anyhow` which enables preexisting Clippy
warnings to check more code, so those warnings are fixed here as well.

prtest:full

* Run rustfmt

show more ...

Add some more metrics to the pooling allocator (#11789)

* Add some more metrics to the pooling allocator

This commit adds a few more metrics to the `PoolingAllocatorMetrics`
type along the lines of

Add some more metrics to the pooling allocator (#11789)

* Add some more metrics to the pooling allocator

This commit adds a few more metrics to the `PoolingAllocatorMetrics`
type along the lines of accounting for more items as well as the unused
slots in the pooling allocator. Notably the count of unused memory and
table slots is exposed along with the number of bytes which are kept
resident in these slots despite them not being in use. This involved a
bit of plumbing to thread around the number of bytes that are actually
kept resident to some more locations but is otherwise a pretty
straightforward plumbing of accounting information we already had
internally.

* Fix configured build

* Add some docs

* Only run new test on Linux

* Try to fix ASAN

prtest:full

* Shrink pooling size in tests

show more ...

Add PoolingAllocatorMetrics (#11490)

This exposes some basic runtime metrics derived from the internal state
of a `PoolingInstanceAllocator`.

Two new atomics were added to PoolingInstanceAllocator:

Add PoolingAllocatorMetrics (#11490)

This exposes some basic runtime metrics derived from the internal state
of a `PoolingInstanceAllocator`.

Two new atomics were added to PoolingInstanceAllocator: `live_memories`
and `live_tables`. While these counts could be derived from existing
state it would require acquiring mutexes on some inner state.

show more ...

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "

Remove unsoundness of widening store borrows (#11481)

* Remove unsoundness of widening store borrows

This commit removes preexisting unsoundness in Wasmtime where a
`&mut StoreOpaque` borrow was "widened" into encompassing the limiter on
the `T` in `StoreInner<T>`, for example, by using the self-pointer
located in an instance or the store. This fix is done by threading
`&mut StoreOpaque` as a parameter separately from a
`StoreResourceLimiter`. This means that various callers now take a new
`Option<&mut StoreResourceLimiter<'_>>` parameter in various locations.

Closes #11409

* Fix gc-less build

show more ...

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `Store

Make table/memory creation async functions (#11470)

* Make core instance allocation an `async` function

This commit is a step in preparation for #11430, notably core instance
allocation, or `StoreOpaque::allocate_instance` is now an `async fn`.
This function does not actually use the `async`-ness just yet so it's a
noop from that point of view, but this propagates outwards to enough
locations that I wanted to split this off to make future changes more
digestable.

Notably some creation functions here such as making an `Instance`,
`Table`, or `Memory` are refactored internally to use this new `async`
function. Annotations of `assert_ready` or `one_poll` are used as
appropriate as well.

For reference this commit was benchmarked with our `instantiation.rs`
benchmark in the pooling allocator and shows no changes relative to the
original baseline from before-`async`-PRs.

* Make table/memory creation `async` functions

This commit is a large-ish refactor which is made possible by the many
previous refactorings to internals w.r.t. async-in-Wasmtime. The end
goal of this change is that table and memory allocation are both `async`
functions. Achieving this, however, required some refactoring to enable
it to work:

* To work with `Send` neither function can close over `dyn VMStore`.
This required changing their `Option<&mut dyn VMStore>` arugment to
`Option<&mut StoreResourceLimiter<'_>>`
* Somehow a `StoreResourceLimiter` needed to be acquired from an
`InstanceAllocationRequest`. Previously the store was stored here as
an unsafe raw pointer, but I've refactored this now so
`InstanceAllocationRequest` directly stores `&StoreOpaque` and
`Option<&mut StoreResourceLimiter>` meaning it's trivial to acquire
them. This additionally means no more `unsafe` access of the store
during instance allocation (yay!).
* Now-redundant fields of `InstanceAllocationRequest` were removed since
they can be safely inferred from `&StoreOpaque`. For example passing
around `&Tunables` is now all gone.
* Methods upwards from table/memory allocation to the
`InstanceAllocator` trait needed to be made `async`. This includes new
`#[async_trait]` methods for example.
* `StoreOpaque::ensure_gc_store` is now an `async` function. This
internally carries a new `unsafe` block carried over from before with
the raw point passed around in `InstanceAllocationRequest`. A future
PR will delete this `unsafe` block, it's just temporary.

I attempted a few times to split this PR up into separate commits but
everything is relatively intertwined here so this is the smallest
"atomic" unit I could manage to land these changes and refactorings.

* Shuffle `async-trait` dep

* Fix configured build

show more ...

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
wi

Refactor `InstanceAllocator` trait impl split (#11457)

Prior to this commit Wasmtime had an `InstanceAllocatorImpl` trait with
a number of required methods as well as an `InstanceAllocator` trait
with a number of provided impls. The `InstanceAllocator` trait is
implemented for everything implementing `InstanceAllocatorImpl` to force
users to be unable to override the default methods. When adding `async`
support internally to Wasmtime these are going to need to be
`#[async_trait]`-annotated-traits which adds a cost to `async` functions
as a future needs to be heap-allocated.

The goal of this commit is to make this future `async`-ification a bit
more optimal. Notably the `InstanceAllocator` trait is removed and
replaced with inherent methods on `impl dyn InstanceAllocatorImpl`.
After that the previous `InstanceAllocatorImpl` trait was renamed to
`InstanceAllocator` meaning that there's just one `InstanceAllocator`
trait which has inherent methods which cannot be overridden. A
consequence of this is that the inherent methods are also forced to do
virtual dispatch unlike before where they would internally use
monomorphization to have static dispatch. Given the complexity of
instance allocation this is expected to be a negligible cost, however.

The main benefit is that `allocate_module`, `allocate_tables`, and
`allocate_memories` all get to be native `async` functions without the
cost of `#[async_trait]`. Only allocation of a single table/memory will
require an allocation of a future which in profiling helps reduce the
cost of instantiation slightly.

Note that `#[async_trait]` is not currently used, this commit is just
preparation for its eventual use.

show more ...

Add a configuration knob for `PAGEMAP_SCAN` (#11433)

* Add a configuration knob for `PAGEMAP_SCAN`

This commit adds `PoolingAlloationConfig::pagemap_scan` and additionally
adds `-Opooling-pagemap-s

Add a configuration knob for `PAGEMAP_SCAN` (#11433)

* Add a configuration knob for `PAGEMAP_SCAN`

This commit adds `PoolingAlloationConfig::pagemap_scan` and additionally
adds `-Opooling-pagemap-scan` to configure on the CLI. This is the same
tri-state configuration option as `MpkEnable` so that enum was renamed
to just `Enabled` and repurposed for both options.

This then additionally turns the option off-by-default instead of the
previous on-by-default-if-able-to to enable more slowly rolling out this
feature.

* Fix broken test

show more ...

Add support for the Linux PAGEMAP_SCAN ioctl (#11372)

* WIP: use the pagemap_scan ioctl to selectively reset an instance's dirty pages

* Less hacky, and supporting tables, too

* Bugfixes

* WIP: m

Add support for the Linux PAGEMAP_SCAN ioctl (#11372)

* WIP: use the pagemap_scan ioctl to selectively reset an instance's dirty pages

* Less hacky, and supporting tables, too

* Bugfixes

* WIP: memcpy instead of pread

* Refactor support for pagemap

* Don't hold a raw pointer to the original data, plumb through an `Arc`
to have a safe reference instead.
* Update pagemap bindings to latest version of abstraction written.
* Include pagemap-specific tests.
* Use a `PageMap` structure created once-per-pool instead of a
static-with-a-file.
* Refactor to use the "pagemap path" unconditionally which blends in the
keep_resident bits.

* Improve safety documentation

prtest:full

* Fix some lints

* Skip ioctl tests when it's not supported

* Fix a memory leak by moving impls around

* Fix no vm build

* Review comments

* Add more pagemap-specific documentation

* Add more docs, refactor implementation slightly

* Improve `category_*` docs

Basically forward to the Linux kernel source itself.

* Fix compile

* Make pagemap integration resilient across forks

* Fix non-pooling-allocator-build

* Fix portability issues of new test

* Actually use config on macos

---------

Co-authored-by: Till Schneidereit <[email protected]>

show more ...

Make table/memory allocation functions safe (#11320)

These were previously marked as `unsafe` trait methods with a
requirement that the memory/table shape must be validated ahead of time.
Neither th

Make table/memory allocation functions safe (#11320)

These were previously marked as `unsafe` trait methods with a
requirement that the memory/table shape must be validated ahead of time.
Neither the ondemand nor pooling allocator actually has an unsafe
contract to uphold with respect to this and both may assert/reject
non-validated shapes but memory unsafety won't happen as a result.
Consequently these functions are made safe.

Instance allocation functions are adjusted to reflect how the
correctness of `imports` is required for the functions to be safe.

show more ...

Deny `unsafe_op_in_unsafe_fn` in `wasmtime::vm::instance` (#11267)

Turn the lint on and add some safety comments where appropriate.

cc #11180

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previou

Enable `allow_attributes_without_reason` (#11195)

* Enable `allow_attributes_without_reason`

This commit enables the `clippy::allow_attributes_without_reason` for
the `wasmtime` crate which previously forcibly allowed it. The reason
this was allowed was that when the workspace was first migrated the
Wasmtime crate had too many instances that I was willing to fix. I've
now come back around and tried to fix everything.

In short: ideally delete `#[allow]`, otherwise use `#[expect]`,
otherwise use `#[allow]`.

prtest:full

* Adjust some directives

* Fix some warnings

* Fix stack switching size tests on unix

* Don't have a conditional `Drop` impl

* Force `testing_freelist` method to be used

Too lazy to write `#[cfg]`, but not too lazy to write a test.

show more ...

Switch some `Error` references to `core::error::Error` (#11067)

Using the trait through `std` at this point is just vestigal.

Closes #11059

Stack switching: Infrastructure and runtime support (#10388)

* [pr1] base

* prtest:full

* make sure to use ControlFlow result in trace_suspended_continuation

* stack-switching: cleanup: remove st

Stack switching: Infrastructure and runtime support (#10388)

* [pr1] base

* prtest:full

* make sure to use ControlFlow result in trace_suspended_continuation

* stack-switching: cleanup: remove stray c-api changes

These are remnants of unrelated wasmfx wasmtime experiments, possibly
suitable for later submission against upstream.

* stack-switching: reuse async_stack_size

* stack-switching: delete delete_me debugging

* stack-switching: address feedback in environ::types

* stack-switching: remove unused code from vmoffsets

* stack-switching: drop dependency on std

* stack-switching: add compilation checks to ci matrix

* stack-switching: remove debug_println cruft

* stack-switching: export environ consts consistently

* stack-switching: export vm pub items consistently

* table_pool: reduced capacity for large elements

VMContRef elements which takes up two words and we don't want to
double the size of all tables in order to support storing these.
This change changes the table to target storing the requested
max number of elements if they are "nominally" sized with
(potentially) reduced capacity for non-nominally sized types when
encountered.

Continuations are the only type of element which may result in
fewer table slots being available than requested.

* stack-switching: extend conditional compilation

A fair bit of the definitions for stack switching are still
enabled, but this patch takes things a bit further to avoid
compilation problems; notably, cont_new is now not compiled
in unless the feature is enabled.

* stack-switching: formatting fixes

* stack-switching: address new clippy checks

In addition, to get clippy to fully pass, plumbed in
additional config to make winch paths happy; there's no
impl for winch yet but plumbing through the feature is
required to make paths incorporating macros at various
layers satisfied (and it is expected we'll use the
features in the future).

* stack-switching: more conditional compilation fixes

* stack-switching: additional conditional compile on table builtins for continuations

* stack-switching: additional conditional compile fixes

* stack-switching: additional conditional compile in store

* stack-switching: remove overly strict assertion

* stack-switching: remove errantly dropped no_mangle in config c-api

* stack-switching: VMContObj::from_raw_parts

* stack-switching: remove duplicate async_stack_size feature check

* stack-switching: VMArray -> VMHostArray

* stack-switching: remove unnecessary clippy exception

* stack-switching: fix docs referenced VMRuntimeLimits

* stack-switching: fix doc typo

* stack-switching: follow recommendations for type casts

* stack-switching: use usize::next_multiple_of

* stack-switching: update outdated comment

* stack-switching: use feature gate instead of allow(dead_code)

* stack-switching: rework backtrace using chunks/zip

* stack-switching: move tests to footer module

This is a bit more consistent with the prevailing style
in tree and (subjectively) makes finding the tests
as a reader more straightforward.

Tests left unchanged sans some import cleanup.

* stack-swictchding: verify stack_chain offsets at runtime

* fixup! stack-switching: use feature gate instead of allow(dead_code)

* stack-switching: document continuation roots tracing using match arms

---------

Co-authored-by: Paul Osborne <[email protected]>

show more ...

Update pinned nightly used in CI (#10893)

* Update pinned nightly used in CI

Resolves some various warnings cropping up from rustc/clippy. This does
trigger what for us is a false positive (I belie

Update pinned nightly used in CI (#10893)

* Update pinned nightly used in CI

Resolves some various warnings cropping up from rustc/clippy. This does
trigger what for us is a false positive (I believe) dead code warning
which I've annotated as `#[allow]` for now (alas this can't be
`#[expect]` because it's a nighly-only warning).

* Remove dead methods when component-model is disabled

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Reuse Wasm linear memories code for GC heaps (#10503)

* Reuse code for Wasm linear memories for GC heaps

Instead of bespoke code paths and structures for Wasm GC, this commit makes it
so that we no

Reuse Wasm linear memories code for GC heaps (#10503)

* Reuse code for Wasm linear memories for GC heaps

Instead of bespoke code paths and structures for Wasm GC, this commit makes it
so that we now reuse VM structures like `VMMemoryDefinition` and bounds-checking
logic. Notably, we also reuse all the associated bounds-checking optimizations
and, when possible, virtual-memory techniques to completely elide them.

Furthermore, this commit adds support for growing GC heaps, reusing the
machinery for growing memories, and makes it so that GC heaps always start out
empty. This allows us to properly delay allocating the GC heap's storage until a
GC object is actually allocated.

Fixes #9350

* fix c api compilation

* use assert_contains

* remove no-longer-necessary extra memory config from limiter tests

* Helper for retry-after-maybe-async-gc in libcalls

* Clean up some comments

* fix wasmtime-fuzzing and no-gc compilation

* fix examples

* fix no-gc+compiler build

* fix build without pooling allocator

* fix +cranelift +gc-drc -gc-null builds

* fix table hash key stability test

* fix oracle usage of `ExternRef::new`

* fix +gc -gc-null -gc-drc build

* fix wasmtime-fuzzing

* make `StorePtr` wrap a `NonNull`

* Fix some doc tests

* Remove some unnecessary retry helpers now that `FooRef::new` will auto-gc

* fix things after rebase

* Reorganize collection/growth methods for GC heap

* rename BoundsCheck variants

* fix cfg'ing of gc only code

* Fix doc tests

* fix one more gc cfg

* disable GC heap OOM test on non-64-bit targets

show more ...

Run CI tests through AddressSanitizer (#10537)

This is similar to running tests in Valgrind (which we should perhaps
also do...) but can be useful for catching use-after-free style bugs
faster than

Run CI tests through AddressSanitizer (#10537)

This is similar to running tests in Valgrind (which we should perhaps
also do...) but can be useful for catching use-after-free style bugs
faster than when a process crashes. Given the unsafe nature of Wasmtime
this is something we should have probably enabled awhile back but
otherwise so long as it doesn't take too long to run on CI seems like an
easy win of a boost-of-confidence.

prtest:asan

show more ...

wasmtime-environ: add initial table support (#10497)

* wasmtime-environ: add initial table support

In order to lower the [`thread.spawn_indirect`] canonical builtin from
the component model, we nee

wasmtime-environ: add initial table support (#10497)

* wasmtime-environ: add initial table support

In order to lower the [`thread.spawn_indirect`] canonical builtin from
the component model, we need to be able to refer to the tables exported
from instance. This change adds the initial plumbing to access such
tables.

[`thread.spawn_indirect`]: https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md#-canon-threadspawn_indirect

* review: add missing doc comment

* review: add missing field calculation

* review: fix a return type

* fix: gate `pub use` behind component-model feature

show more ...

Improve errors with pooling-by-default `serve` command (#10483)

* Improve errors with pooling-by-default `serve` command

This commit is intended to address #10482 where the defaults of the
`wasmtim

Improve errors with pooling-by-default `serve` command (#10483)

* Improve errors with pooling-by-default `serve` command

This commit is intended to address #10482 where the defaults of the
`wasmtime serve` subcommand produced a confusing and surprising error.
Specifically the `-Wmax-table-elements` option, prior to this change, only
affected the store limiter used and didn't actually affect the pooling
allocator settings. That meant that if a module exceeded the limits of
the pooling allocator it would produce an error message that seemed like
`-Wmax-table-elements` would fix but it wouldn't actually.

Two changes in this commit are meant to address this:

* Errors from the pooling allocator have been update to mention "pooling
allocator" within them somewhere to surface where the error is coming
from.
* The `-Wmax-memory-size` and `-Wmax-table-elements` configuration are
now applied to the pooling allocator automatically if the
corresponding `-Opooling-*` option isn't passed.

That should mean that the original error should be a bit easier to debug
while the attempted solution will also work.

* Fix test expectations

* Get new test passing on 32-bit

show more ...