Add a configuration knob for `PAGEMAP_SCAN` (#11433)

* Add a configuration knob for `PAGEMAP_SCAN`

This commit adds `PoolingAlloationConfig::pagemap_scan` and additionally
adds `-Opooling-pagemap-s

Add a configuration knob for `PAGEMAP_SCAN` (#11433)

* Add a configuration knob for `PAGEMAP_SCAN`

This commit adds `PoolingAlloationConfig::pagemap_scan` and additionally
adds `-Opooling-pagemap-scan` to configure on the CLI. This is the same
tri-state configuration option as `MpkEnable` so that enum was renamed
to just `Enabled` and repurposed for both options.

This then additionally turns the option off-by-default instead of the
previous on-by-default-if-able-to to enable more slowly rolling out this
feature.

* Fix broken test

show more ...

Make it easier to reuse fuzzing configuration on the CLI (#10123)

* Make it easier to reuse fuzzing configuration on the CLI

During fuzzing we emit a debug log of configured options to assist with

Make it easier to reuse fuzzing configuration on the CLI (#10123)

* Make it easier to reuse fuzzing configuration on the CLI

During fuzzing we emit a debug log of configured options to assist with
reproducing fuzz test cases outside of the fuzzing harness. Mapping
options the fuzzer is using though to CLI flags, for example, is a bit
of an art though and not obvious. Just today we've got a fuzz bug and I
couldn't figure out how to reproduce on the CLI and it turns out the
issue was that I was forgetting a flag that was being configured in
response to another flag. I got a bit fed up with constantly trying to
map one to the other, so I've decided to fix things.

This commit adds a `Display for CommonOptions` implementation to the
`wasmtime-cli-flags` crate. This is built on the same macro-construction
infrastructure of all our flags making it a relatively low one-time-cost
to implement this. Each option value now implements not only parsing but
printing as well.

Next the `wasmtime-fuzzing` crate was updated to create a
`CommonOptions` first which is then in turn used to create a
`wasmtime::Config`. This provides a layer to insert a log statement with
to emit all configuration options in a form that can be easily
copy/pasted to the CLI to reproduce.

Overall after doing this I was able to quickly reproduce the bug in
question (yay!). The CLI flag logging is pretty verbose right now since
the fuzzing infrastructure sets many settings redundantly to their
defaults, but reducing flags to a minimum is expected to be relatively
easy compared to otherwise trying to extract the options.

* Fix build and dependencies from wasmtime-fuzzing

* Always provide `wasmtime::MpkEnabled`

It's an otherwise very small `enum` which makes it easier to
conditionally compile `wasmtime-cli-flags`

* Frob some crate features some more

* Fix specification of `wasmtime_linkopt_force_jump_veneer` option

show more ...

Wasmtime: generalize `async_stack_zeroing` knob to cover initialization (#10027)

* Wasmtime: generalize `async_stack_zeroing` knob to cover initialization

This commit moves the knob from the `Pooli

Wasmtime: generalize `async_stack_zeroing` knob to cover initialization (#10027)

* Wasmtime: generalize `async_stack_zeroing` knob to cover initialization

This commit moves the knob from the `PoolingInstanceAllocatorConfig` to the
regular `Config` and now controls both whether stacks are zeroed before reuse
and whether they are zeroed before the initial use. The latter doesn't matter
usually, since anonymous mmaps are already zeroed so we don't have to do
anything there, but for no-std environments it is the difference between
manually zeroing the stack or simply using unininitialized memory.

* Fix CLI and test builds

* fix default config value

* fix some more tests

show more ...

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint levels. This new lint annotation will
silence a lint but will itself cause a lint if it doesn't actually
silence anything. This is quite useful to ensure that annotations don't
get stale over time.

Another feature is the ability to use a `reason` directive on the
attribute with a string explaining why the attribute is there. This
string is then rendered in compiler messages if a warning or error
happens.

This commit migrates applies a few changes across the workspace:

* Some `#[allow]` are changed to `#[expect]` with a `reason`.
* Some `#[allow]` have a `reason` added if the lint conditionally fires
(mostly related to macros).
* Some `#[allow]` are removed since the lint doesn't actually fire.
* The workspace configures `clippy::allow_attributes_without_reason = 'warn'`
as a "ratchet" to prevent future regressions.
* Many crates are annotated to allow `allow_attributes_without_reason`
during this transitionary period.

The end-state is that all crates should use
`#[expect(..., reason = "...")]` for any lint that unconditionally fires
but is expected. The `#[allow(..., reason = "...")]` lint should be used
for conditionally firing lints, primarily in macro-related code.
The `allow_attributes_without_reason = 'warn'` level is intended to be
permanent but the transitionary
`#[expect(clippy::allow_attributes_without_reason)]` crate annotations
to go away over time.

* Fix adapter build

prtest:full

* Fix one-core build of icache coherence

* Use `allow` for missing_docs

Work around rust-lang/rust#130021 which was fixed in Rust 1.83 and isn't
fixed for our MSRV at this time.

* More MSRV compat

show more ...

Update wasmi used during fuzzing (#9458)

* Fix differential fuzzing with multi-memory and pooling

Fix an issue found during fuzzing where when multi-memory and the
pooling allocator are enabled the

Update wasmi used during fuzzing (#9458)

* Fix differential fuzzing with multi-memory and pooling

Fix an issue found during fuzzing where when multi-memory and the
pooling allocator are enabled then if MPK is detected and found the
total number of memories needs to be increased because stores belong in
a single stripe.

* Update wasmi used in fuzzing

Pulls in differential fuzzing support for multi-memory

show more ...

Implement the table64 extension to the memory64 proposal (#9206)

This commit implements the table64 extention in both Wasmtime and
Cranelift.

Most of the work was changing a bunch of u32 values to

Implement the table64 extension to the memory64 proposal (#9206)

This commit implements the table64 extention in both Wasmtime and
Cranelift.

Most of the work was changing a bunch of u32 values to u64/usize.
The decisions were made in align with the PR #3153 which
implemented the memory64 propsal itself.

One significant change was the introduction of `IndexType`
and `Limits` which streamline and unify the handling of limits
for both memories and tables.

The spec and fuzzing tests related to table64 are re-enabled which
provides a good coverage of the feature.

show more ...

Enforce `clippy::clone_on_copy` for the workspace (#9025)

* Derive `Copy` for `Val`

* Fix `clippy::clone_on_copy` for the whole repo

* Enforce `clippy::clone_on_copy` for the workspace

* fix some

Enforce `clippy::clone_on_copy` for the workspace (#9025)

* Derive `Copy` for `Val`

* Fix `clippy::clone_on_copy` for the whole repo

* Enforce `clippy::clone_on_copy` for the workspace

* fix some more clippy::clone_on_copy that got missed

show more ...

Use bytes for maximum size of linear memory with pooling (#8628)

* Use bytes for maximum size of linear memory with pooling

This commit changes configuration of the pooling allocator to use a
byte-

Use bytes for maximum size of linear memory with pooling (#8628)

* Use bytes for maximum size of linear memory with pooling

This commit changes configuration of the pooling allocator to use a
byte-based unit rather than a page based unit. The previous
`PoolingAllocatorConfig::memory_pages` configuration option configures
the maximum size that a linear memory may grow to at runtime. This is an
important factor in calculation of stripes for MPK and is also a
coarse-grained knob apart from `StoreLimiter` to limit memory
consumption. This configuration option has been renamed to
`max_memory_size` and documented that it's in terms of bytes rather than
pages as before.

Additionally the documented constraint of `max_memory_size` must be
smaller than `static_memory_bound` is now additionally enforced as a
minor clean-up as part of this PR as well.

* Review comments

* Fix benchmark build

show more ...

Wasmtime(pooling allocator): Batch decommits (#8590)

This introduces a `DecommitQueue` for batching decommits together in the pooling
allocator:

* Deallocating a memory/table/stack enqueues their a

Wasmtime(pooling allocator): Batch decommits (#8590)

This introduces a `DecommitQueue` for batching decommits together in the pooling
allocator:

* Deallocating a memory/table/stack enqueues their associated regions of memory
for decommit; it no longer immediately returns the associated slot to the
pool's free list. If the queue's length has reached the configured batch size,
then we flush the queue by running all the decommits, and finally returning
the memory/table/stack slots to their respective pools and free lists.

* Additionally, if allocating a new memory/table/stack fails because the free
list is empty (aka we've reached the max concurrently-allocated limit for this
entity) then we fall back to a slow path before propagating the error. This
slow path flushes the decommit queue and then retries allocation, hoping that
the queue flush reclaimed slots and made them available for this fallback
allocation attempt. This involved defining a new `PoolConcurrencyLimitError`
to match on, which is also exposed in the public embedder API.

It is also worth noting that we *always* use this new decommit queue now. To
keep the existing behavior, where e.g. a memory's decommits happen immediately
on deallocation, you can use a batch size of one. This effectively disables
queueing, forcing all decommits to be flushed immediately.

The default decommit batch size is one.

This commit, with batch size of one, consistently gives me an increase on
`wasmtime serve`'s requests-per-second versus its parent commit, as measured by
`benches/wasmtime-serve-rps.sh`. I get ~39K RPS on this commit compared to ~35K
RPS on the parent commit. This is quite puzzling to me. I was expecting no
change, and hoping there wouldn't be a regression. I was not expecting a speed
up. I cannot explain this result at this time.

prtest:full

Co-authored-by: Jamey Sharp <[email protected]>

show more ...

mpk: turn on `memory_protection_keys` during fuzzing (#7393)

This also twists the `max_memory_protection_keys` knob.

Wasmtime: refactor the pooling allocator for components (#6835)

* Wasmtime: Rename `IndexAllocator` to `ModuleAffinityIndexAllocator`

We will have multiple kinds of index allocators soon, so clarif

Wasmtime: refactor the pooling allocator for components (#6835)

* Wasmtime: Rename `IndexAllocator` to `ModuleAffinityIndexAllocator`

We will have multiple kinds of index allocators soon, so clarify which one this
is.

* Wasmtime: Introduce a simple index allocator

This will be used in future commits refactoring the pooling allocator.

* Wasmtime: refactor the pooling allocator for components

We used to have one index allocator, an index per instance, and give out N
tables and M memories to every instance regardless how many tables and memories
they need.

Now we have an index allocator for memories and another for tables. An instance
isn't associated with a single instance, each of its memories and tables have an
index. We allocate exactly as many tables and memories as the instance actually
needs.

Ultimately, this gives us better component support, where a component instance
might have varying numbers of internal tables and memories.

Additionally, you can now limit the number of tables, memories, and core
instances a single component can allocate from the pooling allocator, even if
there is the capacity for that many available. This is to give embedders tools
to limit individual component instances and prevent them from hogging too much
of the pooling allocator's resources.

* Remove unused file

Messed up from rebasing, this code is actually just inline in the index
allocator module.

* Address review feedback

* Fix benchmarks build

* Fix ignoring test under miri

The `async_functions` module is not even compiled-but-ignored with miri, it is
completely `cfg`ed off. Therefore we ahve to do the same with this test that
imports stuff from that module.

* Fix doc links

* Allow testing utilities to be unused

The exact `cfg`s that unlock the tests that use these are platform and feature
dependent and ends up being like 5 things and super long. Simpler to just allow
unused for when we are testing on other platforms or don't have the compile time
features enabled.

* Debug assert that the pool is empty on drop, per Alex's suggestion

Also fix a couple scenarios where we could leak indices if allocating an index
for a memory/table succeeded but then creating the memory/table itself failed.

* Fix windows compile errors

show more ...

Reimplement the pooling instance allocation strategy (#5661)

* Reimplement the pooling instance allocation strategy

This commit is a reimplementation of the strategy by which the pooling
instanc

Reimplement the pooling instance allocation strategy (#5661)

* Reimplement the pooling instance allocation strategy

This commit is a reimplementation of the strategy by which the pooling
instance allocator selects a slot for a module. Previously there was a
choice amongst three different algorithms: "reuse affinity", "next
available", and "random". The default was "reuse affinity" but some new
data has come to light which shows that this may not always be a good
default.

Notably the pooling allocator will retain some memory per-slot in the
pooling instance allocator, for example instance data or memory data
if-so-configured. This means that a currently unused, but previously
used, slot can contribute to the RSS usage of a program using Wasmtime.
Consequently the RSS impact here is O(max slots) which can be
counter-intuitive for embedders. This particularly affects "reuse
affinity" because the algorithm for picking a slot when there are no
affine slots is "pick a random slot", which means eventually all slots
will get used.

In discussions about possible ways to tackle this, an alternative to
"pick a strategy" arose and is now implemented in this commit.
Concretely the new allocation algorithm for a slot is now:

* First pick the most recently used affine slot, if one exists.
* Otherwise if the number of affine slots to other modules is above some
threshold N then pick the least-recently used affine slot.
* Otherwise pick a slot that's affine to nothing.

The "N" in this algorithm is configurable and setting it to 0 is the
same as the old "next available" strategy while setting it to infinity
is the same as the "reuse affinity" algorithm. Setting it to something
in the middle provides a knob to allow a modest "cache" of affine slots
while not allowing the total set of slots used to grow too much beyond
the maximal concurrent set of modules. The "random" strategy is now no
longer possible and was removed to help simplify the allocator.

* Resolve rustdoc warnings in `wasmtime-runtime` crate

* Remove `max_cold` as it duplicates the `slot_state.len()`

* More descriptive names

* Add a comment and debug assertion

* Add some list assertions

show more ...

Add support for keeping pooling allocator pages resident (#5207)

When new wasm instances are created repeatedly in high-concurrency
environments one of the largest bottlenecks is the contention on

Add support for keeping pooling allocator pages resident (#5207)

When new wasm instances are created repeatedly in high-concurrency
environments one of the largest bottlenecks is the contention on
kernel-level locks having to do with the virtual memory. It's expected
that usage in this environment is leveraging the pooling instance
allocator with the `memory-init-cow` feature enabled which means that
the kernel level VM lock is acquired in operations such as:

1. Growing a heap with `mprotect` (write lock)
2. Faulting in memory during usage (read lock)
3. Resetting a heap's contents with `madvise` (read lock)
4. Shrinking a heap with `mprotect` when reusing a slot (write lock)

Rapid usage of these operations can lead to detrimental performance
especially on otherwise heavily loaded systems, worsening the more
frequent the above operations are. This commit is aimed at addressing
the (2) case above, reducing the number of page faults that are
fulfilled by the kernel.

Currently these page faults happen for three reasons:

* When memory is first accessed after the heap is grown.
* When the initial linear memory image is accessed for the first time.
* When the initial zero'd heap contents, not part of the linear memory
image, are accessed.

This PR is attempting to address the latter of these cases, and to a
lesser extent the first case as well. Specifically this PR provides the
ability to partially reset a pooled linear memory with `memset` rather
than `madvise`. This is done to have the same effect of resetting
contents to zero but namely has a different effect on paging, notably
keeping the pages resident in memory rather than returning them to the
kernel. This means that reuse of a linear memory slot on a page that was
previously `memset` will not trigger a page fault since everything
remains paged into the process.

The end result is that any access to linear memory which has been
touched by `memset` will no longer page fault on reuse. On more recent
kernels (6.0+) this also means pages which were zero'd by `memset`, made
inaccessible with `PROT_NONE`, and then made accessible again with
`PROT_READ | PROT_WRITE` will not page fault. This can be common when a
wasm instances grows its heap slightly, uses that memory, but then it's
shrunk when the memory is reused for the next instance. Note that this
kernel optimization requires a 6.0+ kernel.

This same optimization is furthermore applied to both async stacks with
the pooling memory allocator in addition to table elements. The defaults
of Wasmtime are not changing with this PR, instead knobs are being
exposed for embedders to turn if they so desire. This is currently being
experimented with at Fastly and I may come back and alter the defaults
of Wasmtime if it seems suitable after our measurements.

show more ...

Refactor configuration for the pooling allocator (#5205)

This commit changes the APIs in the `wasmtime` crate for configuring the
pooling allocator. I plan on adding a few more configuration option

Refactor configuration for the pooling allocator (#5205)

This commit changes the APIs in the `wasmtime` crate for configuring the
pooling allocator. I plan on adding a few more configuration options in
the near future and the current structure was feeling unwieldy for
adding these new abstractions.

The previous `struct`-based API has been replaced with a builder-style
API in a similar shape as to `Config`. This is done to help make it
easier to add more configuration options in the future through adding
more methods as opposed to adding more field which could break prior
initializations.

show more ...