wasmtime-cache: use `wasmtime_environ::Error` instead of `anyhow` (#12210)

CacheConfigs are always enabled; clarify directory validation (#11376)

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as wel

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as well.

* Update dependency directive in `Cargo.toml`

show more ...

wasmtime: Fix rendering of `Cache` and `CacheConfig` docs (#11015)

Use the setting name rather than `$setting` for documentation
rendering.

More clearly flag internal crates as such (#10963)

* More clearly flag internal crates as such

This commit is an attempt to more clearly flag internal crates in this
project as internal and not int

More clearly flag internal crates as such (#10963)

* More clearly flag internal crates as such

This commit is an attempt to more clearly flag internal crates in this
project as internal and not intended for external use. Specifically:

* Many crates are renamed from `wasmtime-foo` to
`wasmtime-internal-foo`.
* All of these crates now have `INTERNAL: ...` in their crates.io
description.
* All of these crates now have a warning at the top of their
documentation discouraging use.

This change is a result of rustsec/advisory-db#1999 where the goal is to
be crystal clear from a project perspective that usage of these crates
are highly discouraged and not supported. We'll still probably get such
advisories but we won't be considering them CVEs from the project itself
due to the internal nature of these crates and the discouraging
warnings.

Some concrete changes used here are:

* Inter-crate dependencies still use `wasmtime_foo` for naming and do
so with Cargo's package-renaming features.
* Crate renames are specified at the workspace level so the rename is
only in one locations and all other inherit it.
* Contribution documentation now has some brief guidelines about crate
organization.

* Update vet config

* Update checks for wasmtime-fiber

prtest:full

* Update publish script

* Another fiber rename

* Fix some doc tests

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Allow creation of a CacheConfig without loading from a file (#10665)

* cache: Add builder pattern for CacheConfig

* wasmtime: Add cache_config method to wasmtime::Config

* Refactor test_builder_de

Allow creation of a CacheConfig without loading from a file (#10665)

* cache: Add builder pattern for CacheConfig

* wasmtime: Add cache_config method to wasmtime::Config

* Refactor test_builder_default to use test_prolog helper

* Remove enabled option from CacheConfigBuilder and always set to true

* Change builder methods to take &mut self and return &mut Self

* Simplify cache configuration API

A new `cache_config(Option<CacheConfig>)` method replaces multiple methods
for controlling module caching. Now `None` disables caching, and users can
directly provide a cache config or load one from a file.

* Make cache configuration optional

* Add Cache struct to separate configuration from runtime (wip)

* Ensure default values earlier

* Consolidate CacheConfig and CacheConfigBuilder

* Set Cache directly on wastime::Config and make it easier to create one
from a file

* Validate after loading file again

* Move cache to top-level module

* Fix tests

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

Enable rustc's `unused-lifetimes` lint (#8711)

* Enable rustc's `unused-lifetimes` lint

This is allow-by-default doesn't seem to have any false positives in
Wasmtime's codebase so enable it by defa

Enable rustc's `unused-lifetimes` lint (#8711)

* Enable rustc's `unused-lifetimes` lint

This is allow-by-default doesn't seem to have any false positives in
Wasmtime's codebase so enable it by default to help clean up vestiges of
old refactorings.

* Remove another unused lifetime

* Remove another unused lifetime

show more ...

Switch from `bincode` to `postcard` for serializing (#8476)

Wasmtime and Cranelift have a few miscellaenous use cases for "just take
this Rust type and make it bytes", for example Wasmtime's seriali

Switch from `bincode` to `postcard` for serializing (#8476)

Wasmtime and Cranelift have a few miscellaenous use cases for "just take
this Rust type and make it bytes", for example Wasmtime's serialization
of internal metadata into a compiled module. Previously Wasmtime used
the `bincode` crate for performing these tasks as the format was
generally optimized to be small and fast, not general purpose (e.g.
JSON). The `bincode` crate on crates.io doesn't work on `no_std`,
however, and with the work in #8341 that's an issue now for Wasmtime.

This crate switches instead to the `postcard` crate. This crate is
listed in Serde's documentation as:

> Postcard, a no_std and embedded-systems friendly compact binary
> format.

While I've not personally used it before it checks all the boxes we
relied on `bincode` for and additionally works with `no_std`. After
auditing the crate this commit then switches out Wasmtime's usage of
`bincode` for `postcard` throughout the repository.

show more ...

crates/cache: print warning information only when the renaming truly fails (#7497)

Update base64 dependency to 0.21.0 (#5702)

* Update base64 to 0.21.0

* Update code for base64 0.21.0

Use an mmap-friendly serialization format (#3257)

* Use an mmap-friendly serialization format

This commit reimplements the main serialization format for Wasmtime's
precompiled artifacts. Previou

Use an mmap-friendly serialization format (#3257)

* Use an mmap-friendly serialization format

This commit reimplements the main serialization format for Wasmtime's
precompiled artifacts. Previously they were generally a binary blob of
`bincode`-encoded metadata prefixed with some versioning information.
The downside of this format, though, is that loading a precompiled
artifact required pushing all information through `bincode`. This is
inefficient when some data, such as trap/address tables, are rarely
accessed.

The new format added in this commit is one which is designed to be
`mmap`-friendly. This means that the relevant parts of the precompiled
artifact are already page-aligned for updating permissions of pieces
here and there. Additionally the artifact is optimized so that if data
is rarely read then we can delay reading it until necessary.

The new artifact format for serialized modules is an ELF file. This is
not a public API guarantee, so it cannot be relied upon. In the meantime
though this is quite useful for exploring precompiled modules with
standard tooling like `objdump`. The ELF file is already constructed as
part of module compilation, and this is the main contents of the
serialized artifact.

THere is some extra information, though, not encoded in each module's
individual ELF file such as type information. This information continues
to be `bincode`-encoded, but it's intended to be much smaller and much
faster to deserialize. This extra information is appended to the end of
the ELF file. This means that the original ELF file is still a valid ELF
file, we just get to have extra bits at the end. More information on the
new format can be found in the module docs of the serialization module
of Wasmtime.

Another refatoring implemented as part of this commit is to deserialize
and store object files directly in `mmap`-backed storage. This avoids
the need to copy bytes after the artifact is loaded into memory for each
compiled module, and in a future commit it opens up the door to avoiding
copying the text section into a `CodeMemory`. For now, though, the main
change is that copies are not necessary when loading from a precompiled
compilation artifact once the artifact is itself in mmap-based memory.

To assist with managing `mmap`-based memory a new `MmapVec` type was
added to `wasmtime_jit` which acts as a form of `Vec<T>` backed by a
`wasmtime_runtime::Mmap`. This type notably supports `drain(..N)` to
slice the buffer into disjoint regions that are all separately owned,
such as having a separately owned window into one artifact for all
object files contained within.

Finally this commit implements a small refactoring in `wasmtime-cache`
to use the standard artifact format for cache entries rather than a
bincode-encoded version. This required some more hooks for
serializing/deserializing but otherwise the crate still performs as
before.

* Review comments

show more ...

wasmtime: Option to disable parallel compilation (#3169)

* Introduce parallel-compilation configuration switch

* Plumb parallel_compilation config to compilation

* Adjust obj.rs

* Address r

wasmtime: Option to disable parallel compilation (#3169)

* Introduce parallel-compilation configuration switch

* Plumb parallel_compilation config to compilation

* Adjust obj.rs

* Address review

* Fix compilation fail in `cache` crate

* Fix obj.rs

Also remove the now unneeded feature in /Cargo.toml

* fmt

show more ...

Don't hash for the cache if it's disabled (#2816)

This fixes an issue where even if the wasmtime cache was disabled we're
still calculating the sha256 of modules for the hash key. This hash
was th

Don't hash for the cache if it's disabled (#2816)

This fixes an issue where even if the wasmtime cache was disabled we're
still calculating the sha256 of modules for the hash key. This hash
was then simply discarded if the cache was disabled!

show more ...

Code review feedback changes.

* Add `anyhow` dependency to `wasmtime-runtime`.
* Revert `get_data` back to `fn`.
* Remove `DataInitializer` and box the data in `Module` translation instead.
* Improv

Code review feedback changes.

* Add `anyhow` dependency to `wasmtime-runtime`.
* Revert `get_data` back to `fn`.
* Remove `DataInitializer` and box the data in `Module` translation instead.
* Improve comments on `MemoryInitialization`.
* Remove `MemoryInitialization::OutOfBounds` in favor of proper bulk memory
semantics.
* Use segmented memory initialization except for when the uffd feature is
enabled on Linux.
* Validate modules with the allocator after translation.
* Updated various functions in the runtime to return `anyhow::Result`.
* Use a slice when copying pages instead of `ptr::copy_nonoverlapping`.
* Remove unnecessary casts in `OnDemandAllocator::deallocate`.
* Better document the `uffd` feature.
* Use WebAssembly page-sized pages in the paged initialization.
* Remove the stack pool from the uffd handler and simply protect just the guard
pages.

show more ...

Allow instance allocators control over module compilation.

This commit introduces two new methods on `InstanceAllocator`:

* `validate_module` - this method is used to validate a module after
tran

Allow instance allocators control over module compilation.

This commit introduces two new methods on `InstanceAllocator`:

* `validate_module` - this method is used to validate a module after
translation but before compilation. It will be used for the upcoming pooling
allocator to ensure a module being compiled adheres to the limits of the
allocator.

* `adjust_tunables` - this method is used to adjust the `Tunables` given the
JIT compiler. The pooling allocator will use this to force all memories to
be static during compilation.

show more ...

Update immediate and transitive dependencies

I don't think this has happened in awhile but I've run a `cargo update`
as well as trimming some of the duplicate/older dependencies in
`Cargo.lock` by u

Update immediate and transitive dependencies

I don't think this has happened in awhile but I've run a `cargo update`
as well as trimming some of the duplicate/older dependencies in
`Cargo.lock` by updating some of our immediate dependencies as well.

show more ...

Making caching support optional in Wasmtime (#2119)

This commit moves all of the caching support that currently lives in
`wasmtime-environ` into a `wasmtime-cache` crate and makes it optional. The

Making caching support optional in Wasmtime (#2119)

This commit moves all of the caching support that currently lives in
`wasmtime-environ` into a `wasmtime-cache` crate and makes it optional. The
goal here is to slim down the `wasmtime-environ` crate and clearly separate
boundaries where caching is a standalone and optional feature, not intertwined
with other crates.

show more ...