cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conver

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conversions for hashing the TargetIsa

* Remove func_body_len

It is identical to buffer.data.len()

* Introduce IsaFlagsHashKey

show more ...

Bump MSRV to 1.86.0 (#11142)

Coupled with today's release of 1.88.0. Also update our nightly version
used in CI to the most recent nightly.

Update nightly used in CI (#10957)

A new lint was added to rustc so this updates the nightly used in CI and
then additionally fixes the lints that are firing.

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Fix more `clippy::doc_lazy_continuation` lints (#9105)

Switch from `bincode` to `postcard` for serializing (#8476)

Wasmtime and Cranelift have a few miscellaenous use cases for "just take
this Rust type and make it bytes", for example Wasmtime's seriali

Switch from `bincode` to `postcard` for serializing (#8476)

Wasmtime and Cranelift have a few miscellaenous use cases for "just take
this Rust type and make it bytes", for example Wasmtime's serialization
of internal metadata into a compiled module. Previously Wasmtime used
the `bincode` crate for performing these tasks as the format was
generally optimized to be small and fast, not general purpose (e.g.
JSON). The `bincode` crate on crates.io doesn't work on `no_std`,
however, and with the work in #8341 that's an issue now for Wasmtime.

This crate switches instead to the `postcard` crate. This crate is
listed in Serde's documentation as:

> Postcard, a no_std and embedded-systems friendly compact binary
> format.

While I've not personally used it before it checks all the boxes we
relied on `bincode` for and additionally works with `no_std`. After
auditing the crate this commit then switches out Wasmtime's usage of
`bincode` for `postcard` throughout the repository.

show more ...

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parall

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parallel to `serde_derive`, allowing it to finish compilation possibly
even before `serde_derive`, unblocking all the crates waiting for
`serde` to start compiling much sooner.

As it turns out the main deciding factor for how long the compile time of a
project is, is primarly determined by the depth of dependencies rather
than the width. In other words, a crate's compile times aren't affected
by how many crates it depends on, but rather by the longest chain of
dependencies that it needs to wait on. In many cases `serde` is part of
that long chain, as it is part of a long chain if the `derive` feature
is active:

`proc-macro2` compile build script > `proc-macro2` run build script >
`proc-macro2` > `quote` > `syn` > `serde_derive` > `serde` >
`serde_json` (or any crate that depends on serde)

By decoupling it from `serde_derive`, the chain is shortened and compile
times get much better.

Check this issue for a deeper elaboration:
https://github.com/serde-rs/serde/issues/2584

For `wasmtime` I'm seeing a reduction from 24.75s to 22.45s when
compiling in `release` mode. This is because wasmtime through `gimli`
has a dependency on `indexmap` which can only start compiling when
`serde` is finished, which you want to happen as early as possible so
some of wasmtime's dependencies can start compiling.

To measure the full effect, the dependencies can't by themselves
activate the `derive` feature. I've upstreamed a patch for
`fxprof-processed-profile` which was the only dependency that activated
it for `wasmtime` (not yet published to crates.io). `wasmtime-cli` and
co. may need patches for their dependencies to see a similar
improvement.

show more ...

Chaos mode MVP: Skip branch optimization in MachBuffer (#6039)

* fuzz: Add chaos mode control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Wa

Chaos mode MVP: Skip branch optimization in MachBuffer (#6039)

* fuzz: Add chaos mode control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: Skip branch optimization with chaos mode

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: Rename chaos engine -> control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* chaos mode: refactoring ControlPlane to be passed through the call stack by reference

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Remo Senekowitsch <[email protected]>

* fuzz: annotate chaos todos

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: cleanup control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: remove control plane from compiler context

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: move control plane into emit state

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: fix remaining compiler errors

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fix tests

* refactor emission state ctrl plane accessors

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* centralize conditional compilation of chaos mode

Also cleanup a few straggling dependencies on cranelift-control
that aren't needed anymore.

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* add cranelift-control to published crates

prtest:full

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* add cranelift-control to public crates

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

---------

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>
Co-authored-by: Remo Senekowitsch <[email protected]>

show more ...

Ensure the sequence number doesn't leak out of Layout (#6061)

Previously it could affect the PartialEq and Hash impls. Ignoring the
sequence number in PartialEq and Hash allows us to not renumber al

Ensure the sequence number doesn't leak out of Layout (#6061)

Previously it could affect the PartialEq and Hash impls. Ignoring the
sequence number in PartialEq and Hash allows us to not renumber all
blocks in the incremental cache.

show more ...

Properly use the VersionMarker in CachedFunc (#6062)

Make sequence numbers local to instructions (#6043)

* Only allow pp_cmp within a single block

Block order shouldn't matter for codegen and restricting pp_cmp to a
single block will allow making ins

Make sequence numbers local to instructions (#6043)

* Only allow pp_cmp within a single block

Block order shouldn't matter for codegen and restricting pp_cmp to a
single block will allow making instruction sequence numbers local to a
block.

* Make sequence numbers local to instructions

This allows renumbering to be localized to a single block where previously it
could affect the entire function. Also saves 32bit of overhead per block.

show more ...

Implement an incremental compilation cache for Cranelift (#4551)

This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested b

Implement an incremental compilation cache for Cranelift (#4551)

This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested by @cfallin (thanks!) in Cranelift, and trait object to provide a backend for an all-included experience in Wasmtime.

After the suggestion of Chris, `Function` has been split into mostly two parts:

- on the one hand, `FunctionStencil` contains all the fields required during compilation, and that act as a compilation cache key: if two function stencils are the same, then the result of their compilation (`CompiledCodeBase<Stencil>`) will be the same. This makes caching trivial, as the only thing to cache is the `FunctionStencil`.
- on the other hand, `FunctionParameters` contain the... function parameters that are required to finalize the result of compilation into a `CompiledCode` (aka `CompiledCodeBase<Final>`) with proper final relocations etc., by applying fixups and so on.

Most changes are here to accomodate those requirements, in particular that `FunctionStencil` should be `Hash`able to be used as a key in the cache:

- most source locations are now relative to a base source location in the function, and as such they're encoded as `RelSourceLoc` in the `FunctionStencil`. This required changes so that there's no need to explicitly mark a `SourceLoc` as the base source location, it's automatically detected instead the first time a non-default `SourceLoc` is set.
- user-defined external names in the `FunctionStencil` (aka before this patch `ExternalName::User { namespace, index }`) are now references into an external table of `UserExternalNameRef -> UserExternalName`, present in the `FunctionParameters`, and must be explicitly declared using `Function::declare_imported_user_function`.
- some refactorings have been made for function names:
- `ExternalName` was used as the type for a `Function`'s name; while it thus allowed `ExternalName::Libcall` in this place, this would have been quite confusing to use it there. Instead, a new enum `UserFuncName` is introduced for this name, that's either a user-defined function name (the above `UserExternalName`) or a test case name.
- The future of `ExternalName` is likely to become a full reference into the `FunctionParameters`'s mapping, instead of being "either a handle for user-defined external names, or the thing itself for other variants". I'm running out of time to do this, and this is not trivial as it implies touching ISLE which I'm less familiar with.

The cache computes a sha256 hash of the `FunctionStencil`, and uses this as the cache key. No equality check (using `PartialEq`) is performed in addition to the hash being the same, as we hope that this is sufficient data to avoid collisions.

A basic fuzz target has been introduced that tries to do the bare minimum:

- check that a function successfully compiled and cached will be also successfully reloaded from the cache, and returns the exact same function.
- check that a trivial modification in the external mapping of `UserExternalNameRef -> UserExternalName` hits the cache, and that other modifications don't hit the cache.
- This last check is less efficient and less likely to happen, so probably should be rethought a bit.

Thanks to both @alexcrichton and @cfallin for your very useful feedback on Zulip.

Some numbers show that for a large wasm module we're using internally, this is a 20% compile-time speedup, because so many `FunctionStencil`s are the same, even within a single module. For a group of modules that have a lot of code in common, we get hit rates up to 70% when they're used together. When a single function changes in a wasm module, every other function is reloaded; that's still slower than I expect (between 10% and 50% of the overall compile time), so there's likely room for improvement.

Fixes #4155.

show more ...