Insert veneers on arm64 in cranelift-jit where necessary (#12239)

* Remove unnecessary methods from JITModule Module impl

* Fix range assertion in cranelift-jit Arm64Call reloc handling

* Rename e

Insert veneers on arm64 in cranelift-jit where necessary (#12239)

* Remove unnecessary methods from JITModule Module impl

* Fix range assertion in cranelift-jit Arm64Call reloc handling

* Rename exception_data to wasmtime_exception_data

It is a wasmtime specific format that doesn't match what cranelift-jit
could register with the system unwinder in the future.

* Move memory allocation into CompiledBlob

In preparation for inserting veneers on arm64.

* Respect addend for Arm64Call relocs

* Insert veneers on arm64 in cranelift-jit where necessary

* Rename const item

show more ...

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable AB

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`.

As discussed in this week's Cranelift meeting, we've discovered a need
to generalize the `patchable_call` mechanism and corresponding
`patchable` ABI slightly. In particular, we will need patchable
`try_call` callsites as well in order to allow breakpoint handlers to
throw exceptions (desirable functionality eventually) and have this work
in the presence of inlining. Also, it's just a nice generalization to
say that patchability is an orthogonal dimension to the call ABI and the
other restrictions we initially imposed, and works as long as the basic
requirement (no return values) is met.

This also renames the `patchable` ABI to `preserve_all`, to make it
clear that its purpose is actually orthogonal, and it can be used
independently of patchable callsites. It also deletes the `cold` ABI,
which never actually did anything and is misleading in the presence of
an actual cold-ish (subzero temperature, actually) ABI like
`preserve_all`.

* Review feedback.

show more ...

Check that symbols that needs to be defined are defined in cranelift-object (#11740)

They aren't defined, you get weird errors and crashes at link time.

Allow marking data objects as used for the linker (#11206)

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Change `allow(missing_docs)` to `expect(..)` (#10384)

This wasn't possible when `expect` was first introduced due to a change
being required in upstream rust-lang/rust. That change
rust-lang/rust#13

Change `allow(missing_docs)` to `expect(..)` (#10384)

This wasn't possible when `expect` was first introduced due to a change
being required in upstream rust-lang/rust. That change
rust-lang/rust#130025) has now rode enough trains to be in our MSRV, so
we can expect missing docs now instead of just allowing it.

show more ...

Remove Function argument from define_function_bytes (#10346)

* Remove Function argument from define_function_bytes

define_function_bytes is supposed to be used when the code was not
compiled by Cra

Remove Function argument from define_function_bytes (#10346)

* Remove Function argument from define_function_bytes

define_function_bytes is supposed to be used when the code was not
compiled by Cranelift and thus wouldn't have any associated Function.

* Avoid collecting the relocations into a Vec twice

show more ...

Burn down the `allow_attributes_without_reason` backlog (#9712)

* Burn down the `allow_attributes_without_reason` backlog

Just a bit, not everything fixed.

* Fix wasi-nn annotations

* Tweak `#[cf

Burn down the `allow_attributes_without_reason` backlog (#9712)

* Burn down the `allow_attributes_without_reason` backlog

Just a bit, not everything fixed.

* Fix wasi-nn annotations

* Tweak `#[cfg]`

show more ...

Improve support for `Module` trait objects (#9173)

* Improve support for `Module` trait objects

* Implement `Module` for `Box<M: Module>`

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

cranelift: Group labels by FuncId instead of Name (#7183)

This prevents confusing label locations when there are multiple functions
with the same Name.

riscv64: Implement ELF TLS GD Relocations (#7003)

* cranelift: Add support for public labels

* cranelift: Allow targeting labels with relocations

* cranelift: Emit label related relocations in mo

riscv64: Implement ELF TLS GD Relocations (#7003)

* cranelift: Add support for public labels

* cranelift: Allow targeting labels with relocations

* cranelift: Emit label related relocations in module

* riscv64: Implement TLS GD

* cranelift: Rename `label_is_public` field

* cranelift: Avoid making MachLabel part of the exposed API

show more ...

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parall

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parallel to `serde_derive`, allowing it to finish compilation possibly
even before `serde_derive`, unblocking all the crates waiting for
`serde` to start compiling much sooner.

As it turns out the main deciding factor for how long the compile time of a
project is, is primarly determined by the depth of dependencies rather
than the width. In other words, a crate's compile times aren't affected
by how many crates it depends on, but rather by the longest chain of
dependencies that it needs to wait on. In many cases `serde` is part of
that long chain, as it is part of a long chain if the `derive` feature
is active:

`proc-macro2` compile build script > `proc-macro2` run build script >
`proc-macro2` > `quote` > `syn` > `serde_derive` > `serde` >
`serde_json` (or any crate that depends on serde)

By decoupling it from `serde_derive`, the chain is shortened and compile
times get much better.

Check this issue for a deeper elaboration:
https://github.com/serde-rs/serde/issues/2584

For `wasmtime` I'm seeing a reduction from 24.75s to 22.45s when
compiling in `release` mode. This is because wasmtime through `gimli`
has a dependency on `indexmap` which can only start compiling when
`serde` is finished, which you want to happen as early as possible so
some of wasmtime's dependencies can start compiling.

To measure the full effect, the dependencies can't by themselves
activate the `derive` feature. I've upstreamed a patch for
`fxprof-processed-profile` which was the only dependency that activated
it for `wasmtime` (not yet published to crates.io). `wasmtime-cli` and
co. may need patches for their dependencies to see a similar
improvement.

show more ...

Allow serializing all cranelift-module data structures (#6172)

* Remove ModuleCompiledFunction

The same information can be retrieved using

ctx.compiled_code().unwrap().code_info().total_size

In a

Allow serializing all cranelift-module data structures (#6172)

* Remove ModuleCompiledFunction

The same information can be retrieved using

ctx.compiled_code().unwrap().code_info().total_size

In addition for Module implementations that don't immediately compile the
given function there is no correct value that can be returned.

* Don't give anonymous functions and data objects an internal name

This internal name can conflict if a module is serialized and then
deserialized into another module. It also wasn't used by any of the
Module implementations anyway.

* Allow serializing all cranelift-module data structures

This allows a Module implementation to serialize it's internal state and
deserialize it in another compilation session. For example to implement
LTO or to load the module into cranelift-interpreter.

* Use expect

show more ...

Remove the DataContext wrapper around DataDescription (#6170)

* Remove the DataContext wrapper around DataDescription

It doesn't have much of a purpose while making it harder to for example
rewrite

Remove the DataContext wrapper around DataDescription (#6170)

* Remove the DataContext wrapper around DataDescription

It doesn't have much of a purpose while making it harder to for example
rewrite the function and data object declarations within it as is
necessary for deserializing a serialized module.

* Derive Debug for DataDescription

show more ...

Rename define_function to define_function_with_control_plane (#6165)

And add a define_function convenience function which uses a default
control plane.

Chaos mode MVP: Skip branch optimization in MachBuffer (#6039)

* fuzz: Add chaos mode control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Wa

Chaos mode MVP: Skip branch optimization in MachBuffer (#6039)

* fuzz: Add chaos mode control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: Skip branch optimization with chaos mode

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: Rename chaos engine -> control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* chaos mode: refactoring ControlPlane to be passed through the call stack by reference

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Remo Senekowitsch <[email protected]>

* fuzz: annotate chaos todos

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: cleanup control plane

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: remove control plane from compiler context

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: move control plane into emit state

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fuzz: fix remaining compiler errors

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* fix tests

* refactor emission state ctrl plane accessors

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* centralize conditional compilation of chaos mode

Also cleanup a few straggling dependencies on cranelift-control
that aren't needed anymore.

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* add cranelift-control to published crates

prtest:full

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* add cranelift-control to public crates

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

---------

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>
Co-authored-by: Remo Senekowitsch <[email protected]>

show more ...

Add `JITBuilder::with_flags` constructor (#5751)

This allows custom flags to be set (e.g. `opt-level`) while still
leaving most of of the boilerplate to select the native target to
the `JITBuilder

Add `JITBuilder::with_flags` constructor (#5751)

This allows custom flags to be set (e.g. `opt-level`) while still
leaving most of of the boilerplate to select the native target to
the `JITBuilder`.

show more ...

cranelift: improve syscall error/oom handling in JIT module (#5173)

* cranelift: improve syscall error/oom handling in JIT module

The JIT module has several places where it `expect`s or `panic`s

cranelift: improve syscall error/oom handling in JIT module (#5173)

* cranelift: improve syscall error/oom handling in JIT module

The JIT module has several places where it `expect`s or `panic`s
on syscall or allocator errors. For example, `mmap` and `mprotect`
can fail if Linux `vm.max_map_count` is not high enough, and some
users may wish to handle this error rather than immediately
crashing.

This commit plumbs these errors upward as new `ModuleError`
types, so that callers of jit module functions like
`finalize_definitions` and `define_function` can handle them
(or just `unwrap()`, as desired).

* cranelift: Remove ModuleError::Syscall variant

Syscall errors can just be folded into the generic Backend error,
which is an anyhow::Error

* cranelift-jit: return io::ErrorKind::OutOfMemory for alloc failure

Just using `io::Error::last_os_error()` is not correct as global
allocator impls are not required to set errno

show more ...

Export a couple of types from cranelift_module that were meant to be exported (#5074)

Align functions according to their ISA's requirements (#4826)

Add a function_alignment function to the TargetIsa trait, and use it to align functions when generating objects. Additionally, collect t

Align functions according to their ISA's requirements (#4826)

Add a function_alignment function to the TargetIsa trait, and use it to align functions when generating objects. Additionally, collect the maximum alignment required for pc-relative constants in functions and pass that value out. Use the max of these two values when padding functions for alignment.

This fixes a bug on x86_64 where rip-relative loads to sse registers could cause a segfault, as functions weren't always guaranteed to be aligned to 16-byte addresses.

Fixes #4812

show more ...

[cranelift] Re-add missing blanket implementations for `&mut Module` (#4719)

Implement an incremental compilation cache for Cranelift (#4551)

This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested b

Implement an incremental compilation cache for Cranelift (#4551)

This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested by @cfallin (thanks!) in Cranelift, and trait object to provide a backend for an all-included experience in Wasmtime.

After the suggestion of Chris, `Function` has been split into mostly two parts:

- on the one hand, `FunctionStencil` contains all the fields required during compilation, and that act as a compilation cache key: if two function stencils are the same, then the result of their compilation (`CompiledCodeBase<Stencil>`) will be the same. This makes caching trivial, as the only thing to cache is the `FunctionStencil`.
- on the other hand, `FunctionParameters` contain the... function parameters that are required to finalize the result of compilation into a `CompiledCode` (aka `CompiledCodeBase<Final>`) with proper final relocations etc., by applying fixups and so on.

Most changes are here to accomodate those requirements, in particular that `FunctionStencil` should be `Hash`able to be used as a key in the cache:

- most source locations are now relative to a base source location in the function, and as such they're encoded as `RelSourceLoc` in the `FunctionStencil`. This required changes so that there's no need to explicitly mark a `SourceLoc` as the base source location, it's automatically detected instead the first time a non-default `SourceLoc` is set.
- user-defined external names in the `FunctionStencil` (aka before this patch `ExternalName::User { namespace, index }`) are now references into an external table of `UserExternalNameRef -> UserExternalName`, present in the `FunctionParameters`, and must be explicitly declared using `Function::declare_imported_user_function`.
- some refactorings have been made for function names:
- `ExternalName` was used as the type for a `Function`'s name; while it thus allowed `ExternalName::Libcall` in this place, this would have been quite confusing to use it there. Instead, a new enum `UserFuncName` is introduced for this name, that's either a user-defined function name (the above `UserExternalName`) or a test case name.
- The future of `ExternalName` is likely to become a full reference into the `FunctionParameters`'s mapping, instead of being "either a handle for user-defined external names, or the thing itself for other variants". I'm running out of time to do this, and this is not trivial as it implies touching ISLE which I'm less familiar with.

The cache computes a sha256 hash of the `FunctionStencil`, and uses this as the cache key. No equality check (using `PartialEq`) is performed in addition to the hash being the same, as we hope that this is sufficient data to avoid collisions.

A basic fuzz target has been introduced that tries to do the bare minimum:

- check that a function successfully compiled and cached will be also successfully reloaded from the cache, and returns the exact same function.
- check that a trivial modification in the external mapping of `UserExternalNameRef -> UserExternalName` hits the cache, and that other modifications don't hit the cache.
- This last check is less efficient and less likely to happen, so probably should be rethought a bit.

Thanks to both @alexcrichton and @cfallin for your very useful feedback on Zulip.

Some numbers show that for a large wasm module we're using internally, this is a 20% compile-time speedup, because so many `FunctionStencil`s are the same, even within a single module. For a group of modules that have a lot of code in common, we get hit rates up to 70% when they're used together. When a single function changes in a wasm module, every other function is reloaded; that's still slower than I expect (between 10% and 50% of the overall compile time), so there's likely room for improvement.

Fixes #4155.

show more ...

[cranelift] Rejigger the `compile` API (#4540)

* Move `emit_to_memory` to `MachCompileResult`

This small refactoring makes it clearer to me that emitting to memory
doesn't require anything else

[cranelift] Rejigger the `compile` API (#4540)

* Move `emit_to_memory` to `MachCompileResult`

This small refactoring makes it clearer to me that emitting to memory
doesn't require anything else from the compilation `Context`. While it's
a trivial change, it's a small public API change that shouldn't cause
too much trouble, and doesn't seem RFC-worthy. Happy to hear different
opinions about this, though!

* hide the MachCompileResult behind a method

* Add a `CompileError` wrapper type that references a `Function`

* Rename MachCompileResult to CompiledCode

* Additionally remove the last unsafe API in cranelift-codegen

show more ...

Remove all Sink traits