aarch64: Disable csdb emission by default (#12932)

* aarch64: Disable csdb emission by default

This has a massive performance penalty on macOS, for example, and peer
compilers are not emitting this

aarch64: Disable csdb emission by default (#12932)

* aarch64: Disable csdb emission by default

This has a massive performance penalty on macOS, for example, and peer
compilers are not emitting this as part of on-by-default mitigations.
This commit preserves the option to emit it with an aarch64-specific
`use_csdb` flag, but the default is now `false` meaning that this is not
emitted by default.

Closes #12789

* Fix tests

* Fix tests & review comments

* Use ISLE rule introduced

show more ...

Migrate winch-codegen to `wasmtime_environ::error` (#12297)

winch: Simplify constant handling, part 2/N (#10989)

* winch: Introduce register allocation for scratch registers

This commit introduces a register allocator for scratch registers. The
objective of

winch: Simplify constant handling, part 2/N (#10989)

* winch: Introduce register allocation for scratch registers

This commit introduces a register allocator for scratch registers. The
objective of this change is to make it generally safer to work with
scratch registers and prevent accidental clobbering of said registers.

This approach also has the advantage that allows for a more natural
abstraction over ISA-dependent scratch register definitions, e.g., we
can easily encode that fact that in aarch64 x16 and x17 are considered
scratch registers, while in x64 Winch's ABI defines a single global
scratch register.

* winch: Hook up the scratch allocator in each of the backends

This commit makes use of the scratch register allocator in both the
x64 and aarch64 backends for immediate value loading.

Given that the MacroAssembler is the boundary between ISA-agnostic
code and ISA-dependent code, it seems to be the natural location for
this allocator to live.

The allocator gives exclusive access to a scratch register of a
particular class, through the `Masm::with_scratch` method. Note that
the semantics of this allocator don't involve spilling or any other
form of register availability resolution. If a register is requested
and it's not available, this method will panic.

* Apply `cargo fmt`

* Update disassembly tests

Even though the entire change doens't contain major funcitonal
changes, a side effect of improving the constant handling in aarch64
is that we perform better instruction selection for instructions that
deal with immediattes, improving the generated code in some cases.

* Review edits

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Update to wasm-tools 229 (#10601)

* Update to wasm-tools 229

A number of changes here:

* New component model intrinsics (ignored for now, they're
async-related)
* Updated interface for reading b

Update to wasm-tools 229 (#10601)

* Update to wasm-tools 229

A number of changes here:

* New component model intrinsics (ignored for now, they're
async-related)
* Updated interface for reading binary operators, fixing a longstanding
panic in Wasmtime.
* Plumbing for error-context being a gated feature now in the component
model.

prtest:full

* Update to crates.io

show more ...

winch: Gracefully handle unsupported Wasm types (#9949)

* winch: Gracefully handle unsuppported Wasm types

Follow-up to https://github.com/bytecodealliance/wasmtime/pull/9851

Prior to this commit,

winch: Gracefully handle unsupported Wasm types (#9949)

* winch: Gracefully handle unsuppported Wasm types

Follow-up to https://github.com/bytecodealliance/wasmtime/pull/9851

Prior to this commit, Winch's ABI layer would panic on unsupported Wasm
types, i.e., `v128`, `externref`.

This commit ensures that a recoverable error is returned in case an
unsupported type is found in a function signature.

This change is particularly helpful to start running spec tests for
aarch64.

* Fix unit tests for x64/aarch64 abi

show more ...

winch: Gracefully handle compilation errors (#9851)

* winch: Gracefully handle compilation errors

Closes: https://github.com/bytecodealliance/wasmtime/issues/8096

This commit threads `anyhow::Resu

winch: Gracefully handle compilation errors (#9851)

* winch: Gracefully handle compilation errors

Closes: https://github.com/bytecodealliance/wasmtime/issues/8096

This commit threads `anyhow::Result` through most of Winch's
compilation process in order to gracefully handle compilation errors
gracefully instead of panicking.

The error classification is intentionally very granular, to avoid string
allocation which could impact compilation performance.

The errors are largely fit in two categories:

* Unimplemented/Unsupported
* Internal

The firs category signals partial or no support for Wasmtime features
and or Wasm proposals. These errors are meant to be temporary while
such features or proposals are in development.

The second category signals that a compilation invariant was not met.
These errors are considered internal and their presence usually means
a bug in the compiler.

* Include `Result` in the MacroAssembler

This commit updates the MacroAssembler trait to require returning
`Result<T>` on every method in the interface, making it easier to
detect partial support for Masm instructions.

show more ...

Update to target-lexicon 0.13.0 (#9752)

* Update to target-lexicon 0.13.0

Pulling in bytecodealliance/target-lexicon#115 to lay the foundation for
some big-endian work in Pulley

* Fix more compile

Update to target-lexicon 0.13.0 (#9752)

* Update to target-lexicon 0.13.0

Pulling in bytecodealliance/target-lexicon#115 to lay the foundation for
some big-endian work in Pulley

* Fix more compile errors

show more ...

winch: Improve frame handling (#9708)

This commit addresses issues identified while working on issue #8091. It
improves the frame handling in Winch to prevent subtle bugs and enhance
the robustness

winch: Improve frame handling (#9708)

This commit addresses issues identified while working on issue #8091. It
improves the frame handling in Winch to prevent subtle bugs and enhance
the robustness of the code generation process.

Previously, there was no clear mechanism to verify when the frame was
fully set up and safe to access the local slots allocated for register
arguments, including the special slots used for the `VMContext`. As
a result, it was possible to inadvertently read from uninitialized
memory if calls were made before the frame was properly set up and
sealed.

This commit introduces two main changes with the objective to help
reduce the risk of introducing bugs related to the above:

* A `CodeGenPhase` trait, used via the type state pattern to clearly
gate the operations allowed during each phase of the code generation
process.

* Improve the semantics of locals, by clearly separating the notion of
Wasm locals and special locals used by the compiler. This
specialization allows a more accurate representation of the semantics
of Wasm locals and their index space.

show more ...

Remove the use of `MemoryStyle` in Winch (#9581)

This applies a similar change as #9576 but to Winch. Similar to the
previous PR one case needed some reshuffling, but otherwise all current
behavior

Remove the use of `MemoryStyle` in Winch (#9581)

This applies a similar change as #9576 but to Winch. Similar to the
previous PR one case needed some reshuffling, but otherwise all current
behavior is maintained and this should otherwise be a refactoring.

show more ...

More refactoring to remove `MemoryStyle` (#9543)

* Remove tuple return value of `MemorysStyle::for_memory`

The second option is always `tunables.memory_guard_size`, so propagate
this to the various

More refactoring to remove `MemoryStyle` (#9543)

* Remove tuple return value of `MemorysStyle::for_memory`

The second option is always `tunables.memory_guard_size`, so propagate
this to the various locations reading it.

* Remove `offset_guard_size` from compiler `Heap` structs

Push reading `Tunables` down further into where it's needed instead of
having duplicate storage per-heap.

* Plumb wasm memory types further down in Winch

This commit plumbs `wasmtime_environ::Memory` further down the stack in
Winch to where heaps are processed. This avoids an extra layer of
indirection through a `Heap` type which peels apart a `Memory` to pick
out a few fields. In the future this'll be used with more helpers on
`Memory` to simplify the static/dynamic memory cases.

* Plumb memory type further down in Cranelift

This commit removes the `HeapStyle` structure from Cranelift and instead
plumbs the `wasmtime_environ::Memory` type further down the stack
through in `HeapData` (same as Winch before this commit). This removes
redundant fields in `MemoryType` and continues to push down the
`MemoryStyle` structure even further.

This commit additionally and unconditionally defines a `GlobalValue` for
the heap limit of memory. This is unused most of the time for 32-bit
wasm and is conditionally used depending on how bounds checks are
generated. This is a small amount of bloat to each function since
previously functions that didn't need this `GlobalValue` elided it. A
future refactoring, however, will make it a bit more clear how this is
used even for "static" memories.

* Update test expectations

show more ...

Remove the `wasmtime_environ::MemoryPlan` type (#9532)

* Remove the `wasmtime_environ::MemoryPlan` type

This is the equivalent of #9530 for memories. The goal of this commit is
to eventually remove

Remove the `wasmtime_environ::MemoryPlan` type (#9532)

* Remove the `wasmtime_environ::MemoryPlan` type

This is the equivalent of #9530 for memories. The goal of this commit is
to eventually remove the abstraction layer of `MemoryPlan` and
`MemoryStyle` in favor of directly reading the configuration of
`Tunables`. The prediction is that it will be simpler to work directly
with configured values instead of a layer of abstraction between the
configuration and the runtime which needs to be evolved independently to
capture how to interpret the configuration.

Like with #9530 my plan is to eventually remove the `MemoryStyle` type
itself, but that'll be a larger change, so it's deferred to a future
PR.

* Fix shared memory disabled build

show more ...

winch: Implement Fuel-Based Interruption (#9472)

* winch: Implement Fuel-Based Interruption

Closes: https://github.com/bytecodealliance/wasmtime/issues/8090

This commit introduces the initial impl

winch: Implement Fuel-Based Interruption (#9472)

* winch: Implement Fuel-Based Interruption

Closes: https://github.com/bytecodealliance/wasmtime/issues/8090

This commit introduces the initial implementation of fuel-based
interruption in Winch.

To maintain consistency with existing fuel semantics, this
implementation closely follows the Wasmtime/Cranelift approach, with the
following exception:

* Local Fuel Cache: Given Winch's emphasis on compilation speed,
this implementation does not optimize for minimizing loads and stores.
As a result, checking and incrementing fuel currently requires
explicit loads and stores. Future optimizations may be considered to
improve this aspect.

This commit also includes a small refactoring in the visitor, which
introduces more generic "visitor hook" which enable handling the invariants that need
to happen before and after emitting machine code for each Wasm operator.

* Use the vmruntime limits directly

* Remove unsupported fuel warning

show more ...

winch: Remove some `#[allow(dead_code)]` directives (#9480)

This commit simply removes some `#[allow(dead_code)]` which are no
longer needed as well as the the notion of callee-saved registers.

The

winch: Remove some `#[allow(dead_code)]` directives (#9480)

This commit simply removes some `#[allow(dead_code)]` which are no
longer needed as well as the the notion of callee-saved registers.

The notion of callee-saved registers was primarily used in the early
days when Winch generated its own trampolines, however, trampolines are
now emitted through Cranelift.

show more ...

winch(arm64): fpu arithmetics (add, sub, mul, div, min, max) and regalloc (#8365)

* winch(arm64): fpu arithmetics (add, sub, mul, div, min, max)

Signed-off-by: Edoardo Vacchi <[email protected]

winch(arm64): fpu arithmetics (add, sub, mul, div, min, max) and regalloc (#8365)

* winch(arm64): fpu arithmetics (add, sub, mul, div, min, max)

Signed-off-by: Edoardo Vacchi <[email protected]>

* disas: add fpu test cases (add)

Signed-off-by: Edoardo Vacchi <[email protected]>

* winch(arm64): regalloc for fpu

Signed-off-by: Edoardo Vacchi <[email protected]>

* disas: add fpu test cases (add64)

Signed-off-by: Edoardo Vacchi <[email protected]>

* disas: add fpu test cases

Signed-off-by: Edoardo Vacchi <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saúl Cabrera <[email protected]>

* Apply suggestions from code review

Co-authored-by: Saúl Cabrera <[email protected]>

* add todo for f32 mov

Signed-off-by: Edoardo Vacchi <[email protected]>

---------

Signed-off-by: Edoardo Vacchi <[email protected]>
Co-authored-by: Saúl Cabrera <[email protected]>

show more ...

winch: Add support for address maps (#8295)

* winch: Add support for address maps

Closes https://github.com/bytecodealliance/wasmtime/issues/8095

This commit adds support for generating address ma

winch: Add support for address maps (#8295)

* winch: Add support for address maps

Closes https://github.com/bytecodealliance/wasmtime/issues/8095

This commit adds support for generating address maps for Winch.

Give that source code locations and machine code offsets are machine independent, one objective of this change is introduce minimal methods to the MacroAssesmbler and Asssembler implementations and tries to accomodate the bulk of the work in the ISA independent `CodeGen` module.

* Update method documentation to match implementation

show more ...

Migrate all Winch filetests to `tests/disas` (#8243)

* Switch Winch tests to ATT syntax

* Update all test expectations

* Move all winch tests to `disas` folder

* Add `test = "winch"` to `disas`

Migrate all Winch filetests to `tests/disas` (#8243)

* Switch Winch tests to ATT syntax

* Update all test expectations

* Move all winch tests to `disas` folder

* Add `test = "winch"` to `disas`

* Add `test = "winch"` to all winch test files

* Stub out bits to get AArch64 Winch tests working

* Update expectations for all aarch64 winch tests

* Update flags in Winch tests

Use CLI syntax as that's what `flags` was repurposes as in the new test
suite.

* Update all test expectations for x64 winch

* Omit more offsets by default

* Delete now-dead code

* Update an error message

* Update non-winch test expectations

show more ...

Exit through Cranelift-generated trampolines for builtins (#8152)

* Exit through Cranelift-generated trampolines for builtins

This commit changes how builtin functions in Wasmtime (think
`memory.gr

Exit through Cranelift-generated trampolines for builtins (#8152)

* Exit through Cranelift-generated trampolines for builtins

This commit changes how builtin functions in Wasmtime (think
`memory.grow`) are implemented. These functions are required to exit
through some manner of trampoline to handle runtime requirements for
backtracing right now. Currently this is done via inline assembly for
each architecture (or external assembly for s390x). This is a bit
unfortunate as it's a lot of hand-coding and making sure everything is
right, and it's not easy to update as it's multiple platforms to update.

The change in this commit is to instead use Cranelift-generated
trampolines for this purpose instead. The path for invoking a builtin
function now looks like:

* Wasm code calls a statically known symbol for each builtin.
* The statically known symbol will perform exit trampoline duties (e.g.
pc/fp/etc) and then load a function pointer to the host
implementation.
* The host implementation is invoked and then proceeds as usual.

The main new piece for this PR is that all wasm modules and functions
are compiled in parallel but an output of this compilation phase is what
builtin functions are required. All builtin functions are then unioned
together into one set and then anything required is generated just
afterwards. That means that only one builtin-trampoline per-module is
generated per-builtin.

This work is inspired by #8135 and my own personal desire to have as
much about our ABI details flowing through Cranelift as we can. This in
theory makes it more flexible to deal with future improvements to our
ABI.

prtest:full

* Fix some build issues

* Update winch test expectations

* Update Winch to use new builtin shims.

This commit refactors the Winch compiler to use the new trampolines for
all Wasmtime builtins created in the previous commits. This required a
fair bit of refactoring to handle plumbing through a new kind of
relocation and function call.

Winch's `FuncEnv` now contains a `PrimaryMap` from `UserExternalNameRef`
to `UserExternalName`. This is because there's now more than one kind of
name than just wasm function relocations, so the raw index space of
`UserExternalNameRef` is no longer applicable. This required threading
`FuncEnv` to more locations along with some refactorings to ensure that
lifetimes work out ok.

The `CompiledFunction` no longer stores a trait object of how to map
name refs to names and now directly has a `Primarymap`. This also means
that Winch's return value from its `TargetIsa` is a `CompiledFunction`
as opposed to the previous just-a-`MachBuffer` so it can also package up
all the relocation information. This ends up having `winch-codegen`
depend on `wasmtime-cranelift-shared` as a new dependency.

* Review feedback

show more ...

winch: Switch to using cranelift for all trampolines (#8109)

* Switch winch over to using cranelift for all trampolines

* Fix unused code warnings

* Fix unused code warnings

prtest:full

winch: Introduce `TypeConverter` in the function environment (#8052)

This commit introduces the `TypeConverter` struct, used to enable partial borrowing of a `ModuleTranslation` and a `ModuleTypesBu

winch: Introduce `TypeConverter` in the function environment (#8052)

This commit introduces the `TypeConverter` struct, used to enable partial borrowing of a `ModuleTranslation` and a `ModuleTypesBuilder`. This makes it easier to defer type conversions until they are actually needed, for example when resolving callee signatures.
winch: Create converter until it's actually needed.

show more ...

winch: Overhaul the internal ABI (#7974)

* winch: Overhaul the internal ABI

This change overhauls Winch's ABI. This means that as part of this change, the default ABI now closely resembles Craneli

winch: Overhaul the internal ABI (#7974)

* winch: Overhaul the internal ABI

This change overhauls Winch's ABI. This means that as part of this change, the default ABI now closely resembles Cranelift's ABI, particularly on the treatment of the VMContext. This change also fixes many wrong assumptions about trampolines, which are tied to how the previous ABI operated.

The main motivation behind this change is:

* To make it easier to integrate Winch-generated functions with Wasmtime
* Fix fuzz bugs related to imports
* Solidify the implementation regarding the usage of a pinned register to hold the VMContext value throughout the lifetime of a function.


The previous implementation had the following characteristics, and wrong assumptions):

* Assumed that nternal functions don't receive a caller or callee VMContexts as parameters.
* Worked correctly in the following scenarios:
* `Wasm -> Native`: since we can explicitly load the caller and callee `VMContext`, because we're
calling a native import.
* `(Native, Array) -> Wasm`: because the native signatures define a tuple of `VMContext` as arguments.

* It didn't work in the following scenario:
* `Wasm->Wasm`: When calling imports from another WebAssembly instance (via
direct call or `call_indirect`. The previous implementation wrongly assumes
that there should be a trampoline in this case, but there isn't. The code
was generated by the same compiler, so the same ABI should be used in
both functions, but it doesn't.


This change introduces the following changes, which fix the previous assumptions and bugs:

* All internal functions declare a two extra pointer-sized parameters, which will hold the callee and caller `VMContext`s
* Use a pinned register that will be considered live through the lifetime of the function instead of pinning it at the trampoline level. The pinning explicitlly happens when entering the function body and no other assumptions are made from there on.
* Introduce the concept of special `ContextArgs` for function calls. This enum holds metadata about which context arguments are needed depending on the callee. The previous implementation of introducing register values at arbitrary locations in the value stack conflicts with the stack ordering principle which states that older values must *always* precede newer values. So we can't insert a register, because if a spill happens the order of the values will be wrong.


Finally, given that this change also enables the `imports.wast` test suite, it also includes a fix to `global.{get, set}` instructions which didn't account entirely for imported globals.


Resolved conflicts
Update Winch filetests

* Fix typos

* Use `get_wasm_local` and `get_frame_local` instead of `get_local` and `get_local_unchecked`

* Introduce `MAX_CONTEXT_ARGS` and use it in the trampoline to skip context arguments.

show more ...

winch: Optimize calls (#7927)

* winch: Optimize calls

This commit introduces several optimizations to speed up the compilation
of function calls:

* Keep track of previously resolved function signa

winch: Optimize calls (#7927)

* winch: Optimize calls

This commit introduces several optimizations to speed up the compilation
of function calls:

* Keep track of previously resolved function signatures for local or
imported callees to avoid computing the `ABISig` on every
function call.
* Keep track of previously resolved type signatures for indirect calls
to avoid computing the `ABISig` on every function call.
* Refactor `CallKnown` and `CallUnknown` instructions to make the
`BoxCallInfo` field in the struct optional. Prior to this change,
from Winch's perspective each call lowering involved a heap
allocation, using the default values for `BoxCallInfo`, which in the
end are not used by Winch.
* Switch Winch's internal `Stack` to use a `SmallVec` rather than
a `Vec`. Many of the operations involving builtin function calls
require inserting elements at arbitrary offsets in the stack and
using a `SmallVec` makes this process more efficient.

With the changes mentioned above, I observed ~30% improvement in
compilation times for modules that are call-heavy.

* Expect `CallInfo` where applicable and add a comment about the type
definition

* Remove unneeded types and lifetimes

show more ...

winch: Add support for WebAssembly loads/stores (#7894)

* winch: Add support for WebAssembly loads/stores

Closes https://github.com/bytecodealliance/wasmtime/issues/6529

This patch adds support fo

winch: Add support for WebAssembly loads/stores (#7894)

* winch: Add support for WebAssembly loads/stores

Closes https://github.com/bytecodealliance/wasmtime/issues/6529

This patch adds support for all the instructions involving WebAssembly
loads and stores for 32-bit memories. Given that the `memory64` proposal
is not enabled by default, this patch doesn't include an
implementation/tests for it; in theory minimal tweaks to the
currrent implementation will be needed in order to support 64-bit
memories.

Implemenation-wise, this change, follows a similar pattern as Cranelift
in order to calculate addresses for dynamic/static heaps, the main
difference being that in some cases, doing less work at compile time is
preferred; the current implemenation only checks for the general case of
out-of-bounds access for dynamic heaps for example.

Another important detail regarding the implementation, is the
introduction of `MacroAssembler::wasm_load` and
`MacroAssembler::wasm_store`, which internally use a common
implemenation for loads and stores, with the only difference that the
`wasm_*` variants set the right flags in order to signal that these
operations are not trusted and might trap.

Finally, given that this change introduces support for the last set of
instructions missing for a Wasm MVP, it removes most of Winch's copy of
the spectest suite, and switches over to using the official test suite
where possible (for tests that don't use SIMD or Reference Types).

Follow-up items:

* Before doing any deep benchmarking I'm planning on landing a couple of
improvements regarding compile times that I've identified in parallel
to this change.
* The `imports.wast` tests are disabled because I've identified a bug
with `call_indirect`, which is not related to this change and exists
in main.
* Find a way to run the `tests/all/memory.rs` (or perhaps most of
integration tests) with Winch.

--
prtest:full

* Review comments

show more ...

winch: Emit unwind info in the x64 backend (#7798)

* Enable all winch tests on windows

prtest:mingw-x64

* Plumb through x64 unwind info creation

* Add the frame regs unwind info

* Emit UnwindInf

winch: Emit unwind info in the x64 backend (#7798)

* Enable all winch tests on windows

prtest:mingw-x64

* Plumb through x64 unwind info creation

* Add the frame regs unwind info

* Emit UnwindInfo::SaveReg instructions

* Review feedback

* Comment the offset_downward_to_clobbers value

show more ...

Update the wasm-tools family of crates (#7587)

This commit updates to the latest wasm-tools and `wit-bindgen` to bring
the family of crates forward. This update notably includes Nick's work
on packe

Update the wasm-tools family of crates (#7587)

This commit updates to the latest wasm-tools and `wit-bindgen` to bring
the family of crates forward. This update notably includes Nick's work
on packed indices in the `wasmparser` crate for validation for the
upcoming implementation of GC types. This meant that translation from
`wasmparser` types to Wasmtime types now may work with a "type id"
instead of just a type index which required plumbing not only Wasmtime's
own type information but additionally `wasmparser`'s type information
throughout translation.

This required a fair bit of refactoring to get this working but no
change in functionality is intended, only a different way of doing
everything prior.

show more ...