Migrate winch-codegen to `wasmtime_environ::error` (#12297)

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Winch: Add replace_lane instructions for x64 with AVX (#10082)

* Winch: Add replace_lane instructions for x64 with AVX

* Add check for AVX

* Make operand size conversion a helper function

Winch: i8x16.shuffle for x64 with AVX (#9959)

* i8x16_shuffle for x64 with AVX512

* Use AVX2 instructions instead

* Change panic to error

* Forgot to update callsite

* Ignore SIMD misc test on n

Winch: i8x16.shuffle for x64 with AVX (#9959)

* i8x16_shuffle for x64 with AVX512

* Use AVX2 instructions instead

* Change panic to error

* Forgot to update callsite

* Ignore SIMD misc test on non-AVX hardware

* Add xmm_ prefix to function name

* Use early exit in shuffle

show more ...

winch: Gracefully handle compilation errors (#9851)

* winch: Gracefully handle compilation errors

Closes: https://github.com/bytecodealliance/wasmtime/issues/8096

This commit threads `anyhow::Resu

winch: Gracefully handle compilation errors (#9851)

* winch: Gracefully handle compilation errors

Closes: https://github.com/bytecodealliance/wasmtime/issues/8096

This commit threads `anyhow::Result` through most of Winch's
compilation process in order to gracefully handle compilation errors
gracefully instead of panicking.

The error classification is intentionally very granular, to avoid string
allocation which could impact compilation performance.

The errors are largely fit in two categories:

* Unimplemented/Unsupported
* Internal

The firs category signals partial or no support for Wasmtime features
and or Wasm proposals. These errors are meant to be temporary while
such features or proposals are in development.

The second category signals that a compilation invariant was not met.
These errors are considered internal and their presence usually means
a bug in the compiler.

* Include `Result` in the MacroAssembler

This commit updates the MacroAssembler trait to require returning
`Result<T>` on every method in the interface, making it easier to
detect partial support for Masm instructions.

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

Add v128.const support to Winch (#8990)

* Add v128.const support to Winch

* Remove next_vr and vector_reg_for methods

* Adjust alignment and slot size for v128

* Forgot to update disas tests

* U

Add v128.const support to Winch (#8990)

* Add v128.const support to Winch

* Remove next_vr and vector_reg_for methods

* Adjust alignment and slot size for v128

* Forgot to update disas tests

* Update unit tests

* Use 8 byte stack slot sizes

* Fix broken unit tests, add tests for vecs, and use ty size for vecs

show more ...

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

* crates/test-programs typos

* crates/c-api typos

* crates/cache typos

* crates other typos

* cranelift/codegen/src/isa typos

* cranelift/codegen/src other typos

* cranelift/codegen other typos

* cranelift other typos

* ci js typo

* .github workflows typo

* RELEASES typo

* Fix clang-format documentation line

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

winch: Optimize calls (#7927)

* winch: Optimize calls

This commit introduces several optimizations to speed up the compilation
of function calls:

* Keep track of previously resolved function signa

winch: Optimize calls (#7927)

* winch: Optimize calls

This commit introduces several optimizations to speed up the compilation
of function calls:

* Keep track of previously resolved function signatures for local or
imported callees to avoid computing the `ABISig` on every
function call.
* Keep track of previously resolved type signatures for indirect calls
to avoid computing the `ABISig` on every function call.
* Refactor `CallKnown` and `CallUnknown` instructions to make the
`BoxCallInfo` field in the struct optional. Prior to this change,
from Winch's perspective each call lowering involved a heap
allocation, using the default values for `BoxCallInfo`, which in the
end are not used by Winch.
* Switch Winch's internal `Stack` to use a `SmallVec` rather than
a `Vec`. Many of the operations involving builtin function calls
require inserting elements at arbitrary offsets in the stack and
using a `SmallVec` makes this process more efficient.

With the changes mentioned above, I observed ~30% improvement in
compilation times for modules that are call-heavy.

* Expect `CallInfo` where applicable and add a comment about the type
definition

* Remove unneeded types and lifetimes

show more ...

Rename `WasmType` to `WasmValType` (#7838)

Purely mechanical, no functional changes.

This is to help differentiate between value types (i32, i64, reference types,
etc...) and defined types (functio

Rename `WasmType` to `WasmValType` (#7838)

Purely mechanical, no functional changes.

This is to help differentiate between value types (i32, i64, reference types,
etc...) and defined types (function signatures, struct definitions, array
definitions).

show more ...

winch: Multi-Value Part 2: Blocks (#7707)

* winch: Multi-Value Part 2: Blocks

This commit adds support for the Multi-Value proposal for blocks.

In general, this change, introduces multiple buildin

winch: Multi-Value Part 2: Blocks (#7707)

* winch: Multi-Value Part 2: Blocks

This commit adds support for the Multi-Value proposal for blocks.

In general, this change, introduces multiple building blocks to enable
supporting arbitrary params and results in blocks:

* `BlockType`: Introduce a block type, to categorize the type of each
block, this makes it easier to categorize blocks per type and also
makes it possible to defer the calculation of the `ABIResults` until
they are actually needed rather than calculating everyghing upfront
even though they might not be needed (when in an unreachable state).
* Push/pop operations are now frame aware. Given that each
`ControlStackFrame` contains all the information needed regarding
params and results, this change moves the the implementation of the
push and pop operations to the `ControlStackFrame` struct.
* `StackState`: this struct holds the entry and exit invariants of each
block; these invariants are pre-computed when entering the block and
used throughout the code generation, to handle params, results and
assert the respective invariants.

In terms of the mechanics of the implementation: when entering each
block, if there are results on the stack, the expected stack pointer
offsets will be calculated via the `StackState`, and the `target_offset`
will be used to create the block's `RetArea`. Note that when entering
the block and calculating the `StackState` no space is actually reserved
for the results, any space increase in the stack is deffered until the
results are popped from the value stack via
`ControlStackFrame::pop_abi_results`.

The trickiest bit of the implementation is handling constant values that
need to be placed on the right location on the machine stack. Given that
constants are generally not spilled, this means that in order to keep
the machine and value stack in sync (spilled-values-wise), values must
be shuffled to ensure that constants are placed in the expected location results wise.
See the comment in `ControlStackFrame::adjust_stack_results` for more
details.

* Review fixes

show more ...

Switch winch's internal stack from a VecDeque to a Vec (#7756)

It doesn't look like which needs to modify the front of the `VecDeque`
that it uses to model the value stack, and can instead use a `Ve

Switch winch's internal stack from a VecDeque to a Vec (#7756)

It doesn't look like which needs to modify the front of the `VecDeque`
that it uses to model the value stack, and can instead use a `Vec`.

show more ...

Winch: Change emits to return TypedReg to put on stack (#7682)

Winch: fix bug by spilling when emitting func call (#7573)

* Winch: fix bug by spilling when calling a func

* Forgot to commit new filetest

* Only support WasmHeapType::Func

* Elaborate on call_i

Winch: fix bug by spilling when emitting func call (#7573)

* Winch: fix bug by spilling when calling a func

* Forgot to commit new filetest

* Only support WasmHeapType::Func

* Elaborate on call_indirect jump details

* Update docs for call

* Verify stack is only consts and memory entries

show more ...

winch: Solidify unreachable code handling (#7547)

This commit solidifies the approach for unreachable code handling in
control flow.

Prior to this change, at unconditional jump sites, the compiler

winch: Solidify unreachable code handling (#7547)

This commit solidifies the approach for unreachable code handling in
control flow.

Prior to this change, at unconditional jump sites, the compiler would
reset the machine stack as well as the value stack. Even though this
appoach might seem natural at first, it actually broke several of the
invariants that must be met at the end of each contol block, this was
specially noticeable with programs that conditionally entered in an
unreachable state, like for example

```wat
(module
(func (;0;) (param i32) (result i32)
local.get 0
local.get 0
if (result i32)
i32.const 1
return
else
i32.const 2
end
i32.sub
)
(export "main" (func 0))
)
```

The approach followed in this commit ensures that all the invariants are
met and introduces more guardrails around those invariants. In short,
instead of resetting the value stack at unconditional jump sites, the
value stack handling is deferred until the reachability analysis
restores the reachability of the code generation process, ensuring that
the value stack contains the exact amount of values expected by the
frame where reachability is restored. Given that unconditional jumps
reset the machine stack, when the reachability of the code generation
process is restored, the SP offset is also restored which should match
the size of the value stack.

show more ...

winch: Multi-Value Part 1 (#7535)

* winch: Introduce `ABIParams` and `ABIResults`

This commit prepares Winch to support WebAssembly Multi-Value.

The most notorious piece of this change is the int

winch: Multi-Value Part 1 (#7535)

* winch: Introduce `ABIParams` and `ABIResults`

This commit prepares Winch to support WebAssembly Multi-Value.

The most notorious piece of this change is the introduction of the
`ABIParams` and `ABIResults` structs which are type wrappers around the
concept of an `ABIOperand`, which is the underlying main representation
of a param or result.

This change also consolidates how the size for WebAssembly types is
derived by introducing `ABI::sizeof`, as well as introducing
`ABI::stack_slot_size` to concretely indicate the stack slot size in
bytes for stack params, which is ABI dependent.

* winch: Add the necessary ABI building blocks for multi-value

This change adds the necessary changes at the ABI level in order to
handle multi-value.

The most notable modifications in this change are:

* Modifying Winch's default ABI to reverse the order of results,
ensuring that results that go in the stack should always come first;
this makes it easier to respect the following two stack invariants:

* Spilled memory values always precede register values
* Spilled values are stored from oldest to newest, matching their
respective locations on the machine stack.

* Modify all calling conventions supported by Winch so that only one result, the first one is stored in
registers. This differs from their vanilla counterparts in that these
ABIs can handle multiple results in registers. Given that Winch is not
a generic code generator, keeping the ABI close to what Wasmtime
expects makes it easier to pass multiple results at trampolines.

* Add more multi-value tests

This commit adds more tests for multi-value and improves documentation.

prtest:full

* Address review feedback

show more ...

winch: Materialize latent locals when setting them (#7531)

* winch: Materialize latent locals when setting them

* Put method on stack and stop when encountering mem entry

winch: Add a subset of known libcalls and improve call emission (#7228)

* winch: Add known a subset of known libcalls and improve call emission

This change is a follow up to:
- https://github.com/b

winch: Add a subset of known libcalls and improve call emission (#7228)

* winch: Add known a subset of known libcalls and improve call emission

This change is a follow up to:
- https://github.com/bytecodealliance/wasmtime/pull/7155
- https://github.com/bytecodealliance/wasmtime/pull/7035

One of the objectives of this change is to make it easy to emit
function calls at the MacroAssembler layer, for cases in which it's
challenging to know ahead-of-time if a particular functionality can be
achieved natively (e.g. rounding and SSE4.2). The original implementation
of function call emission, made this objective difficult to achieve and
it was also difficult to reason about.
I decided to simplify the overall approach to function calls as part of
this PR; in essence, the `call` module now exposes a single function
`FnCall::emit` which is reponsible of gathtering the dependencies and
orchestrating the emission of the call. This new approach deliberately
avoids holding any state regarding the function call for simplicity.

This change also standardizes the usage of `Callee` as the main
entrypoint for function call emission, as of this change 4 `Callee`
types exist (`Local`, `Builtin`, `Import`, `FuncRef`), each callee kind
is mappable to a `CalleeKind` which is the materialized version of
a callee which Cranelift understands.

This change also moves the creation of the `BuiltinFunctions` to the
`ISA` level given that they can be safely used accross multiple function
compilations.

Finally, this change also introduces support for some of the
"well-known" libcalls and hooks those libcalls at the
`MacroAssembler::float_round` callsite.

--

prtest:full

* Review comments

* Remove unnecessary `into_iter`

* Fix remaining lifetime parameter names

show more ...

winch(x64): Add support for table instructions (#7155)

* winch(x64): Add support for table instructions

This change adds support for the following table insructions:
`elem.drop`, `table.copy`, `tab

winch(x64): Add support for table instructions (#7155)

* winch(x64): Add support for table instructions

This change adds support for the following table insructions:
`elem.drop`, `table.copy`, `table.set`, `table.get`, `table.fill`,
`table.grow`, `table.size`, `table.init`.

This change also introduces partial support for the `Ref` WebAssembly
type, more conretely the `Func` heap type, which means that all the
table instructions above, only work this WebAssembly type as of this
change.

Finally, this change is also a small follow up to the primitives
introduced in https://github.com/bytecodealliance/wasmtime/pull/7100,
more concretely:

* `FnCall::with_lib`: tracks the presence of a libcall and ensures that
any result registers are freed right when the call is emitted.
* `MacroAssembler::table_elem_addr` returns an address rather than the
value of the address, making it convenient for other use cases like
`table.set`.

--

prtest:full

* chore: Make stack functions take impl IntoIterator<..>

* Update winch/codegen/src/codegen/call.rs

Co-authored-by: Trevor Elliott <[email protected]>

* Remove a dangling `dbg!`

* Add comment on branching

---------

Co-authored-by: Trevor Elliott <[email protected]>

show more ...

winch(x64): Call indirect (#7100)

* winch(x64): Call indirect

This change adds support for the `call_indirect` instruction to Winch.

Libcalls are a pre-requisite for supporting `call_indirect` in

winch(x64): Call indirect (#7100)

* winch(x64): Call indirect

This change adds support for the `call_indirect` instruction to Winch.

Libcalls are a pre-requisite for supporting `call_indirect` in order to
lazily initialy funcrefs. This change adds support for libcalls to
Winch by introducing a `BuiltinFunctions` struct similar to Cranelift's
`BuiltinFunctionSignatures` struct.

In general, libcalls are handled like any other function call, with the
only difference that given that not all the information to fulfill the
function call might be known up-front, control is given to the caller
for finalizing the call.

The introduction of function references also involves dealing with
pointer-sized loads and stores, so this change also adds the required
functionality to `FuncEnv` and `MacroAssembler` to be pointer aware,
making it straight forward to derive an `OperandSize` or `WasmType` from
the target's pointer size.

Finally, given the complexity of the call_indirect instrunction, this
change bundles an improvement to the register allocator, allowing it to
track the allocatable vs non-allocatable registers, this is done to
avoid any mistakes when allocating/de-allocating registers that are not
alloctable.

--
prtest:full

* Address review comments

* Fix typos
* Better documentation for `new_unchecked`
* Introduce `max` for `BitSet`
* Make allocatable property `u64`

* winch(calls): Overhaul `FnCall`

This commit simplifies `FnCall`'s interface making its usage more
uniform throughout the compiler. In summary, this change:

* Avoids side effects in the `FnCall::new` constructor, and also makes
it the only constructor.
* Exposes `FnCall::save_live_registers` and
`FnCall::calculate_call_stack_space` to calculate the stack space
consumed by the call and so that the caller can decide which one to
use at callsites depending on their use-case.

* tests: Fix regset tests

show more ...

winch: Initial support for floats (#6860)

* winch: Initial support for floats

This change introuduces the necessary building blocks to support floats in
Winch as well as support for both `f32.const

winch: Initial support for floats (#6860)

* winch: Initial support for floats

This change introuduces the necessary building blocks to support floats in
Winch as well as support for both `f32.const` and `f64.const` instructions.

To achieve support for floats, this change adds several key enhancements to the
compiler:

* Constant pool: A constant pool is implemented, at the Assembler level, using the machinery
exposed by Cranelift's `VCode` and `MachBuffer`. Float immediates are stored
using their bit representation in the value stack, and whenever they are
used at the MacroAssembler level they are added to the constant
pool, from that point on, they are referenced through a `Constant` addressing
mode, which gets translated to a RIP-relative addressing mode during emission.

* More precise value tagging: aside from immediates, from which the type can
be easily inferred, all the other value stack entries (`Memory`, `Reg`, and `Local`) are
modified to explicitly contain a WebAssembly type. This allows for better
instruction selection.

--

prtest:full

* fix: Account for relative sp position when pushing float regs

This was an oversight of the initial implementation. When pushing float
registers, always return an address that is relative to the current position of
the stack pointer, essentially storing to (%rsp). The previous implementation
accounted for static addresses, which is not correct.

* fix: Introduce `stack_arg_slot_size_for_type`

To correctly calculate the stack argument slot sizes, instead of overallocating
for `word_bytes`, since for `f32` floating points we only need to worry about
loading/storing 4 bytes.

* fix: Correctly type the result register.

The previous version wrongly typed the register as a general purpose register.

* refactor: Re-write `add_constants` through `add_constant`

* docs: Replace old comment

* chore: Rust fmt

* refactor: Index regset per register class

This commit implements `std::ops::{Index, IndexMut}` for `RegSet` to index each
of the bitsets by class. This reduces boilerplate and repetition throuhg the
code generation context, register allocator and register set.

* refactor: Correctly size callee saved registers

To comply with the expectation of the underlying architecture: for example in
Aarch64, only the low 64 bits of VRegs are callee saved (the D-view) and in the
`fastcall` calling convention it's expected that the callee saves the entire 128
bits of the register xmm6-xmm15.

This change also fixes the the stores/loads of callee saved float registers in the
fastcall calling convention, as in the previous implementation only the low 64
bits were saved/restored.

* docs: Add comment regarding typed-based spills

show more ...

winch(x64) Add support for if/else (#6550)

* winch(x64) Add support for if/else

This change adds the necessary building blocks to support control flow;
this change also adds support for the `If` /

winch(x64) Add support for if/else (#6550)

* winch(x64) Add support for if/else

This change adds the necessary building blocks to support control flow;
this change also adds support for the `If` / `Else` operators.

This change does not include multi-value support. The idea is to add
support for multi-value across the compiler (functions and blocks) as
a separate future change.

The general gist of the change is to track the presence of control flow
frames as part of the code generation context and emit the corresponding
labels as and instructions as control flow blocks are found.

* PR review

* Allocate 64 slots for `ControlStackFrames`
* Explicitly track else branches through an else entry in
`ControlStackFrame`

show more ...

winch: Implement new trampolines (#6358)

* winch: Implement new trampolines

This change is a follow-up to
https://github.com/bytecodealliance/wasmtime/pull/6262, in which the new
trampolines, descr

winch: Implement new trampolines (#6358)

* winch: Implement new trampolines

This change is a follow-up to
https://github.com/bytecodealliance/wasmtime/pull/6262, in which the new
trampolines, described [here](https://github.com/bytecodealliance/rfcs/blob/main/accepted/tail-calls.md#new-trampolines-and-vmcallercheckedanyfunc-changes),
were introduced to Wasmtime.

This change, focuses on the `array-to-wasm`,
`native-to-wasm` and `wasm-to-native` trampolines to restore Winch's
working state prior to the introduction of the new trampolines. It's
worth noting that the new approach for trampolines make it easier to support
the `TypedFunc` API in Winch. Prior to the introduction of the new
trampolines, it was not obvious how to approach it.

This change also introduces a pinned register that will hold the
`VMContext` pointer, which is loaded in the `*-to-wasm` trampolines;
the `VMContext` register is a pre-requisite to this change to support
the `wasm-to-native` trampolines.

Lastly, with the introduction of the `VMContext` register and the
`wasm-to-native` trampolines, this change also introduces support for
calling function imports, which is a variation of the already existing
calls to locally defined functions.

The other notable piece of this change aside from the trampolines is
`winch-codegen`'s dependency on `wasmtime-environ`. Winch is so closely
tied to the concepts exposed by the wasmtime crates that it makes sense
to tie them together, even though the separation provides some
advantages like easier testing in some cases, in the long run, there's
probably going to be less need to test Winch in isolation and rather
we'd rely more on integration style tests which require all of Wasmtime
pieces anyway (fuzzing, spec tests, etc).

This change doesn't update the existing implmenetation of
`winch_codegen::FuncEnv`, but the intention is to update that part after
this change.

prtest:full

* tests: Ignore miri in Winch integration tests

* Remove hardcoded alignment and addend

show more ...

winch(x64): Initial implementation for function calls (#6067)

* winch(x64): Initial implementation for function calls

This change adds the main building blocks for calling locally defined
functions

winch(x64): Initial implementation for function calls (#6067)

* winch(x64): Initial implementation for function calls

This change adds the main building blocks for calling locally defined
functions. Support for function imports will be added iteratively after this
change lands and once trampolines are supported.

To support function calls, this change introduces the following functionality to
the MacroAssembler:

* `pop` to pop the machine stack into a given register, which in the case of
this change, translates to the x64 pop instruction.

* `call` to a emit a call to locally defined functions.

* `address_from_sp` to construct memory addresses with the SP as a base.

* `free_stack` to emit the necessary instrunctions to claim stack space.

The heavy lifting of setting up and emitting the function call is done through
the implementation of `FnCall`.

* Fix spill behaviour in function calls and add more documentation

This commits adds a more detailed documentation to the `call.rs` module.

It also fixes a couple of bugs, mainly:

* The previous commit didn't account for memory addresses used as arguments for
the function call, any memory entry in the value stack used as a function
argument should be tracked and then used to claim that memory when the function
call ends. We could `pop` and do this implicitly, but we can also track this
down and emit a single instruction to decrement the stack pointer, which will
result in better code.

* Introduce a differentiator between addresses relative or absolute to the stack
pointer. When passing arguments in the stack -- assuming that SP at that point
is aligned for the function call -- we should store the arguments relative to
the absolute position of the stack pointer and when addressing a memory entry in
the Wasm value stack, we should use an address relative to the offset and the
position of the stack pointer.

* Simplify tracking of the stack space needed for emitting a function call

show more ...

winch: Add full support for integer `sub` and `add` instructions (#5737)

This patch adds complete support for the `sub` and `add` WebAssembly instructions
for x64, and complete support for the `add

winch: Add full support for integer `sub` and `add` instructions (#5737)

This patch adds complete support for the `sub` and `add` WebAssembly instructions
for x64, and complete support for the `add` WebAssembly instruction for aarch64.

This patch also refactors how the binary operations get constructed within the
`VisitOperator` trait implementation. The refactor adds methods in the
`CodeGenContext` to abstract all the common steps to emit binary operations,
making this process less repetitive and less brittle (e.g. omitting to push the resulting value
to the stack, or omitting to free registers after used).

This patch also improves test coverage and refactors the filetests directory to make it
easier to add tests for other instructions.

show more ...