S390x: emit new instructions added in z17 (#12319)

* s390x: Emit instructions from MIE4 & VXRS_EXT3 on z17

This emits & tests a bunch of instructions:
* from Miscellaneous-Instruction-Extensions

S390x: emit new instructions added in z17 (#12319)

* s390x: Emit instructions from MIE4 & VXRS_EXT3 on z17

This emits & tests a bunch of instructions:
* from Miscellaneous-Instruction-Extensions Facility 4:
* CLZ, 64bit
* CTZ, 64bit
* from Vector-Enhancements Facility 3:
* 32x4, 64x2 & 128x1 variants of the following:
* Divide
* Remainder
* 64x2 & 128x1 multiply variants
* 128x1 vaiants of:
* Compare
* CLZ
* CTZ
* Max
* Min
* Average
* Negation
* Evaluate

Co-authored-by: Jimmy Brisson <[email protected]>

* s390x: Emit vector blend on z17

* Rename x86_blendv to blendv

Now that s390x implements blendv as well, we should refer to
the instruction without the x86 prefix.

---------

Co-authored-by: Ulrich Weigand <[email protected]>

show more ...

fuzzgen: Don't generate float unordered vector ops on aarch64 (#12442)

This is a refinement of the change in #12196 to fix fuzzgen on oss-fuzz
because aarch64 doesn't currently have lowerings for th

fuzzgen: Don't generate float unordered vector ops on aarch64 (#12442)

This is a refinement of the change in #12196 to fix fuzzgen on oss-fuzz
because aarch64 doesn't currently have lowerings for these float
orderings, only the scalar versions were added in #12196.

show more ...

Add cranelift fcmp ult, ule, ugt, uge for aarch64 (#12196)

* Run cranelift fcmp ord and uno tests on aarch64

FloatCC::Ordered and FloatCC:Unordered are already implemented for
aarch64, so these can

Add cranelift fcmp ult, ule, ugt, uge for aarch64 (#12196)

* Run cranelift fcmp ord and uno tests on aarch64

FloatCC::Ordered and FloatCC:Unordered are already implemented for
aarch64, so these can be enabled as they already pass.

* Run more cranelift fcmp tests on aarch64

These currently fail as they are not implemented yet.

* Add cranelift fcmp ult, ule, ugt, uge for aarch64

show more ...

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable AB

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`.

As discussed in this week's Cranelift meeting, we've discovered a need
to generalize the `patchable_call` mechanism and corresponding
`patchable` ABI slightly. In particular, we will need patchable
`try_call` callsites as well in order to allow breakpoint handlers to
throw exceptions (desirable functionality eventually) and have this work
in the presence of inlining. Also, it's just a nice generalization to
say that patchability is an orthogonal dimension to the call ABI and the
other restrictions we initially imposed, and works as long as the basic
requirement (no return values) is met.

This also renames the `patchable` ABI to `preserve_all`, to make it
clear that its purpose is actually orthogonal, and it can be used
independently of patchable callsites. It also deletes the `cold` ABI,
which never actually did anything and is misleading in the presence of
an actual cold-ish (subzero temperature, actually) ABI like
`preserve_all`.

* Review feedback.

show more ...

Cranelift: support fuzzing `patchable_call`. (#12118)

This allows the opcode for call-format instructions and verifies that we
are calling it with the patchable ABI.

Fixes https://oss-fuzz.com/test

Cranelift: support fuzzing `patchable_call`. (#12118)

This allows the opcode for call-format instructions and verifies that we
are calling it with the patchable ABI.

Fixes https://oss-fuzz.com/testcase-detail/6014638929281024.

show more ...

Never use `colocated` symbols in `cranelift-fuzzgen` (#11872)

As I've just discovered it turns out that all functions are in their own
mmap, and mmaps aren't guaranteed to be anywhere near one anoth

Never use `colocated` symbols in `cranelift-fuzzgen` (#11872)

As I've just discovered it turns out that all functions are in their own
mmap, and mmaps aren't guaranteed to be anywhere near one another. This
means that functions cannot ever be `colocated` and must always have
absolute relocations to one another.

show more ...

Don't fuzz `sequence_point` in `cranelift-fuzzgen` (#11831)

It's not implemented in the interpreter right now so add it to the list
of opcodes to exclude.

Cranelift: add get_exception_handler_address. (#11629)

* Cranelift: add get_exception_handler_address.

This is designed to enable applications such as #11592 that use
alternative unwinding mechanis

Cranelift: add get_exception_handler_address. (#11629)

* Cranelift: add get_exception_handler_address.

This is designed to enable applications such as #11592 that use
alternative unwinding mechanisms that may not necessarily want to walk a
stack and look up exception tables. The idea is that whenever it would
be valid to resume to an exception handler that is active on the stack,
we can provide the same PC as a first-class runtime value that would be
found in the exception table for the given handler edge. A "custom"
resume step can then use this PC as a resume-point as long as it follows
the relevant exception ABI (i.e.: restore SP, FP, any other saved
registers that the exception ABI specifies, and provide appropriate
payload value(s)).

Handlers are associated with edges out of `try_call`s (or
`try_call_indirect`s); and edges specifically, not blocks, because there
could be multiple out-edges to one block. The instruction thus takes the
block that contains the try-call and an immediate that indexes its
exceptional edges.

This CLIF instruction required a bit of infrastructure to (i) allow
naming raw blocks, not just block calls, as instruction arguments, and
(ii) allow getting the MachLabel for any other lowered block during
lowering. But given that, the lowerings themselves are straightforward
uses of MachBuffer labels to fix-up PC-relative address-loading
instructions (e.g., `LEA` or `ADR` or `AUIPC`+`ADDI`).

* Review feedback.

* Review feedback: more tests.

show more ...

Never colocate libcalls in cranelift-fuzzgen (#11335)

I think that this is the cause of a fuzz-bug on OSS-Fuzz and also
explains why I can't reproduce locally. Why it's just now reproducing is
somet

Never colocate libcalls in cranelift-fuzzgen (#11335)

I think that this is the cause of a fuzz-bug on OSS-Fuzz and also
explains why I can't reproduce locally. Why it's just now reproducing is
something I'm not sure of but I believe this has always been an issue.
Effectively ensure that cranelift-fuzzgen never flags libcalls as
`colocated` because we don't know where the libcalls will be relative to
the finalized generated code.

show more ...

`cranelift-frontend`: Refactor how variables are declared (#10965)

* cranelift-frontend: Refactor how variables are declared

Rather than requiring users to construct a `Variable` entity out of thin

`cranelift-frontend`: Refactor how variables are declared (#10965)

* cranelift-frontend: Refactor how variables are declared

Rather than requiring users to construct a `Variable` entity out of thin air
and *then* declare it to the frontend, declaring a new variable now returns the
new `Variable` entity. And instead of having a `SecondaryMap<Variable, Type>`
inside of `FunctionBuilder` we now have a `PrimaryMap<Variable, Type>`. Using
variables in `cranelift-frontend` should generally feel a little bit better and
more idiomatic now.

* Fix doc test

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect`

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect` instructions, and
lowerings on four of five ISAs (x86-64, aarch64, riscv64, pulley; s390x
has its own non-shared ABI code that will need separate work).

It extends CLIF to support these instructions as new kinds of branches,
and extends block-calls to accept `retN` and `exnN` block-call args that
carry the normal return values or exception payloads (respectively) into
the appropriate successor blocks.

It wires up the "normal return path" so that it continues to work.
It updates the ABI so that unwinding is possible without an initial
register state at throw: specifically, as per our RFC, all registers are
clobbered. It also includes metadata in the `MachBuffer` that describes
exception-catch destinations. However, no unwinder exists to interpret
these catch-destinations yet, so they are untested.

* Add try_call_indirect lowering as well.

show more ...

Leverage Rust 1.79, 1.80 (#9498)

This commit is a follow-up to #9496 to leverage various APIs that the
workspace now has access to. For example most dependencies on the
`once_cell` crate are now rem

Leverage Rust 1.79, 1.80 (#9498)

This commit is a follow-up to #9496 to leverage various APIs that the
workspace now has access to. For example most dependencies on the
`once_cell` crate are now removed in favor of the types stabilized in
the standard library: `LazyLock` and `LazyCell`. One dependency remains
in the `wasmtime` crate due to the `get_or_try_init` not being stable
yet.

Some additional helper methods on raw pointer slices are also available
for removing a few minor `unsafe` blocks.

show more ...

Don't generate 128-bit atomics in fuzzing (#9474)

This commit fixes some issues cropping up due to fuzzing 128-bit atomics
in `cranelift-{icache,fuzzgen}`. Not all backends support these
operations

Don't generate 128-bit atomics in fuzzing (#9474)

This commit fixes some issues cropping up due to fuzzing 128-bit atomics
in `cranelift-{icache,fuzzgen}`. Not all backends support these
operations and x64 only supports them with a feature enabled, so for now
disable fuzzing until it's ready to fuzz later.

show more ...

Disable the `stack_switch` instruction in `cranelift-fuzzgen` (#9226)

An OSS-Fuzz bug came in last night about `stack_switch` not being
supported on s390x, understandably so. This commit disables th

Disable the `stack_switch` instruction in `cranelift-fuzzgen` (#9226)

An OSS-Fuzz bug came in last night about `stack_switch` not being
supported on s390x, understandably so. This commit disables the
instruction entirely in fuzzing because it requires precise runtime
support which the fuzzer doesn't support at this time.

show more ...

Remove `iadd_cin` and `isub_bin`, split `isub_borrow` and `iadd_carry` (#9199)

* Remove `iadd_cin` and `isub_bin`, split `isub_borrow` and `iadd_carry`

This commit refactors the opcodes the Craneli

Remove `iadd_cin` and `isub_bin`, split `isub_borrow` and `iadd_carry` (#9199)

* Remove `iadd_cin` and `isub_bin`, split `isub_borrow` and `iadd_carry`

This commit refactors the opcodes the Cranelift supports for
add-with-carry and subtract-with-borrow. None of these opcodes are
currently in use by the wasm frontend nor supported by any backend. In
that sense it's unlikely they have many users and the hope is that
refactoring won't cause much impact.

The `iadd_cin` and `isub_bin` opcodes are the equivalent of `*_borrow`
and `*_carry` except that they don't return the carry flag, they only
return the result of the operation. While theoretically useful I've
elected to remove them here in favor of only the borrow-returning
operations. They can be added back in in the future though if necessary.

I've split the preexisting operations `isub_borrow` and `iadd_carry`
additionally into signed/unsigned portions:

* `isub_borrow` => `usub_borrow` and `ssub_borrow`
* `iadd_carry` => `uadd_carry` and `sadd_carry`

This reflects how the condition needs to differ on the carry flag
computation for signed/unsigned inputs. I've additionally fixed the
interpreter's implementation of `IsubBorrow` when switching to the
signed/unsigned opcodes.

Finally the documentation for these instructions now explicitly say that
the incoming carry/borrow is zero-or-nonzero even though it's typed as
`i8`. Additionally the tests have been refactored to make use of
multi-return which may not have existed when they were first written.

* Rename instructions

* Fix more renames

* Update instruction descriptions

show more ...

Remove reference types (`r32` and `r64`) from Cranelift (#9164)

* Remove reference types (`r32` and `r64`) from Cranelift

* restore fuzz regression test

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

fuzzgen: Add stack map variables (#8941)

cranelift: Add `f16const` and `f128const` instructions (#8893)

* cranelift: Add `f16const` and `f128const` instructions

* cranelift: Add constant propagation for `f16` and `f128`

Cranelift: Remove resumable traps (#8809)

These were originally a SpiderMonkey-ism and have been unused ever
since. It was introduced for GC integration, where the runtime could do
something to make

Cranelift: Remove resumable traps (#8809)

These were originally a SpiderMonkey-ism and have been unused ever
since. It was introduced for GC integration, where the runtime could do
something to make Cranelift code hit a trap and pause for a GC and then resume
execution once GC completed. But it is unclear that, as implemented, this is
actually a useful mechanism for doing that (compared to, say, loading from some
Well Known page and the GC protecting that page and catching signals to
interrupt the mutator, or simply branching and doing a libcall). And if someone
has that particular use case in the future (Wasmtime and its GC integration
doesn't need exactly this) then we can design something for what is actually
needed at that time, instead of carrying this cruft forward forever.

show more ...

riscv64: Support scalar-vector bitcasts (#8692)

* riscv64: Implement scalar to vector bitcast and vice-versa

* riscv64: Delete `vfslide1up.vf` instruction

* fuzzgen: Enable `bitcast.i128` for RISC

riscv64: Support scalar-vector bitcasts (#8692)

* riscv64: Implement scalar to vector bitcast and vice-versa

* riscv64: Delete `vfslide1up.vf` instruction

* fuzzgen: Enable `bitcast.i128` for RISC-V

show more ...

fuzzgen: Add support for stack slot alignment (#8650)

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

* crates/test-programs typos

* crates/c-api typos

* crates/cache typos

* crates other typos

* cranelift/codegen/src/isa typos

* cranelift/codegen/src other typos

* cranelift/codegen other typos

* cranelift other typos

* ci js typo

* .github workflows typo

* RELEASES typo

* Fix clang-format documentation line

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

Cranelift: add alignment parameter to stack slots. (#8635)

* Cranelift: add alignment parameter to stack slots.

Fixes #6716.

Currently, stack slots on the stack are aligned only to a machine-word

Cranelift: add alignment parameter to stack slots. (#8635)

* Cranelift: add alignment parameter to stack slots.

Fixes #6716.

Currently, stack slots on the stack are aligned only to a machine-word
boundary. This is insufficient for some use-cases: for example, storing
SIMD data or structs that require a larger alignment.

This PR adds a parameter to the `StackSlotData` to specify alignment,
and the associated logic to the CLIF parser and printer. It updates the
shared ABI code to compute the stackslot layout taking the alignment
into account. In order to ensure the alignment is always a power of two,
it is stored as a shift amount (log2 of actual alignment) in the IR.

* Apply suggestions from code review

Co-authored-by: Trevor Elliott <[email protected]>

* Update filetest.

* Update alignment to ValRaw vector.

* Fix printer test.

* cargo-fmt from suggestion update.

---------

Co-authored-by: Trevor Elliott <[email protected]>

show more ...