Preserve `try_call[_indirect]` stack maps during lowering (#12934)

* Preserve `try_call[_indirect]` stack maps during lowering

Branch instructions are skipped in the main lowering loop, which means

Preserve `try_call[_indirect]` stack maps during lowering (#12934)

* Preserve `try_call[_indirect]` stack maps during lowering

Branch instructions are skipped in the main lowering loop, which means the stack
map forwarding code is never reached for them. The branch lowering path didn't
forward stack maps either. This was fine because branch instructions couldn't
previously ever be safepoints. However, with the introduction of `try_call` and
`try_call_indirect`, we now have instructions that are both safepoints and
branches.

This caused GC references live across `try_call[_indirect]` instructions to not
be traced during garbage collection, leading to use-after-free within the GC
heap sandbox when the collector swept those untraced-but-still-live objects.

The fix adds stack map forwarding after branch lowering, mirroring the
existing logic for non-branch instructions.

Fixes bytecodealliance/wasmtime#11753.

* update disas test

show more ...

PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built

PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built verification of these facts after lowering to
machine instructions. We also added "memory types" describing layout
of memory and a "checked" flag on memory operations such that we could
verify that any checked memory operation accessed valid memory (as
defined by memory types attached to pointer values via
facts). Wasmtime's Cranelift backend then put appropriate memory types
and facts in its IR such that all accesses to memory (aspirationally)
could be checked, taking the whole mid-end and lowering backend of
Cranelift out of the trusted core that enforces SFI.

This basically worked, at the time, for static memories; but never for
dynamic memories, and then work on the feature lost
prioritization (aka I had to work on other things) and I wasn't able
to complete it and put it in fuzzing/enable it as a production option.

Unfortunately since then it has bit-rotted significantly -- as we add
new backend optimizations and instruction lowerings we haven't kept
the PCC framework up to date.

Inspired by the discussion in #12497 I think it's time to delete
it (hopefully just "for now"?) unless/until we can build it again. And
when we do that, we should probably get it to the point of validating
robust operation on all combinations of memory configurations before
merging. (That implies a big experiment branch rather than a bunch of
eager PRs in-tree, but so it goes.) I still believe it is possible to
build this (and I have ideas on how to do it!) but not right now.

show more ...

Only emit value label aliases for lowered instructions (#12779)

If we emit a value label alias for an instruction that isn't lowered,
then that signals that the value has been optimised out. However

Only emit value label aliases for lowered instructions (#12779)

If we emit a value label alias for an instruction that isn't lowered,
then that signals that the value has been optimised out. However, since
it is an alias we know that the value also exists in an earlier vreg, so
we should skip the alias and use that instead.

This situation occurs often for memory indexes on AArch64. We translate
memory stores into instructions such as:

v8 = iconst.i32 42
v9 = uextend.i64 v6
v10 = load.i64 notrap aligned readonly can_move checked v0+56
v11 = iadd v10, v9
v12 = iconst.i64 20
v13 = iadd v11, v12 ; v12 = 20
store little heap v8, v13 ; v8 = 42

Here, v6 is a memory index (which has a label) and v9 is an
extension of the memory index (which has a label alias, added by
cast_index_to_pointer_ty()). This is lowered to:

40c: 52800540 mov w0, #0x2a // #42
410: f9401c41 ldr x1, [x2, #56]
414: 91005021 add x1, x1, #0x14
418: b8384820 str w0, [x1, w24, uxtw]

The uextend has been folded into the str, so v9 has been optimised
out. But v6 is still present in w24, so the debuginfo should use that
instead.

This fixes the following tests for AArch64:

native_debug::lldb::dwarf_cold_block
native_debug::lldb::dwarf_fib_wasm
native_debug::lldb::dwarf_fib_wasm_dwarf5
native_debug::lldb::dwarf_fib_wasm_split4
native_debug::lldb::dwarf_fission
native_debug::lldb::dwarf_fraction_norm
native_debug::lldb::dwarf_imported_memory
native_debug::lldb::dwarf_shared_memory
native_debug::lldb::dwarf_simple
native_debug::lldb::dwarf_spilled_frame_base

show more ...

Partial support for no_std in cranelift_codegen (#12222)

* Move most things from std to core and alloc

* Port assembler_x64 to no_std

* before adding prelude to each file

* Most of the files now

Partial support for no_std in cranelift_codegen (#12222)

* Move most things from std to core and alloc

* Port assembler_x64 to no_std

* before adding prelude to each file

* Most of the files now work with no_std

* update isle to use alloc and core

* some instances shouldn't have been renamed, fixes cargo test

* add cranelift-assembler-x64 (no_std) to CI

* fix codegen_meta, missed one spot with std::slice

* automatically remove prelude with cargo fix

* update isle changes

* update assembler changes

* update assembler changes

* use latest codegen changes + fix FxHash problem

* add imports

* fix floating issues with libm

* remove unused import

* temporarily remove OnceLock

* add no_std arm support and add it into CI

* Move most things from std to core and alloc

* Port assembler_x64 to no_std

* before adding prelude to each file

* Most of the files now work with no_std

* update isle to use alloc and core

* some instances shouldn't have been renamed, fixes cargo test

* add cranelift-assembler-x64 (no_std) to CI

* automatically remove prelude with cargo fix

* update isle changes

* update assembler changes

* update assembler changes

* use latest codegen changes + fix FxHash problem

* add imports

* fix floating issues with libm

* remove unused import

* temporarily remove OnceLock

* add no_std arm support and add it into CI

* Move most things from std to core and alloc

* Port assembler_x64 to no_std

* before adding prelude to each file

* Most of the files now work with no_std

* update isle to use alloc and core

* add cranelift-assembler-x64 (no_std) to CI

* automatically remove prelude with cargo fix

* update isle changes

* update assembler changes

* use latest codegen changes + fix FxHash problem

* add imports

* fix floating issues with libm

* temporarily remove OnceLock

* add no_std arm support and add it into CI

* revert Cargo.toml formating

* remove prelude and fix cargo.toml

* cargo fmt

* remove empty lines

* bad renames

* macro_use only on no_std

* revert OnceLock change

* only use stable libm features

* update regalloc2

* update comment

* use continue instead

* Update vets

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable AB

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`.

As discussed in this week's Cranelift meeting, we've discovered a need
to generalize the `patchable_call` mechanism and corresponding
`patchable` ABI slightly. In particular, we will need patchable
`try_call` callsites as well in order to allow breakpoint handlers to
throw exceptions (desirable functionality eventually) and have this work
in the presence of inlining. Also, it's just a nice generalization to
say that patchability is an orthogonal dimension to the call ABI and the
other restrictions we initially imposed, and works as long as the basic
requirement (no return values) is met.

This also renames the `patchable` ABI to `preserve_all`, to make it
clear that its purpose is actually orthogonal, and it can be used
independently of patchable callsites. It also deletes the `cold` ABI,
which never actually did anything and is misleading in the presence of
an actual cold-ish (subzero temperature, actually) ABI like
`preserve_all`.

* Review feedback.

show more ...

Cranelift: add debug tag infrastructure. (#11768)

* Cranelift: add debug tag infrastructure.

This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCo

Cranelift: add debug tag infrastructure. (#11768)

* Cranelift: add debug tag infrastructure.

This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCode instructions and as metadata on
the produced compiled code. It also adds opaque descriptor blobs
carried with stackslots. Together, these two features allow decorating
IR with first-class debug instrumentation that is properly preserved
by the compiler, including across optimizations and
inlining. (Wasmtime's use of these features will come in followup
PRs.)

The key idea of a "debug tag" is to allow the Cranelift embedder to
express whatever information it needs to, in a format that is opaque
to Cranelift itself, except for the parts that need translation during
lowering. In particular, the `DebugTag::StackSlot` variant gets
translated to a physical offset into the stackframe in the compiled
metadata output. So, for example, the embedder can emit a tag
referring to a stackslot, and another describing an offset in that
stackslot.

The debug tags exist as a *sequence* on any given instruction; the
meaning of the sequence is known only to the embedder, *except* that
during inlining, the tags for the inlining call instruction are
prepended to the tags of inlined instructions. In this way, a
canonical use-case of tags as describing original source-language
frames can preserve the source-language view even when multiple
functions are inlined into one.

The descriptor on a stackslot may look a little odd at first, but its
purpose is to allow serializing some description of
stackslot-contained runtime user-program data, in a way that is firmly
attached to the stackslot. In particular, in the face of inlining,
this descriptor is copied into the inlining (parent) function from the
inlined function when the stackslot entity is copied; no other
metadata outside Cranelift needs to track the identity of stackslots
and know about that motion. This fits nicely with the ability of tags
to refer to stackslots; together, the embedder can annotate
instructions as having certain state in stackslots, and describe the
format of that state per stackslot.

This infrastructure is tested with some compile-tests now;
testing of the interpretation of the metadata output will come with
end-to-end debug instrumentation tests in a followup PR.

* Review feedback: add back sequence points and enforce tags only on sequence points or calls.

* Use Vecs for debug metadata in MachBuffer to avoid SmallVec size penalty in not-used case.

* Review feedback: switch from inlined stackslot descriptor blobs to u64 keys.

show more ...

Cranelift: add get_exception_handler_address. (#11629)

* Cranelift: add get_exception_handler_address.

This is designed to enable applications such as #11592 that use
alternative unwinding mechanis

Cranelift: add get_exception_handler_address. (#11629)

* Cranelift: add get_exception_handler_address.

This is designed to enable applications such as #11592 that use
alternative unwinding mechanisms that may not necessarily want to walk a
stack and look up exception tables. The idea is that whenever it would
be valid to resume to an exception handler that is active on the stack,
we can provide the same PC as a first-class runtime value that would be
found in the exception table for the given handler edge. A "custom"
resume step can then use this PC as a resume-point as long as it follows
the relevant exception ABI (i.e.: restore SP, FP, any other saved
registers that the exception ABI specifies, and provide appropriate
payload value(s)).

Handlers are associated with edges out of `try_call`s (or
`try_call_indirect`s); and edges specifically, not blocks, because there
could be multiple out-edges to one block. The instruction thus takes the
block that contains the try-call and an immediate that indexes its
exceptional edges.

This CLIF instruction required a bit of infrastructure to (i) allow
naming raw blocks, not just block calls, as instruction arguments, and
(ii) allow getting the MachLabel for any other lowered block during
lowering. But given that, the lowerings themselves are straightforward
uses of MachBuffer labels to fix-up PC-relative address-loading
instructions (e.g., `LEA` or `ADR` or `AUIPC`+`ADDI`).

* Review feedback.

* Review feedback: more tests.

show more ...

Clarify docs around exceptions in Cranelift (#11601)

* Clarify docs around exceptions in Cranelift

This is a result of today's Cranelift meeting with some of my questions
around the ABI bits here a

Clarify docs around exceptions in Cranelift (#11601)

* Clarify docs around exceptions in Cranelift

This is a result of today's Cranelift meeting with some of my questions
around the ABI bits here and there. Notably:

* Cranelift is audited and now documented to always consider the callee
calling convention in `try_call`, disregarding the caller calling convention.

* Wasmtime's exception throw now explicitly names the `tailcc` in the
name to indicate that it's only compatible with the tail calling convention.

* The verifier test for exceptions is expanded with a few more cases
here and there that I could think of.

* Backends now assert that the size of the calling-convention list of
payload types is the same as the size of the list of registers the
backend places results into.

* Fix a typo

show more ...

Add lowering rules for `{add,sub,or,and} mem, imm` on x64 (#11043)

* Add lowering rules for `{add,sub,or,and} mem, imm` on x64

* Fold `<op> reg, imm` lowering rules into existing `x64_op_mem` helpe

Add lowering rules for `{add,sub,or,and} mem, imm` on x64 (#11043)

* Add lowering rules for `{add,sub,or,and} mem, imm` on x64

* Fold `<op> reg, imm` lowering rules into existing `x64_op_mem` helpers

* Remove duplicate decl that got added elsewhere in a different commit on main

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Refactor call ABI implementation (#10722)

This refactors implementation of call ABI handling across architectures
with the goal of bringing s390x in line with other platforms.

The main idea is to
-

Refactor call ABI implementation (#10722)

This refactors implementation of call ABI handling across architectures
with the goal of bringing s390x in line with other platforms.

The main idea is to
- handle main call instruction selection and generation in ISLE
(like s390x but unlike other platforms today)
- handle argument setup mostly outside of ISLE
(like other platforms but unlike s390x today)
- handle return value processing as part of the call instructio
(like all platforms today)

All platforms now emit the main call instruction directly from ISLE,
which e.g. handles selection of the correct ISA instruction depending
on the call destination. This ISLE code calls out to helper routines
to handle argument and return value processing. These helpers are
mostly common code and provided by the Callee and/or Lower layers,
with some platform-specific additions via ISLE Context routines.

The old CallSite abstraction is no longer needed; most of the
differences between call and return_call handling disappear.
(There is still a common-code CallInfo vs. a platform-specifc
ReturnCallInfo. At this point, it should be relatively straight-
forward to make CallInfo platform-specific as well if desired,
but this is not done here.)

Some ISLE infrastructure for iterators / loops, which was only
ever used by the s390x argument processing code, has been removed.

s390x now closely matches all other platforms, with only a few
special cases (slightly different tail-call ABI requires some
differences in stack offset computations; we still need to
handle vector lane swaps for cross-ABI calls), which should
simplify future maintenance.

show more ...

[DI] Fix live range tracking off-by-one confusions (#10570)

* Dump blocks in the VL table

* Add a test

* Work around #10572 in tests

* [DI] Fix live range tracking off-by-one confusions

How thin

[DI] Fix live range tracking off-by-one confusions (#10570)

* Dump blocks in the VL table

* Add a test

* Work around #10572 in tests

* [DI] Fix live range tracking off-by-one confusions

How things used to work w.r.t. instruction indices (IIs):
1) In lowering:
- Reversed order: IIs represented "before IP"s.
- Block args were defined one instruction too late,
but this issue was masked due to how RA allocates,
at least in simple examples.
- Execution order: IIs represented "after IP"s.
2) In RA:
- IIs represented "before IP"s.
- Notice the mismatch.
3) In emit:
- RA directions w.r.t. the explicit ProgPoint
positions were not respected and always treated
as "after".

How things work after this change:
1) In lowering:
- Reversed order: IIs represent "after IP"s.
- Execution order: IIs represent "before IP"s.
2) In RA:
- No change; mismatch fixed.
3) In emit:
- ProgPoint positions now respected.

This fixes various "silent bad debug info" issues.

show more ...

Cranelift: simplify some side-effectful instructions in ISLE (#10524)

* Cranelift: simplify some side-effectful instructions in ISLE

This commit adds a new top-level ISLE entrypoint specifically fo

Cranelift: simplify some side-effectful instructions in ISLE (#10524)

* Cranelift: simplify some side-effectful instructions in ISLE

This commit adds a new top-level ISLE entrypoint specifically for instructions
in the side-effectful skeleton: `simplify_skeleton`. While these rewrites are
processed during the egraph pass, values from skeleton instructions still do not
get inserted into the egraph. Indeed, `simplify_skeleton` operates
on *instructions* rather than *values* because we do not represent side effects
as values; values do not have side effects in CLIF, instructions do. Therefore,
rather than doing a whole dynamic-programming style extraction of the best
candidate simplification like we do with the egraph, we take an eager and greedy
approach.

Furthermore, `simplify_skeleton` is limited only to skeleton instructions that
do not involve control-flow or terminators right now. This is because changing
the control-flow graph can change whether a use is dominated by a def or not,
and we do not currently have the machinery to track and fix up invalidated
uses. Addressing this is left for future commits.

* fix `MIN / -1` cprop and add negative tests for things simplify_skeleton cannot handle yet

show more ...

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect`

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect` instructions, and
lowerings on four of five ISAs (x86-64, aarch64, riscv64, pulley; s390x
has its own non-shared ABI code that will need separate work).

It extends CLIF to support these instructions as new kinds of branches,
and extends block-calls to accept `retN` and `exnN` block-call args that
carry the normal return values or exception payloads (respectively) into
the appropriate successor blocks.

It wires up the "normal return path" so that it continues to work.
It updates the ABI so that unwinding is possible without an initial
register state at throw: specifically, as per our RFC, all registers are
clobbered. It also includes metadata in the `MachBuffer` that describes
exception-catch destinations. However, no unwinder exists to interpret
these catch-destinations yet, so they are untested.

* Add try_call_indirect lowering as well.

show more ...

Cranelift: remove block params on critical-edge blocks. (#10485)

When a block has a terminator branch that targets two or more other
blocks at the CLIF level, and any of these blocks have two or mor

Cranelift: remove block params on critical-edge blocks. (#10485)

When a block has a terminator branch that targets two or more other
blocks at the CLIF level, and any of these blocks have two or more
precessors, the edge is a "critical edge" and we split it (insert a
new empty block) so that the register allocator has a place to put
moves that happen only on that edge. Otherwise, there is no location
that works: in the predecessor, code runs no matter which outgoing
edge we take; and in the successor, code runs no matter which incoming
edge we came from.

Currently, when we generate these critical-edge blocks, we insert
exactly one instruction: an unconditional branch. We wire up the
blockparam dataflow by (i) adding block parameters to the
critical-edge block with the same signature as the original target,
and (ii) adding all of these arguments to the unconditional branch. In
other words, we maintain the original block signature throughout.

This is fine and correct, but it has two downsides. The first is a
minor loss in compile-time efficiency (more SSA values and
block-params to process). The second, more interesting, is that it
hinders future work with certain kinds of branches that may define
values *on edges*.

In particular, this approach prevents exception-handling support: a
`try_call` instruction that acts as a terminator branch (with
normal-return and exceptional out-edges) defines normal-return values
as block-call arguments that are usable on the normal-return
edge. Some of these normal-return values may be defined by loads from
a return-value area. These loads need somewhere to go; they can't go
"after the terminator" (then it wouldn't be a terminator), so they go
in an edge block; as a result, the block-call for the normal-return
needs to use its arguments only in the unconditional branch out of the
edge block, not in the initial branch to the edge block.

This PR alters the critical-edge blockparam handling to have no
block-call args on the branch into the edge block, and use the
original values (not the newly defined edge-block blockparams) in the
block-call out of the edge block. This will allow these values to be
possibly defined in the edge block rather than in the predecessor (the
block with the original terminator).

This has no functional change today other than some perturbation of
regalloc decisions and a possibly slight compile-time speedup.

show more ...

Introduce log2_min_function_alignment flag (#10391)

* Remove function_alignment handling from cranelift-object and cranelift-jit

It is already handled by MachBuffer. The symbol_alignment could also

Introduce log2_min_function_alignment flag (#10391)

* Remove function_alignment handling from cranelift-object and cranelift-jit

It is already handled by MachBuffer. The symbol_alignment could also be
removed as no current backend has a symbol alignment bigger than the
function alignment, but keeping it around is a bit safer when new
backends are introduced.

* Introduce log2_min_function_alignment flag

This is required for cg_clif to implement -Zmin-function-alignment.

show more ...

Fix minor formatting issues (#9748)

* format: fix typo

* format: wrap line length

* format: re-wrap comment

* format: organize crate dependencies

Don't panic when instructions aren't in ISLE (#9631)

Instead return a codegen error which can be more easily caught,
displayed, and rendered. This is intended to help with enabling testing
of Pulley

Don't panic when instructions aren't in ISLE (#9631)

Instead return a codegen error which can be more easily caught,
displayed, and rendered. This is intended to help with enabling testing
of Pulley in the near future where Pulley doesn't implement most
instructions at this time and this should make the in-progress state of
the backend a bit easier to test.

show more ...

Optimize 128-bit multiplication some more in backends (#9215)

* Add some tests that will get optimized next

These tests currently reflect the output of Cranelift today and will
showcase the diff i

Optimize 128-bit multiplication some more in backends (#9215)

* Add some tests that will get optimized next

These tests currently reflect the output of Cranelift today and will
showcase the diff in the next commit of the optimizations implemented.

* Optimize 128-bit multiplication some more in backends

This commits adds support for more patterns of improved 128-bit
multiplication to the various backends of Cranelift. Notably:

* Using `isplit` to discard the lower bits of a multiplication maps
directly to instructions on aarch64 and riscv64.
* Multiplying sign-extended 64-bit halves maps directly to supported
instructions on s390x/riscv64/aarch64 (x86_64 already has these
implemented).

This relies on adding a new method to test whether a `Value` is dead and
unused during lowering. While generally not useful this is applicable
for multi-result instructions such as `isplit` where one result may end
up not being used. This also is required because the egraph
optimizations can't currently handle multi-result instructions like
`isplit` so this can't be optimized to `umulhi` or `smulhi` in CLIF.

show more ...

Don't force `Multiple` on multi-result instructions (#9137)

* Move `compute_use_states` to be able to test it

Make it a free-function so it doesn't depend on the `I` type variable of `Lower`

* Do

Don't force `Multiple` on multi-result instructions (#9137)

* Move `compute_use_states` to be able to test it

Make it a free-function so it doesn't depend on the `I` type variable of `Lower`

* Don't force `Multiple` on multi-result instructions

This commit is a result of [discussion on Zulip][Zulip] and is
attempting to fix an issue where some 128-bit instructions aren't fully
benefitting from load sinking on x64. At a high level 128-bit
addition isn't able to sink loads into instructions for halves of the
128-bit operation. At a lower level the reason for this is that
currently all operands of a multiple-result instruction are considered
multiply-used (as each result could be used) which prevents load
sinking.

Operations on 128-bit integers may be coupled with `isplit` afterwards
which is a multi-result instruction. This then means that the `Multiple`
state flows backwards to the 128-bit operation and all its operands,
including whatever is necessary to produce the individual components of
each 128-bit integer.

The fix in this commit is to introduce the concept of a "root"
instruction from the perspective of the calculation of `ValueUseState`.
In other words `ValueUseState` is no longer an accurate picture of the
function as a whole, but only the parts of the function rooted at a
"root" instruction. This is currently defined as multi-result
instructions meaning that `isplit` for example is a root instruction.
This is coupled with documentation/changes to
`get_value_as_source_or_const` to never allow looking through root
instructions (or considering them a `UniqueUse`).

This commit additionally updates some documentation in a few places and
refactors some usage of `get_value_as_source_or_const` to use other
helpers instead to reduce callers of `get_value_as_source_or_const` and
who to audit when modifying this function.

[Zulip]: https://bytecodealliance.zulipchat.com/#narrow/stream/217117-cranelift/topic/ValueUseState.3A.3AMultiple.20and.20multi-result.20instructions/near/462833578

* Review comments

show more ...

Enforce `clippy::clone_on_copy` for the workspace (#9025)

* Derive `Copy` for `Val`

* Fix `clippy::clone_on_copy` for the whole repo

* Enforce `clippy::clone_on_copy` for the workspace

* fix some

Enforce `clippy::clone_on_copy` for the workspace (#9025)

* Derive `Copy` for `Val`

* Fix `clippy::clone_on_copy` for the whole repo

* Enforce `clippy::clone_on_copy` for the workspace

* fix some more clippy::clone_on_copy that got missed

show more ...

Cranelift: Take user stack maps through lowering and emission (#8876)

* Cranelift: Take user stack maps through lowering and emission

Previously, user stack maps were inserted by the frontend and p

Cranelift: Take user stack maps through lowering and emission (#8876)

* Cranelift: Take user stack maps through lowering and emission

Previously, user stack maps were inserted by the frontend and preserved in the
mid-end. This commit takes them from the mid-end CLIF into the backend vcode,
and then from that vcode into the finalized mach buffer during emission.

During lowering, we compile the `UserStackMapEntry`s into packed
`UserStackMap`s. This is the appropriate moment in time to do that coalescing,
packing, and compiling because the stack map entries are immutable from this
point on.

Additionally, we include user stack maps in the `Debug` and disassembly
implementations for vcode, just after their associated safepoint
instructions. This allows us to see the stack maps we are generating when
debugging, as well as write filetests that check we are generating the expected
stack maps for the correct instructions.

Co-Authored-By: Trevor Elliott <[email protected]>

* uncomment debug assert that was commented out for debugging

* Address review feedback

* remove new method that was actually never needed

---------

Co-authored-by: Trevor Elliott <[email protected]>

show more ...

Add a fuzz target for exercising bounds checks with various memory configs (#8742)

cranelift: Always consider sret arguments used (#8664)

* cranelift: Always consider sret arguments used

In #8438 we stopped emitting register bindings for unused arguments,
based on the use-counts

cranelift: Always consider sret arguments used (#8664)

* cranelift: Always consider sret arguments used

In #8438 we stopped emitting register bindings for unused arguments,
based on the use-counts from `compute_use_states`. However, that doesn't
count the use of the struct-return argument that's automatically added
after lowering when the `rets` instruction is generated in the epilogue.
As a result, using a struct-return argument caused register allocation
to panic due to the VReg not being defined anywhere.

This commit adds a use to the struct-return argument so that it's always
available in the epilogue.

Fixes #8659

* Review comments

show more ...