| 763622c3 | 01-Apr-2026 |
Nick Fitzgerald <[email protected]> |
Preserve `try_call[_indirect]` stack maps during lowering (#12934)
* Preserve `try_call[_indirect]` stack maps during lowering
Branch instructions are skipped in the main lowering loop, which means
Preserve `try_call[_indirect]` stack maps during lowering (#12934)
* Preserve `try_call[_indirect]` stack maps during lowering
Branch instructions are skipped in the main lowering loop, which means the stack
map forwarding code is never reached for them. The branch lowering path didn't
forward stack maps either. This was fine because branch instructions couldn't
previously ever be safepoints. However, with the introduction of `try_call` and
`try_call_indirect`, we now have instructions that are both safepoints and
branches.
This caused GC references live across `try_call[_indirect]` instructions to not
be traced during garbage collection, leading to use-after-free within the GC
heap sandbox when the collector swept those untraced-but-still-live objects.
The fix adds stack map forwarding after branch lowering, mirroring the
existing logic for non-branch instructions.
Fixes bytecodealliance/wasmtime#11753.
* update disas test
show more ...
|
| baa6b27b | 26-Mar-2026 |
Chris Fallin <[email protected]> |
Cranelift: rework MachBuffer to handle very short-deadline jumps. (#12842)
* Cranelift: rework MachBuffer to handle very short-deadline jumps.
In #12811 it was reported that riscv64 compressed jump
Cranelift: rework MachBuffer to handle very short-deadline jumps. (#12842)
* Cranelift: rework MachBuffer to handle very short-deadline jumps.
In #12811 it was reported that riscv64 compressed jumps (`c.j`
instructions), with a +/- 2048-byte range, could cause panics when
combined with queued-up/deferred constants in a constant pool during
binary emission.
Our `MachBuffer` handles single-pass machine code emission, resolution
of labels, and upgrading of label ranges via "veneers" (jumps that a
shorter jump can reach that themselves have a longer range). We track a
pending "deadline" of all unresolved branches, and when the deadline is
too close (including the max size of all veneers yet to be emitted), we
emit an "island" of all veneers to resolve the deadline.
After its initial design, we added support for deferred traps and
constants to the `MachBuffer`. These worked by emitting their contents
*before* the "island" of veneers, which turns out to be slightly nicer
for code layout in some cases.
Unfortunately the full implications of those additions weren't realized
against the invariants of the deadline-resolution algorithm. In
particular, when a new branch is added with a very short range (e.g.,
`c.j`), it is possible that there are *already* too many queued-up
traps/constants for the range of that just-emitted branch to reach even
the first possible veneer site if we start an island right away.
Thus it is strictly necessary to emit the veneers before
constants/traps. Unfortunately this requires some alterations to other
aspects of label resolution as well: in particular, we can't resolve
fixups for label references to constants before we emit those constants,
and likewise for traps. Note that we do a fixpoint loop over emitting
island(s) at the end of emission, so all constants/traps *will* be
emitted and label references to them *will* be resolved eventually; just
in the opposite order, now.
No compile test because the particular reduced testcase in #12811 only
worked in the `release-36.0.0` branch, and not on `main`, and it was too
hard to tweak the test to hit the right case on `main` as well. In lieu
of that, I've added a unit test directly to the `MachBuffer`
implementation to exercise this case.
Fixes #12811.
* fix filetest with errant comments confusing precise-output check
show more ...
|
| 326599f3 | 16-Mar-2026 |
Philip Craig <[email protected]> |
Only emit value label aliases for lowered instructions (#12779)
If we emit a value label alias for an instruction that isn't lowered,
then that signals that the value has been optimised out. However
Only emit value label aliases for lowered instructions (#12779)
If we emit a value label alias for an instruction that isn't lowered,
then that signals that the value has been optimised out. However, since
it is an alias we know that the value also exists in an earlier vreg, so
we should skip the alias and use that instead.
This situation occurs often for memory indexes on AArch64. We translate
memory stores into instructions such as:
v8 = iconst.i32 42
v9 = uextend.i64 v6
v10 = load.i64 notrap aligned readonly can_move checked v0+56
v11 = iadd v10, v9
v12 = iconst.i64 20
v13 = iadd v11, v12 ; v12 = 20
store little heap v8, v13 ; v8 = 42
Here, v6 is a memory index (which has a label) and v9 is an
extension of the memory index (which has a label alias, added by
cast_index_to_pointer_ty()). This is lowered to:
40c: 52800540 mov w0, #0x2a // #42
410: f9401c41 ldr x1, [x2, #56]
414: 91005021 add x1, x1, #0x14
418: b8384820 str w0, [x1, w24, uxtw]
The uextend has been folded into the str, so v9 has been optimised
out. But v6 is still present in w24, so the debuginfo should use that
instead.
This fixes the following tests for AArch64:
native_debug::lldb::dwarf_cold_block
native_debug::lldb::dwarf_fib_wasm
native_debug::lldb::dwarf_fib_wasm_dwarf5
native_debug::lldb::dwarf_fib_wasm_split4
native_debug::lldb::dwarf_fission
native_debug::lldb::dwarf_fraction_norm
native_debug::lldb::dwarf_imported_memory
native_debug::lldb::dwarf_shared_memory
native_debug::lldb::dwarf_simple
native_debug::lldb::dwarf_spilled_frame_base
show more ...
|
| b5d2ff5d | 18-Feb-2026 |
Chris Fallin <[email protected]> |
Cranelift: update regalloc2 to 0.15.0 to permit more VRegs. (#12611)
* Cranelift: update regalloc2 to 0.15.0 to permit more VRegs.
This pulls in bytecodealliance/regalloc2#257 to permit more VRegs
Cranelift: update regalloc2 to 0.15.0 to permit more VRegs. (#12611)
* Cranelift: update regalloc2 to 0.15.0 to permit more VRegs.
This pulls in bytecodealliance/regalloc2#257 to permit more VRegs to
be used in a single function body, addressing #12229 and our followup
discussions about supporting function body sizes up to the Wasm
implementation limit standard.
In addition to the RA2 upgrade, this also includes a bit more explicit
limit-checking on the Cranelift side: note that we don't directly use
`regalloc2::VReg` but instead we further bitpack it into `Reg`, which
is logically a sum type of `VReg`, `PReg` and `SpillSlot` (the last
one needed to represent stack allocation locations on defs, e.g. on
callsites with many returns). `PReg`s are packed into the beginning of
the `VReg` index space but `SpillSlot`s are distinguished by stealing
the upper bit of a `u32`. This was previously not a problem given the
smaller `VReg` index space but now we need to check explicitly; hence
`Reg::from_virtual_reg_checked` and its use in the lowering vreg
allocator. Because the `VReg` index packs the class into the bottom
two bits, and index into the upper 30, but we steal one bit at the
top, the true limit for VReg count is thus actually 2^29, or
512M.
Fixes #12229.
* Drop `code_too_large` test.
show more ...
|
| c00e9ea2 | 02-Dec-2025 |
Chris Fallin <[email protected]> |
Cranelift: add patchable call instructions. (#12101)
* Cranelift: add patchable call instructions.
The new `patchable_call` CLIF instruction pairs with the `patchable`
ABI, and emits a callsite wit
Cranelift: add patchable call instructions. (#12101)
* Cranelift: add patchable call instructions.
The new `patchable_call` CLIF instruction pairs with the `patchable`
ABI, and emits a callsite with one new key property: the MachBuffer
carries metadata that describes exactly which byte range to "NOP out"
(overwrite with NOP instructions) to disable that callsite. Doing so is
semantically valid and explicitly supported.
This enables patching of code at runtime to dynamically turn on and off
features such as instrumentation or debugging hooks. We plan to use this
to implement breakpoints in Wasmtime's guest debugging support.
As part of this change, I added a notion of "unit of NOP bytes" to the
MachBuffer so that the consumer (e.g., Wasmtime's Cranelift-based code
compilation pipeline and metadata-producing logic) can handle patchable
callsites without any other special knowledge of the ISA.
For the "real metal" ISAs there are perfectly well-defined NOPs to use,
but for Pulley, where all opcodes are assigned at compile time by macro
magic, I explicitly defined NOP as opcode byte 0 by moving `Nop`'s
definition to the top of the list and adding a unit test asserting its
encoding.
A design note: in principle it would be possible, as an alternative, to
treat "patchability" as an orthogonal dimension of all callsites, and
emit the metadata describing the instruction-offset range for any
callsite with the flag set. The only truly necessary semantic
restriction is that there are no return values (because if we turn the
callsite off, nothing writes to them); we could support patchability for
other ABIs and for the other kinds of call instructions. The `patchable`
ABI would then be better described as something like the "no clobbers
ABI". I opted not to generalize in this way because it creates some
less-tested corners and the generalized form, at least at the MachInst
level, is not really much simpler in the end.
A testing note: I opted not to implement actual code patching in the
`cranelift-tools` filetest runner and test patching callsites in/out via
some actuation (e.g. a magic hostcall, like we do for throws) because
(i) that's a lot of new plumbing and (ii) we are going to test this very
shortly in Wasmtime anyway and (iii) the correctness (or not) of the
location-and-length metadata is easy enough to verify in the
disassemblies in the compile-tests.
* Review feedback: remove dependence on (and test for) NOP being the literal byte 0.
show more ...
|
| a3d6e407 | 06-Oct-2025 |
Chris Fallin <[email protected]> |
Cranelift: add debug tag infrastructure. (#11768)
* Cranelift: add debug tag infrastructure.
This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCo
Cranelift: add debug tag infrastructure. (#11768)
* Cranelift: add debug tag infrastructure.
This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCode instructions and as metadata on
the produced compiled code. It also adds opaque descriptor blobs
carried with stackslots. Together, these two features allow decorating
IR with first-class debug instrumentation that is properly preserved
by the compiler, including across optimizations and
inlining. (Wasmtime's use of these features will come in followup
PRs.)
The key idea of a "debug tag" is to allow the Cranelift embedder to
express whatever information it needs to, in a format that is opaque
to Cranelift itself, except for the parts that need translation during
lowering. In particular, the `DebugTag::StackSlot` variant gets
translated to a physical offset into the stackframe in the compiled
metadata output. So, for example, the embedder can emit a tag
referring to a stackslot, and another describing an offset in that
stackslot.
The debug tags exist as a *sequence* on any given instruction; the
meaning of the sequence is known only to the embedder, *except* that
during inlining, the tags for the inlining call instruction are
prepended to the tags of inlined instructions. In this way, a
canonical use-case of tags as describing original source-language
frames can preserve the source-language view even when multiple
functions are inlined into one.
The descriptor on a stackslot may look a little odd at first, but its
purpose is to allow serializing some description of
stackslot-contained runtime user-program data, in a way that is firmly
attached to the stackslot. In particular, in the face of inlining,
this descriptor is copied into the inlining (parent) function from the
inlined function when the stackslot entity is copied; no other
metadata outside Cranelift needs to track the identity of stackslots
and know about that motion. This fits nicely with the ability of tags
to refer to stackslots; together, the embedder can annotate
instructions as having certain state in stackslots, and describe the
format of that state per stackslot.
This infrastructure is tested with some compile-tests now;
testing of the interpretation of the metadata output will come with
end-to-end debug instrumentation tests in a followup PR.
* Review feedback: add back sequence points and enforce tags only on sequence points or calls.
* Use Vecs for debug metadata in MachBuffer to avoid SmallVec size penalty in not-used case.
* Review feedback: switch from inlined stackslot descriptor blobs to u64 keys.
show more ...
|
| 8be22ddf | 04-Sep-2025 |
Alex Crichton <[email protected]> |
pulley: Fix big-endian results in a spillslot (#11597)
This commit fixes an issue for Pulley on big-endian targets where if a
call's result was mapped to a spillslot it was loaded with a too-wide
lo
pulley: Fix big-endian results in a spillslot (#11597)
This commit fixes an issue for Pulley on big-endian targets where if a
call's result was mapped to a spillslot it was loaded with a too-wide
load when moving from the return location to the storage location. This
only affects the shared `CallInfo::emit_retval_loads` implementation
which is unused on s390x. All other targets, with the exception of
big-endian Pulley targets, using this are little-endian where the width
won't matter since slots are always integer-register-in-size.
The fix here is to perform a load with the exact type of the return
value as opposed to a full machine word width. This fixes the big-endian
behavior for Pulley and fixes the attached test cases which reproduce
the issue. I've also expanded some of the many-results tests to also
have many parameters to try to exercise more parts of the ABI pipeline
too.
show more ...
|