Changes in buffer.rs

bac0e78f - aarch64: Disable csdb emission by default (#12932)

Wed, 01 Apr 2026 22:16:55 +0000

aarch64: Disable csdb emission by default (#12932)* aarch64: Disable csdb emission by defaultThis has a massive performance penalty on macOS, for example, and peercompilers are not emitting this as part of on-by-default mitigations.This commit preserves the option to emit it with an aarch64-specific`use_csdb` flag, but the default is now `false` meaning that this is notemitted by default.Closes #12789* Fix tests* Fix tests & review comments* Use ISLE rule introduced List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

baa6b27b - Cranelift: rework MachBuffer to handle very short-deadline jumps. (#12842)

Thu, 26 Mar 2026 19:35:15 +0000

Cranelift: rework MachBuffer to handle very short-deadline jumps. (#12842)* Cranelift: rework MachBuffer to handle very short-deadline jumps.In #12811 it was reported that riscv64 compressed jumps (`c.j`instructions), with a +/- 2048-byte range, could cause panics whencombined with queued-up/deferred constants in a constant pool duringbinary emission.Our `MachBuffer` handles single-pass machine code emission, resolutionof labels, and upgrading of label ranges via "veneers" (jumps that ashorter jump can reach that themselves have a longer range). We track apending "deadline" of all unresolved branches, and when the deadline istoo close (including the max size of all veneers yet to be emitted), weemit an "island" of all veneers to resolve the deadline.After its initial design, we added support for deferred traps andconstants to the `MachBuffer`. These worked by emitting their contents*before* the "island" of veneers, which turns out to be slightly nicerfor code layout in some cases.Unfortunately the full implications of those additions weren't realizedagainst the invariants of the deadline-resolution algorithm. Inparticular, when a new branch is added with a very short range (e.g.,`c.j`), it is possible that there are *already* too many queued-uptraps/constants for the range of that just-emitted branch to reach eventhe first possible veneer site if we start an island right away.Thus it is strictly necessary to emit the veneers beforeconstants/traps. Unfortunately this requires some alterations to otheraspects of label resolution as well: in particular, we can't resolvefixups for label references to constants before we emit those constants,and likewise for traps. Note that we do a fixpoint loop over emittingisland(s) at the end of emission, so all constants/traps *will* beemitted and label references to them *will* be resolved eventually; justin the opposite order, now.No compile test because the particular reduced testcase in #12811 onlyworked in the `release-36.0.0` branch, and not on `main`, and it was toohard to tweak the test to hit the right case on `main` as well. In lieuof that, I've added a unit test directly to the `MachBuffer`implementation to exercise this case.Fixes #12811.* fix filetest with errant comments confusing precise-output check List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

ab78bd82 - fix: correct various typos (#12807)

Sun, 22 Mar 2026 18:10:20 +0000

fix: correct various typos (#12807)Signed-off-by: Ho Kim List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

0889323a - cranelift-codegen: rename most uses of std to core and alloc (#12237)

Sat, 03 Jan 2026 00:54:48 +0000

cranelift-codegen: rename most uses of std to core and alloc (#12237)* rename most std uses to core and alloc* cargo fmt List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

17fbd3c6 - Debug: implement breakpoints and single-stepping. (#12133)

Fri, 12 Dec 2025 20:02:54 +0000

Debug: implement breakpoints and single-stepping. (#12133)* Debug: implement breakpoints and single-stepping.This is a PR that puts together a bunch of earlier pieces (patchablecalls in #12061 and #12101, private copies of code in #12051, and allthe prior debug event and instrumentation infrastructure) to implementbreakpoints in the guest debugger.These are implemented in the way we have planned in #11964: eachsequence point (location prior to a Wasm opcode) is now a patchable callinstruction, patched out (replaced with NOPs) by default. When patchedin, the breakpoint callsite calls a trampoline with the `patchable` ABIwhich then invokes the `breakpoint` hostcall. That hostcall emits thedebug event and nothing else.A few of the interesting bits in this PR include:- Implementations of "unpublish" (switch permissions back to read/write from read/execute) for mmap'd code memory on all our platforms.- Infrastructure in the frame-tables (debug info) metadata producer and parser to record "breakpoint patches".- A tweak to the NOP metadata packaged with the `MachBuffer` to allow multiple NOP sizes. This lets us use one 5-byte NOP on x86-64, for example (did you know x86-64 had these?!) rather than five 1-byte NOPs.This PR also implements single-stepping with a global-per-`Store` flag,because at this point why not; it's a small additional bit of logic todo *all* patches in all modules registered in the `Store` when that flagis enabled.A few realizations for future work:- The need for an introspection API available to a debugger to see the modules within a component is starting to become clear; either that, or the "module and PC" location identifier for a breakpoint switches to a "module or component" sum type. Right now, the tests for this feature use only core modules. Extending to components should not actually be hard at all, we just need to build the API for it.- The interaction between inlining and `patchable_call` is interesting: what happens if we inline a `patchable_call` at a `try_call` callsite? Right now, we do *not* update the `patchable_call` to a `try_call`, because there is no `patchable_try_call`; this is fine in the Wasmtime embedding in practice because we never (today!) throw exceptions from a breakpoint handler. This does suggest to me that maybe we should make patchability a property of any callsite, and allow try-calls to be patchable too (with the same restriction about no return values as the only restriction); but happy to discuss that one further.* Add missing debug.wat disas test.* Review feedback.* Fix comment on `CodeMemory::text_mut`.* Review feedback.* Review feedback: abort process on failure to re-apply executable permissions.* Implement icache flush for aarch64.This appears to be necessary as we otherwise see a failure in CI onmacOS/aarch64 that is consistent with patched-in breakpoint calls stillbeing incorrectly cached after we remove them and republish the code.There is a longstanding issue in #3310 tracking proper icache coherencehandling on aarch64. We implemented this for Linux with the `membarrier`syscall but never did so for macOS. Maybe this is the first point atwhich it matters, because code was always loaded at new addresses (hencedid not have coherence issues because nothing would have been cached)previously.prtest:full* Review feedback: use `next_multiple_of`. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

c00e9ea2 - Cranelift: add patchable call instructions. (#12101)

Tue, 02 Dec 2025 00:59:15 +0000

Cranelift: add patchable call instructions. (#12101)* Cranelift: add patchable call instructions.The new `patchable_call` CLIF instruction pairs with the `patchable`ABI, and emits a callsite with one new key property: the MachBuffercarries metadata that describes exactly which byte range to "NOP out"(overwrite with NOP instructions) to disable that callsite. Doing so issemantically valid and explicitly supported.This enables patching of code at runtime to dynamically turn on and offfeatures such as instrumentation or debugging hooks. We plan to use thisto implement breakpoints in Wasmtime's guest debugging support.As part of this change, I added a notion of "unit of NOP bytes" to theMachBuffer so that the consumer (e.g., Wasmtime's Cranelift-based codecompilation pipeline and metadata-producing logic) can handle patchablecallsites without any other special knowledge of the ISA.For the "real metal" ISAs there are perfectly well-defined NOPs to use,but for Pulley, where all opcodes are assigned at compile time by macromagic, I explicitly defined NOP as opcode byte 0 by moving `Nop`'sdefinition to the top of the list and adding a unit test asserting itsencoding.A design note: in principle it would be possible, as an alternative, totreat "patchability" as an orthogonal dimension of all callsites, andemit the metadata describing the instruction-offset range for anycallsite with the flag set. The only truly necessary semanticrestriction is that there are no return values (because if we turn thecallsite off, nothing writes to them); we could support patchability forother ABIs and for the other kinds of call instructions. The `patchable`ABI would then be better described as something like the "no clobbersABI". I opted not to generalize in this way because it creates someless-tested corners and the generalized form, at least at the MachInstlevel, is not really much simpler in the end.A testing note: I opted not to implement actual code patching in the`cranelift-tools` filetest runner and test patching callsites in/out viasome actuation (e.g. a magic hostcall, like we do for throws) because(i) that's a lot of new plumbing and (ii) we are going to test this veryshortly in Wasmtime anyway and (iii) the correctness (or not) of thelocation-and-length metadata is easy enough to verify in thedisassemblies in the compile-tests.* Review feedback: remove dependence on (and test for) NOP being the literal byte 0. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

d55a5c8b - docs: minor improvement for docs (#11952)

Wed, 29 Oct 2025 07:08:49 +0000

docs: minor improvement for docs (#11952)Signed-off-by: geogrego List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

a3d6e407 - Cranelift: add debug tag infrastructure. (#11768)

Mon, 06 Oct 2025 19:38:03 +0000

Cranelift: add debug tag infrastructure. (#11768)* Cranelift: add debug tag infrastructure.This PR adds *debug tags*, a kind of metadata that can attach to CLIFinstructions and be lowered to VCode instructions and as metadata onthe produced compiled code. It also adds opaque descriptor blobscarried with stackslots. Together, these two features allow decoratingIR with first-class debug instrumentation that is properly preservedby the compiler, including across optimizations andinlining. (Wasmtime's use of these features will come in followupPRs.)The key idea of a "debug tag" is to allow the Cranelift embedder toexpress whatever information it needs to, in a format that is opaqueto Cranelift itself, except for the parts that need translation duringlowering. In particular, the `DebugTag::StackSlot` variant getstranslated to a physical offset into the stackframe in the compiledmetadata output. So, for example, the embedder can emit a tagreferring to a stackslot, and another describing an offset in thatstackslot.The debug tags exist as a *sequence* on any given instruction; themeaning of the sequence is known only to the embedder, *except* thatduring inlining, the tags for the inlining call instruction areprepended to the tags of inlined instructions. In this way, acanonical use-case of tags as describing original source-languageframes can preserve the source-language view even when multiplefunctions are inlined into one.The descriptor on a stackslot may look a little odd at first, but itspurpose is to allow serializing some description ofstackslot-contained runtime user-program data, in a way that is firmlyattached to the stackslot. In particular, in the face of inlining,this descriptor is copied into the inlining (parent) function from theinlined function when the stackslot entity is copied; no othermetadata outside Cranelift needs to track the identity of stackslotsand know about that motion. This fits nicely with the ability of tagsto refer to stackslots; together, the embedder can annotateinstructions as having certain state in stackslots, and describe theformat of that state per stackslot.This infrastructure is tested with some compile-tests now;testing of the interpretation of the metadata output will come withend-to-end debug instrumentation tests in a followup PR.* Review feedback: add back sequence points and enforce tags only on sequence points or calls.* Use Vecs for debug metadata in MachBuffer to avoid SmallVec size penalty in not-used case.* Review feedback: switch from inlined stackslot descriptor blobs to u64 keys. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

fa1d6867 - Wasmtime/Cranelift: carry "FP to SP offset" in exception data, and use it in stackwalk. (#11500)

Thu, 21 Aug 2025 22:40:47 +0000

Wasmtime/Cranelift: carry "FP to SP offset" in exception data, and use it in stackwalk. (#11500)* Wasmtime/Cranelift: carry "FP to SP offset" in exception data, and use it in stackwalk.Currently Wasmtime unwinds stack frames to look for exception handlersby walking frames one-by-one, following the FP chain as usual, andassuming that *these frames are contiguous*: that is, that the SP inany given frame (bottom of that frame) is immediately above the FP ofthe next lower frame, plus the FP/return address pair (e.g. 16 bytes).This allows us to get the SP for any given frame in addition to FP. Weneed SP for two reasons:- To look up dynamic context, to match Wasm tag instances for handlers against the thrown tag;- To actually set SP when we resume, if we do resume to a handler in this frame.This logic *almost but not quite* worked: I had forgotten that in ourtail-call ABI, we need to clean up incoming stack args in thecallee (because only the final callee in a parade of tail-callingfunctions that reuse the same stack frame location knows how many argsit has, not the original caller). This implies that there is an"incoming args area" *above* the FP/return address pair. Thus, framesare not necessarily contiguous by the above definition.In #11489 we see a case where a function of signature `(func)`tail-calls one of `(func (param i32 i32 i32 i32 i32))`, which onx86-64 (with four arg registers left for Wasm) is sufficient to createincoming stack args, which then trips up the unwinder, reading a bogusvmctx and segfaulting.The most reasonable solution seems to be to embed the SP-to-FP offsetin the exception metadata itself, so from only the FP (which istotally robust -- we rely on the FP chain for multiple kinds ofstack-walking) we can get the SP, allowing us to read dynamic contextand to reset SP during resume.This PR does just that. Technically, in our ABI, the SP-to-FP offsetis constant for an entire function, but it was simpler in theexception metadata to encode this per callsite instead (there is noother notion of "per-function" data, only "per-callsite", so it wouldbe a separate binary search).Fixes #11489.prtest:full* Review feedback. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

4590076f - Cranelift: support dynamic contexts in exception-handler lists. (#11321)

Sat, 26 Jul 2025 01:35:57 +0000

Cranelift: support dynamic contexts in exception-handler lists. (#11321)In #11285, we realized that Wasm semantics require us to match ondynamic instances of exception tags, rather than static tag types. Thisfundamentally requires the unwinder to be able to resolve the currentWasm instance for each Wasm frame on the stack that has any handlers,and our frame format does not provide this today.We discussed many options, some of which solve the more general problem(Wasm vmctx for any frame), but ultimately landed on a notion of"dynamic context for evaluating tags", specific to Cranelift'sexception-catch metadata; and storing that context and carrying itthrough to a place that is named in the unwind metadata. The reasoningis fairly straightforward: we cannot afford a more general approach thatstores vmctx in every frame (I measured this at 20% overhead for arecursive-Fibonacci benchmark that is call-intensive); and inliningmeans that we may have *multiple* contexts at any given program point,each associated with a different slice of the handler tags; so we need amechanism that, *just for a try-call*, intersperses contexts with tags(or puts a context on each tag) and stores these somewhere that theexception-unwind ABI doesn't clobber (e.g., on the stack).This PR implements "option 4" from that issue, namely, *dynamicexception contexts*. The idea is that this is the dual to exceptionpayload: while payload lets the unwinder communicate state *to* thecatching code, context lets the unwinder take state *from* the catchingcode that lets it decide whether the tag is a match. Because ofinlining, we need to either associate (optional) context with every tag,or intersperse context-updates with handler tags. I've opted for thelatter for efficiency at the CLIF level (in most cases there will bemultiple tags per context), though they are isomorphic.The new tag-matching semantics are: when walking up the stack, uponreaching a `try_call`, evaluate catch-clauses in listed order. A`context` clause sets the current context. A `tagN: block(...)` clauseattempts to match the throwing exception against `tagN`, *evaluated inthe current context*, and branches to the named block if it matches. A`default: block(...)` always branches to the named block.Note that this lets us assume less about tags than before, and thisparticularly manifests in the changes to the inliner. Whereas before,`tagN` is `tagN` and an inner handler for that tag shadows an outerhandler (that is, tags always alias if identical indices); and whereasbefore, `tagN` is not `tagM` and so we can order the tags arbitrarily(that is, tags never alias if non-identical indices); now any two statictag indices may or may not alias depending on the dynamic context ofeach. Or, even in the same context, two may alias, because we leave thematch-predicate as an unspecified (user-chosen) algorithm duringunwinding. (This mirrors the reality that, for example, a Wasm instancemay import two tags, and dynamically these tags may be equal ordifferent at runtime, even instantiation-to-instantiation.) Cranelift'sonly job is to faithfully carry the list of contexts and tags through tothe compiled-code metadata; and to ensure that they remain in the orderthey were specified in the CLIF.This PR introduces the Cranelift-level feature, and it will be used ina subsequent PR that introduces Wasm exception handling. Because ofthat, I've opted not to update the clif-utils runtest "runtime" to readout contexts and do something with them -- we will have plenty of testcoverage via a bunch of Wasm tests for corner cases such as the above.This PR does include filetests that show that contexts are carriedthrough to spillslots and those appear in the metadata.Fixes #11285. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

0854775b - Couple of optimizations to the Cranelift incremental cache (#11186)

Tue, 08 Jul 2025 16:11:24 +0000

Couple of optimizations to the Cranelift incremental cache (#11186)* Fix a couple of comments* Remove flags.predicate_view()It is a remenant of the old backend framework.* Avoid string conversions for hashing the TargetIsa* Remove func_body_lenIt is identical to buffer.data.len()* Introduce IsaFlagsHashKey List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

703871a2 - Enable the `useless_conversion` Clippy lint (#10838)

Tue, 27 May 2025 16:49:22 +0000

Enable the `useless_conversion` Clippy lint (#10838)* Enable the `useless_conversion` Clippy lintWe've got lots of types in Wasmtime and convert between them quite alot, but often over time conversions become unnecessary throughrefactorings or similar. This will hopefully enable us to clean up someconversions as they come up to try to have as few as possible ideally.* Review comments List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

90ac295e - Update Wasmtime to the 2024 Rust Edition (#10806)

Mon, 19 May 2025 16:40:55 +0000

Update Wasmtime to the 2024 Rust Edition (#10806)* Update Wasmtime to the 2024 Rust EditionNow that our MSRV supports the 2024 edition it's possible to make thisswitch. This commit moves Wasmtime to the 2024 Edition to keepup-to-date with Rust idioms and access many of the edition featuresexclusive to the 2024 edition.prtest:full* Reformat with the 2024 edition List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

c9db233a - Cranelift: move exception-handler metadata into callsites. (#10609)

Fri, 18 Apr 2025 17:06:33 +0000

Cranelift: move exception-handler metadata into callsites. (#10609)* Rework MachBuffer interface for exception_handlers* Rework MachBuffer to store exception handler records in flattened vector.This commit updates the call-site metadata to refer to a range in aflattened vector containing tuples of handler tags and labels (beforefinalization) or code offsets (after finalization). It also provides aniterator accessor `.call_sites()` on the finalized buffer that yieldsthis information in a safe way.---------Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com> List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

94ec88ea - Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

Tue, 08 Apr 2025 00:02:16 +0000

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)* Cranelift: initial try_call / try_call_indirect (exception) support.This PR adds `try_call` and `try_call_indirect` instructions, andlowerings on four of five ISAs (x86-64, aarch64, riscv64, pulley; s390xhas its own non-shared ABI code that will need separate work).It extends CLIF to support these instructions as new kinds of branches,and extends block-calls to accept `retN` and `exnN` block-call args thatcarry the normal return values or exception payloads (respectively) intothe appropriate successor blocks.It wires up the "normal return path" so that it continues to work.It updates the ABI so that unwinding is possible without an initialregister state at throw: specifically, as per our RFC, all registers areclobbered. It also includes metadata in the `MachBuffer` that describesexception-catch destinations. However, no unwinder exists to interpretthese catch-destinations yet, so they are untested.* Add try_call_indirect lowering as well. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

2af0a1f7 - Introduce log2_min_function_alignment flag (#10391)

Thu, 13 Mar 2025 22:55:43 +0000

Introduce log2_min_function_alignment flag (#10391)* Remove function_alignment handling from cranelift-object and cranelift-jitIt is already handled by MachBuffer. The symbol_alignment could also beremoved as no current backend has a symbol alignment bigger than thefunction alignment, but keeping it around is a bit safer when newbackends are introduced.* Introduce log2_min_function_alignment flagThis is required for cg_clif to implement -Zmin-function-alignment. List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

b09b892c - refactor: unify how bits are accessed in `cranelift-entity` (#10126)

Mon, 27 Jan 2025 22:16:49 +0000

refactor: unify how bits are accessed in `cranelift-entity` (#10126)* refactor: unify how bits are accessed in `cranelift-entity`While using `MachLabel`, a `cranelift-entity`-created type, I noticedthat there were three ways to access the contained bits: `.get()`,`.as_u32()`, and `.as_bits()`. All performed essentially the samefunction and it was unclear which to use.This change removes `MachLabel::get()`, replacing it with `as_u32()`.It also replaces all uses of `from_bits()` and `as_bits()` with`from_u32()` and `as_u32()`. Why? I would have preferred the "bits"naming since it seems more clear ("just unwrap this thing") and it couldavoid a large rename if the type were changed in the future, I realizedthat there are vastly more uses of the "u32" naming that alreadyexist--it's just easier.While this refactoring _should_ result in no functional change, you maynotice a couple of failing tests related to a pre-existing check on`from_u32` that did not exist on `from_bits`. For some reason,`from_u32` asserted that we would never pick `u32::MAX` for an entityvalue; unfortunately, some parsing code, `decode_narrow_field`, doesjust this. Why did we have such an assertion in the first place? Is itstill needed? Should `decode_narrow_field` do something else?* Re-add `from_bits`, `as_bits` and uses* doc: tweak doc comment List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

f6f447b0 - pulley: Add macro `CallN` instructions (#9874)

Fri, 20 Dec 2024 16:17:49 +0000

pulley: Add macro `CallN` instructions (#9874)* pulley: Add macro `CallN` instructionsThis commit adds new macro instructions to assist with speeding up callsbetween functions. Pulley's previous `Call` instruction was similar tonative call instructions where arguments/results are implicitly in theright location according to the ABI, but movement between registers ismore expensive with Pulley than with native architectures. The `CallN`instructions here enable listing a few arguments (only integerregisters) in the opcode itself. This removes the need for individual `xmov`instructions into individual registers and instead it can all be donewithin the opcode handlers.This additionally enables passing the same argument twice to a functionto reside only in one register. Finally parallel-copies between theseregisters are supported as the interpreter loads all registers and thenstores all registers.These new instructions participate in register allocation differentlyfrom before where the first few arguments are allowed to be in anyregister and no longer use `reg_fixed_use`. All other arguments (and allfloat arguments for example) continue to use `reg_fixed_use`.Locally sightglass reports this change speeding up `pulldown-cmark` by2-10%. On a `fib(N)` micro-benchmark it didn't help as much as I washoping that it was going to.* Fix MSRV List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

031a28a4 - aarch64: support udiv for 32bit integers (#9798)

Tue, 17 Dec 2024 18:47:29 +0000

aarch64: support udiv for 32bit integers (#9798)* emit 32bit udiv* winch: aarch64 udiv/urem without extension* remove stray dbg!* fmt* remove println* fix formatting in ISLE* Sized TrapIf* move operand size into CondBrKind variant* show_reg_sized fallback List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs

66989d9d - Fix minor formatting issues (#9748)

Thu, 05 Dec 2024 22:33:30 +0000

Fix minor formatting issues (#9748)* format: fix typo* format: wrap line length* format: re-wrap comment* format: organize crate dependencies List of files: /wasmtime-44.0.1/cranelift/codegen/src/machinst/buffer.rs