PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built

PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built verification of these facts after lowering to
machine instructions. We also added "memory types" describing layout
of memory and a "checked" flag on memory operations such that we could
verify that any checked memory operation accessed valid memory (as
defined by memory types attached to pointer values via
facts). Wasmtime's Cranelift backend then put appropriate memory types
and facts in its IR such that all accesses to memory (aspirationally)
could be checked, taking the whole mid-end and lowering backend of
Cranelift out of the trusted core that enforces SFI.

This basically worked, at the time, for static memories; but never for
dynamic memories, and then work on the feature lost
prioritization (aka I had to work on other things) and I wasn't able
to complete it and put it in fuzzing/enable it as a production option.

Unfortunately since then it has bit-rotted significantly -- as we add
new backend optimizations and instruction lowerings we haven't kept
the PCC framework up to date.

Inspired by the discussion in #12497 I think it's time to delete
it (hopefully just "for now"?) unless/until we can build it again. And
when we do that, we should probably get it to the point of validating
robust operation on all combinations of memory configurations before
merging. (That implies a big experiment branch rather than a bunch of
eager PRs in-tree, but so it goes.) I still believe it is possible to
build this (and I have ideas on how to do it!) but not right now.

show more ...

Cranelift: ISLE: add type parameters to most clif terms in lower (#12717)

* Cranelift: ISLE: add type parameters to most clif terms in lower

Right now in ISLE, most clif instructions have a type pa

Cranelift: ISLE: add type parameters to most clif terms in lower (#12717)

* Cranelift: ISLE: add type parameters to most clif terms in lower

Right now in ISLE, most clif instructions have a type parameter in opt for the midend, but not in lowering. From our previous discussions, this is mostly due to the order of implementation rather than a design deciscion. For our verification work, it's useful to also have the type parameters in the lowering terms, as well, if they produce values (e.g., excluding things like `store`). This is based on an earlier commit from @mmcloughlin, with changes to merge into upstream after #10524.

* Remove commented out build log

* pulley changes, cargo fmt

* fmt all

* Update cranelift/codegen/meta/src/gen_isle.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable AB

Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`. (#12160)

* Cranelift: make all non-tail, non-indirect calls patchable, and rename patchable ABI to `preserve_all`.

As discussed in this week's Cranelift meeting, we've discovered a need
to generalize the `patchable_call` mechanism and corresponding
`patchable` ABI slightly. In particular, we will need patchable
`try_call` callsites as well in order to allow breakpoint handlers to
throw exceptions (desirable functionality eventually) and have this work
in the presence of inlining. Also, it's just a nice generalization to
say that patchability is an orthogonal dimension to the call ABI and the
other restrictions we initially imposed, and works as long as the basic
requirement (no return values) is met.

This also renames the `patchable` ABI to `preserve_all`, to make it
clear that its purpose is actually orthogonal, and it can be used
independently of patchable callsites. It also deletes the `cold` ABI,
which never actually did anything and is misleading in the presence of
an actual cold-ish (subzero temperature, actually) ABI like
`preserve_all`.

* Review feedback.

show more ...

Cranelift: add patchable call instructions. (#12101)

* Cranelift: add patchable call instructions.

The new `patchable_call` CLIF instruction pairs with the `patchable`
ABI, and emits a callsite wit

Cranelift: add patchable call instructions. (#12101)

* Cranelift: add patchable call instructions.

The new `patchable_call` CLIF instruction pairs with the `patchable`
ABI, and emits a callsite with one new key property: the MachBuffer
carries metadata that describes exactly which byte range to "NOP out"
(overwrite with NOP instructions) to disable that callsite. Doing so is
semantically valid and explicitly supported.

This enables patching of code at runtime to dynamically turn on and off
features such as instrumentation or debugging hooks. We plan to use this
to implement breakpoints in Wasmtime's guest debugging support.

As part of this change, I added a notion of "unit of NOP bytes" to the
MachBuffer so that the consumer (e.g., Wasmtime's Cranelift-based code
compilation pipeline and metadata-producing logic) can handle patchable
callsites without any other special knowledge of the ISA.

For the "real metal" ISAs there are perfectly well-defined NOPs to use,
but for Pulley, where all opcodes are assigned at compile time by macro
magic, I explicitly defined NOP as opcode byte 0 by moving `Nop`'s
definition to the top of the list and adding a unit test asserting its
encoding.

A design note: in principle it would be possible, as an alternative, to
treat "patchability" as an orthogonal dimension of all callsites, and
emit the metadata describing the instruction-offset range for any
callsite with the flag set. The only truly necessary semantic
restriction is that there are no return values (because if we turn the
callsite off, nothing writes to them); we could support patchability for
other ABIs and for the other kinds of call instructions. The `patchable`
ABI would then be better described as something like the "no clobbers
ABI". I opted not to generalize in this way because it creates some
less-tested corners and the generalized form, at least at the MachInst
level, is not really much simpler in the end.

A testing note: I opted not to implement actual code patching in the
`cranelift-tools` filetest runner and test patching callsites in/out via
some actuation (e.g. a magic hostcall, like we do for throws) because
(i) that's a lot of new plumbing and (ii) we are going to test this very
shortly in Wasmtime anyway and (iii) the correctness (or not) of the
location-and-length metadata is easy enough to verify in the
disassemblies in the compile-tests.

* Review feedback: remove dependence on (and test for) NOP being the literal byte 0.

show more ...

Cranelift: use SP-offset amodes for `stack_addr`+load/store. (#11727)

We provide `stack_load`/ `stack_store` / `stack_addr` instructions in
Cranelift to operate on stack slots, and the first two are

Cranelift: use SP-offset amodes for `stack_addr`+load/store. (#11727)

We provide `stack_load`/ `stack_store` / `stack_addr` instructions in
Cranelift to operate on stack slots, and the first two are legalized
to a `stack_addr` plus an ordinary load or store instruction.

We currently have lowerings for `stack_addr` that materialize an
SP-relative address into a register: for example, `leaq 8(%rsp), %rax`
on x86-64 or `add x0, sp, #8` on aarch64.

Taken together, we see sequences like (aarch64 / x86-64)

```
add x0, sp, #8 / leaq 8(%rsp), %rax
str x1, [x0] / movq %rdx, (%rax)
```

when using `stack_store`s. In particular, we do *not* use the direct
SP-relative form, which would look like

```
str x1, [sp, #8] / movq %rdx, 8(%rsp)
```

and which we can already generate in other cases, e.g. spillslot
moves (spills/reloads) and clobber saves/restores.

This inefficiency is undesirable whenever the embedder is using
stackslots, but in particular when we expect to have high memory
traffic to stack slots (e.g., I am seeing this now when implementing
debug instrumentation in Wasmtime, and user stack map instrumentation
for GC will also benefit).

This PR adds new lowerings that use the existing synthetic address
mode we already use for spillslots to emit loads/stores to stackslots
directly when possible. The PR does this for x86-64 and aarch64;
others could be updated later.

show more ...

Cranelift: add get_exception_handler_address. (#11629)

* Cranelift: add get_exception_handler_address.

This is designed to enable applications such as #11592 that use
alternative unwinding mechanis

Cranelift: add get_exception_handler_address. (#11629)

* Cranelift: add get_exception_handler_address.

This is designed to enable applications such as #11592 that use
alternative unwinding mechanisms that may not necessarily want to walk a
stack and look up exception tables. The idea is that whenever it would
be valid to resume to an exception handler that is active on the stack,
we can provide the same PC as a first-class runtime value that would be
found in the exception table for the given handler edge. A "custom"
resume step can then use this PC as a resume-point as long as it follows
the relevant exception ABI (i.e.: restore SP, FP, any other saved
registers that the exception ABI specifies, and provide appropriate
payload value(s)).

Handlers are associated with edges out of `try_call`s (or
`try_call_indirect`s); and edges specifically, not blocks, because there
could be multiple out-edges to one block. The instruction thus takes the
block that contains the try-call and an immediate that indexes its
exceptional edges.

This CLIF instruction required a bit of infrastructure to (i) allow
naming raw blocks, not just block calls, as instruction arguments, and
(ii) allow getting the MachLabel for any other lowered block during
lowering. But given that, the lowerings themselves are straightforward
uses of MachBuffer labels to fix-up PC-relative address-loading
instructions (e.g., `LEA` or `ADR` or `AUIPC`+`ADDI`).

* Review feedback.

* Review feedback: more tests.

show more ...

aarch64: Add support for "near" in `LoadExtName` (#11570)

* aarch64: Add support for "near" in `LoadExtName`

The current aarch64 backend does not support `symbol_value` to get the
value of a functi

aarch64: Add support for "near" in `LoadExtName` (#11570)

* aarch64: Add support for "near" in `LoadExtName`

The current aarch64 backend does not support `symbol_value` to get the
value of a function, for example, with a "near" relocation using a
relative relocation. Currently it uses an `Abs8` relocation which means
that it's not suitable in Wasmtime, for example.

This commit refactors relocation/external name support in the aarch64
backend to support this mode of relocation. The previous `LoadExtName`
was split into `LoadExtName{Got,Near,Far}` where the "near" bit is
what's new to the backend. The preexisting `symbol-value.clif`-style
tests were updated to match the x64 backend which has a more
comprehensive suite of examples of what it looks like to refer to
various symbols.

The goal of this commit is to enable Wasmtime to generate code which
refers to a relative point elsewhere in the code (e.g. an exception
handler) and load the value into a register. This part isn't filled out
yet, but it seemed good to at least in the meantime fill out these
missing relocations in the backend.

* Fix clippy warning

* Add support for new relocations to cranelift-jit

Needed for filetests

show more ...

x64: Migrate `cmov*` to the new assembler (#11095)

This commit adds all `cmov*` variants from the Intel manual to the new
assembler. This then additionally removes the `Cmove` pseudo-inst in
favor o

x64: Migrate `cmov*` to the new assembler (#11095)

This commit adds all `cmov*` variants from the Intel manual to the new
assembler. This then additionally removes the `Cmove` pseudo-inst in
favor of these new instructions. One difference from before is that the
naming in the `CC` enum does not exactly match what mnemonics Capstone
uses to disassemble. For example `CC.NB` in ISLE corresponds to the
Intel instruction `CMOVNB`. This instruction, however, has the same
encoding as `CMOVAE` and Capstone disassembles as `CMOVAE`. This means
that the instruction selection in ISLE isn't a 1:1 match with mnemonics.

This additionally adds support in the assembler ISLE generation to
understand that instructions which read EFLAGS generate a
`ConsumesFlags` variant in their instruction helpers.

show more ...

Cranelift: Generate integer numeric ops and conversions for ISLE in the meta crate (#11065)

* Cranelift: Generate integer numeric ops and conversions for ISLE in the meta crate

This automatically g

Cranelift: Generate integer numeric ops and conversions for ISLE in the meta crate (#11065)

* Cranelift: Generate integer numeric ops and conversions for ISLE in the meta crate

This automatically generates operations and conversions for integer types for
use in ISLE.

Supported types are: `{i,u}{8,16,32,64,128}`

We generate

* Comparisons (eq, ne, lt, lt_eq, gt, gt_eq)
* Arithmetic operations (add, sub, mul, div, neg)
* These each have checked, wrapping, and unwrapping variants
* Bitwise operations (and, or, xor, shifts, counting leading/trailing zeros/ones)
* A variety of predicates (is_zero, is_power_of_two, is_odd, etc...)
* These generate both partial constructors and a handful of extractors
* Conversions
* These come in a variety of flavors: fallible, infallible, truncating,
unwrapping, sign-reinterpretation
* Fallible conversions are also available as an extractor

* Fix copy paste

* Rename `x_reinterpret_as_y` to `x_cast_[un]signed`

* Collapse some fallible conversions in pulley lowering

* Clean up pulley iconst lowering, make sure narrowest `xconst*` instruction is always used

* Avoid an unnecessary truncation in riscv64 lowering

* Use extractor instead of partial constructor in x64 `imm` rule

* Clean up `op mem, imm` x64 lowering rules

* Use `(i64_eq a b)` instead of `(u64_eq (i64_cast_unsigned a) (i64_cast_unsigned b))`

* Rename `<ty>_unwrapping_<op>` to `<ty>_<op>`

show more ...

Add lowering rules for `{add,sub,or,and} mem, imm` on x64 (#11043)

* Add lowering rules for `{add,sub,or,and} mem, imm` on x64

* Fold `<op> reg, imm` lowering rules into existing `x64_op_mem` helpe

Add lowering rules for `{add,sub,or,and} mem, imm` on x64 (#11043)

* Add lowering rules for `{add,sub,or,and} mem, imm` on x64

* Fold `<op> reg, imm` lowering rules into existing `x64_op_mem` helpers

* Remove duplicate decl that got added elsewhere in a different commit on main

show more ...

[s390x] Use VCodeConstant literal pool (#10756)

This switches s390x to use the common-code VCodeConstant
literal pool instead of constants inlined into the instruction
stream.

Refactor call ABI implementation (#10722)

This refactors implementation of call ABI handling across architectures
with the goal of bringing s390x in line with other platforms.

The main idea is to
-

Refactor call ABI implementation (#10722)

This refactors implementation of call ABI handling across architectures
with the goal of bringing s390x in line with other platforms.

The main idea is to
- handle main call instruction selection and generation in ISLE
(like s390x but unlike other platforms today)
- handle argument setup mostly outside of ISLE
(like other platforms but unlike s390x today)
- handle return value processing as part of the call instructio
(like all platforms today)

All platforms now emit the main call instruction directly from ISLE,
which e.g. handles selection of the correct ISA instruction depending
on the call destination. This ISLE code calls out to helper routines
to handle argument and return value processing. These helpers are
mostly common code and provided by the Callee and/or Lower layers,
with some platform-specific additions via ISLE Context routines.

The old CallSite abstraction is no longer needed; most of the
differences between call and return_call handling disappear.
(There is still a common-code CallInfo vs. a platform-specifc
ReturnCallInfo. At this point, it should be relatively straight-
forward to make CallInfo platform-specific as well if desired,
but this is not done here.)

Some ISLE infrastructure for iterators / loops, which was only
ever used by the s390x argument processing code, has been removed.

s390x now closely matches all other platforms, with only a few
special cases (slightly different tail-call ABI requires some
differences in stack offset computations; we still need to
handle vector lane swaps for cross-ABI calls), which should
simplify future maintenance.

show more ...

Cranelift: simplify some side-effectful instructions in ISLE (#10524)

* Cranelift: simplify some side-effectful instructions in ISLE

This commit adds a new top-level ISLE entrypoint specifically fo

Cranelift: simplify some side-effectful instructions in ISLE (#10524)

* Cranelift: simplify some side-effectful instructions in ISLE

This commit adds a new top-level ISLE entrypoint specifically for instructions
in the side-effectful skeleton: `simplify_skeleton`. While these rewrites are
processed during the egraph pass, values from skeleton instructions still do not
get inserted into the egraph. Indeed, `simplify_skeleton` operates
on *instructions* rather than *values* because we do not represent side effects
as values; values do not have side effects in CLIF, instructions do. Therefore,
rather than doing a whole dynamic-programming style extraction of the best
candidate simplification like we do with the egraph, we take an eager and greedy
approach.

Furthermore, `simplify_skeleton` is limited only to skeleton instructions that
do not involve control-flow or terminators right now. This is because changing
the control-flow graph can change whether a use is dominated by a def or not,
and we do not currently have the machinery to track and fix up invalidated
uses. Addressing this is left for future commits.

* fix `MIN / -1` cprop and add negative tests for things simplify_skeleton cannot handle yet

show more ...

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect`

Cranelift: initial try_call / try_call_indirect (exception) support. (#10510)

* Cranelift: initial try_call / try_call_indirect (exception) support.

This PR adds `try_call` and `try_call_indirect` instructions, and
lowerings on four of five ISAs (x86-64, aarch64, riscv64, pulley; s390x
has its own non-shared ABI code that will need separate work).

It extends CLIF to support these instructions as new kinds of branches,
and extends block-calls to accept `retN` and `exnN` block-call args that
carry the normal return values or exception payloads (respectively) into
the appropriate successor blocks.

It wires up the "normal return path" so that it continues to work.
It updates the ABI so that unwinding is possible without an initial
register state at throw: specifically, as per our RFC, all registers are
clobbered. It also includes metadata in the `MachBuffer` that describes
exception-catch destinations. However, no unwinder exists to interpret
these catch-destinations yet, so they are untested.

* Add try_call_indirect lowering as well.

show more ...

asm: add `LOCK`-prefixed instructions (#10477)

* asm: add initial `LOCK`-prefixed instructions

This change is the first in a series to add new instructions that emit
the `LOCK` prefix during encodi

asm: add `LOCK`-prefixed instructions (#10477)

* asm: add initial `LOCK`-prefixed instructions

This change is the first in a series to add new instructions that emit
the `LOCK` prefix during encoding.

* asm: add all `lock and*` instructions

This defines all `LOCK`-prefixed variants for `and*`.

* asm: add all `lock add|adc*` instructions

* asm: add all `lock or*` instructions

* asm: add all `lock sub|sbb*` instructions

* asm: add all `lock xor*` instructions

* asm: reformat using slightly longer lines

This allows the `LOCK`-prefixed definitions to still fit on a single
line.

* asm: integrate mem-only operands in `cranelift-codegen`

`LOCK`-prefixed instructions can only write to memory operands (see
prior commits). This change wires up the necessary codegen to handle the
new `m*` operands.

* asm: lower RMW instructions with new `LOCK`-prefixed instructions

* Bump allowed size of the `Inst` enum

It is unclear why this would happen now, though perhaps the addition of
all the `LOCK`-prefixed instructions could do it (?).

show more ...

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stor

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stores are translated to Pulley opcodes when using the
interpreter. Additionally the functionality related to memory support
has changed quite a bit with the interpreter as well. This is all based
off comments on #10102 with the end goal of folding the two Pulley
opcodes today of "do the bounds check" and "do the load" into one
opcode. This is intended to reduce the number of opcodes and overall
improve interpreter throughput by minimizing turns of the interpreter
loop.

The basic idea behind this PR is that a new basic suite of loads/stores
are added to Pulley which trap if the address is zero. This provides a
route to translate trapping loads/stores in CLIF to Pulley bytecode
without actually causing segfaults at runtime. WebAssembly translation
to CLIF is then updated to use the `select` trick for wasm loads/stores
where either 0 is loaded from or the actual address is loaded from.
Basic support for translation and such is added for this everywhere, and
this ensures that all loads/stores for wasm will be translated
successfully with Pulley.

The next step was to extend the "g32" addressing mode preexisting in
Pulley to support a bounds check as well. New pattern-matches were added
to ISLE to search for a bounds check in the address of a trapping
load/store. If found then the entire chain of operations necessary to
compute the address are folded into a single "g32" opcode which ends up
being a fallible load/store at runtime.

To fit all this into Pulley this commit contains a number of
refactorings to shuffle around existing opcodes related to memory and
extend various pieces of functionality here and there:

* Pulley now uses a `AddrFoo` types to represent addressing modes as a
single immediate rather than splitting it up into pieces for each
method. For example `AddrO32` represents "base + offset32". `AddrZ`
represents the same thing but traps if the address is zero. The
`AddrG32` mode represents a bounds-checked 32-bit linear memory access
on behalf of wasm.

* Pulley loads/stores were reduced to always using an `AddrFoo`
immediate. This means that the old `offset8` addressing mode was
removed without replacement here (to be added in the future if
necessary). Additionally the suite of sign-extension modes supported
were trimmed down to remove 8-to-64, 16-to-64, and 32-to-64 extensions
folded as part of the opcode. These can of course always be re-added
later but probably want to be added just for the `G32` addressing mode
as opposed to all addressing modes.

* The interpreter itself was refactored to have an `AddressingMode`
trait to ensure that all memory accesses, regardless of addressing
modes, are largely just copy/pastes of each other. In the future it
might make sense to implement these methods with a macro, but for now
it's copy/paste.

* In ISLE the `XLoad` generic instruction removed its `ext` field to
have extensions handled exclusively in ISLE instead of partly in
`emit.rs`.

* Float/vector loads/stores now have "g32" addressing (in addition to
the "z" that's required for wasm) since it was easy to add them.

* Translation of 1-byte accesses on Pulley from WebAssembly to CLIF no
longer has a special case for using `a >= b` instead of `a > b - 1` to
ensure that the same bounds-check instruction can be used for all
sizes of loads/stores.

* The bounds-check which folded a load-of-the-bound into the opcode is
now present as a "g32bne" addressing mode. with its of suite of
instructions to boo.

Overall this PR is not a 1:1 replacement of all previous opcodes with
exactly one opcode. For example loading 8 bits sign-extended to 64-bits
is now two opcodes instead of one. Additionally some previous opcodes
have expanded in size where for example the 8-bit offset mode was remove
in favor of only having 32-bit offsets. The goal of this PR is to reboot
how memory is handled in Pulley. All loads/stores now use a specific
addressing mode and currently all operations supported across addressing
modes are consistently supported. In the future it's expected that some
features will be added to some addressing modes and not others as
necessary, for example extending the "g32" addressing mode only instead
of all addressing modes.

For an evaluation of this PR:

* Code size: `spidermonkey.cwasm` file is reduced from 19M to 16M.
* Sightglass: `pulldown-cmark` is improved by 15%
* Sightglass: `bz2` is improved by 20%
* Sightglass: `spidermonkey` is improved by 22%
* Coremark: score improved by 40%

Overall this PR and new design looks to be a large win. This is all
driven by the reduction in opcodes both for compiled code size and
execution speed by minimizing turns of the interpreter loop. In the end
I'm also pretty happy with how this turned out and I think the
refactorings are well worth it.

* Use new `is_pulley` helper more

* Improve `addrz` helper, tighten up `memory-inbounds.wat` a bit

* Improve codegen in a few `memory-inbounds.wat` cases

* Fix test expectation

show more ...

pulley: Add immediate payloads to more opcodes (#9861)

* pulley: Add immediate payloads to more opcodes

This commit adds immediate payloads to the following instructions:

* `xmul32` - `xmul32_s8`

pulley: Add immediate payloads to more opcodes (#9861)

* pulley: Add immediate payloads to more opcodes

This commit adds immediate payloads to the following instructions:

* `xmul32` - `xmul32_s8` / `xmul32_s32`
* `xmul64` - `xmul64_s8` / `xmul64_s32`
* `xband32` - `xband32_s8` / `xband32_s32`
* `xband64` - `xband64_s8` / `xband64_s32`
* `xbor32` - `xbor32_s8` / `xbor32_s32`
* `xbor64` - `xbor64_s8` / `xbor64_s32`
* `xbxor32` - `xbxor32_s8` / `xbxor32_s32`
* `xbxor64` - `xbxor64_s8` / `xbxor64_s32`
* `xshl32` - `xshl32_u6`
* `xshl64` - `xshl64_u6`
* `xshr32_u` - `xshl32_u_u6`
* `xshr64_u` - `xshl64_u_u6`
* `xshr32_s` - `xshl32_s_u6`
* `xshr64_s` - `xshl64_s_u6`

For shifts there's no need to have 32-bit immediates (or even 8-bit)
since 6 bits is enough to encode all the immediates. This means that the
6-bit immediate is packed within `BinaryOperands` as a new `U6` type.

This commit unfortunately does not shrink `spidermonkey.cwasm`
significantly beyond the prior 29M. This is nevertheless expected to be
relatively important for performance.

* Fix test expectations

show more ...

pulley: Add add/sub-with immediate opcodes (#9859)

This commit extends the pulley opcode space with integer
addition/subtraction where `src2` is an immediate. The goal here is to
be a "sort of macro

pulley: Add add/sub-with immediate opcodes (#9859)

This commit extends the pulley opcode space with integer
addition/subtraction where `src2` is an immediate. The goal here is to
be a "sort of macro instruction" despite it not being too too macro
here. This cuts down on `xconstN` instructions which both saves space in
the final binary and should be slightly more optimal perf-wise due to
only one dispatch being needed.

In this commit the `xadd32` instruction is previously 3 bytes: one for
an opcode and 2 bytes for the dst/src1/src2 binary operands. Adding a
small constant to a register previously took 5 bytes where 2 bytes were
needed for `xconst8 N` then 3 for the addition. Here the encoding size
of the new instruction is 4 bytes: 1 for the opcode, 2 for dst/src1, and
one for the immediate. This is currently chosen to mostly optimize
dispatch in the interpreter loop as opposed to code size (as only a
single byte is saved). In the future thought it would be possible to
extend `BinaryOperands` to one operand being a 6-bit immediate to
preserve the same code size.

This also notably adds, for addition/subtraction, only unsigned
immediates. With addition/subtraction being inverses of one another
supporting signed immediates isn't necessary and helps free up another
bit for packing numbers into these opcodes.

This change reduces the size of `spidermonkey.cwasm` from 31M to 29M
locally.

show more ...

ISLE: Add proper bool type (#9593)

* ISLE: add proper booleans (expressions)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* ISLE: add proper booleans (pattern

ISLE: Add proper bool type (#9593)

* ISLE: add proper booleans (expressions)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* ISLE: add proper booleans (patterns)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* ISLE: add proper booleans (spec expressions)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* ISLE: replace opaque boolean constants

Copyright (c) 2024, Arm Limited.

Replace all occurences of `$true` and `$false` with `true` and `false`.

Signed-off-by: Karl Meakin <[email protected]>

* ISLE: remove `on_lhs` argument

Instead of threading `on_lhs` through all the calls to `translate_expr`,
we can just set `is_partial` and `is_pure` on `root_flags` to true.

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

* ISLE: add proper booleans (language reference)

Copyright (c) 2024, Arm Limited.

Signed-off-by: Karl Meakin <[email protected]>

---------

Signed-off-by: Karl Meakin <[email protected]>

show more ...

Gate support for implicit return area pointers behind an option (#9511)

* Remove couple of unused isle helpers

* Gate support for implicit return area pointers behind an option

It implements an in

Gate support for implicit return area pointers behind an option (#9511)

* Remove couple of unused isle helpers

* Gate support for implicit return area pointers behind an option

It implements an incorrect ABI and may be removed in the future due to
complexity reasons. By requiring to enable an option to use it, it
becomes harder to accidentally hit the ABI issue and allows a more
deprecation and eventual removal.

* Fix review comments

* Enable enable_multi_ret_implicit_sret for s390x tests that use i128

* Enable enable_multi_ret_implicit_sret for riscv tests that use vector types

* Enable enable_multi_ret_implicit_sret for more riscv tests

* Enable enable_multi_ret_implicit_sret for windows tests that use i128

show more ...

ISLE: upstream prototype ISLE verifier (Crocus) (#9178)

* Squash of verification changes 2024-08-28

* Cleanup

* fmt

* More cleanup

* clippy

* update example spec

* fmt

* null

* clippy part 1

ISLE: upstream prototype ISLE verifier (Crocus) (#9178)

* Squash of verification changes 2024-08-28

* Cleanup

* fmt

* More cleanup

* clippy

* update example spec

* fmt

* null

* clippy part 1

* clippy fix, null

* clippy part 3

* fmt

* disable veri_engine tests in CI

* clap dependency

* remove strum

* use easy-smt version

* Get `cargo vet/deny` working and passing

Undoes a seeming `cargo update` that was performed previously and then
adds a vet for the new crate added.

* Flag new crates as not published

* Revert "remove strum"

This reverts commit df7b540b51ebf14d45a7b9f9bbedeba018a53e9a.

* itertools

* Review feedback

* Future work comments for encodings

* fmt

* post merge Cargo.lock updates

* authors

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Remove StructArgument support from the arm64, riscv64 and s390x backends (#9258)

Unlike x86_64 and some other architectures where large structs are
passed at fixed stack offsets, necessitating speci

Remove StructArgument support from the arm64, riscv64 and s390x backends (#9258)

Unlike x86_64 and some other architectures where large structs are
passed at fixed stack offsets, necessitating special Cranelift support,
arm64, riscv64 and s390x pass them using regular pointer arguments that
can trivially be implemented by frontends.

The current implementation of StructArgument on arm64 and riscv64 uses
fixed stack offsets just like on x86_64, which is incorrect as per the
ABI documentation. In other words any correct frontend already had
to avoid StructArgument on arm64 and riscv64 anyway. For s390x it is
correctly implemented but unnecessary and taints even non-s390x specific
code.

show more ...

Optimize 128-bit multiplication some more in backends (#9215)

* Add some tests that will get optimized next

These tests currently reflect the output of Cranelift today and will
showcase the diff i

Optimize 128-bit multiplication some more in backends (#9215)

* Add some tests that will get optimized next

These tests currently reflect the output of Cranelift today and will
showcase the diff in the next commit of the optimizations implemented.

* Optimize 128-bit multiplication some more in backends

This commits adds support for more patterns of improved 128-bit
multiplication to the various backends of Cranelift. Notably:

* Using `isplit` to discard the lower bits of a multiplication maps
directly to instructions on aarch64 and riscv64.
* Multiplying sign-extended 64-bit halves maps directly to supported
instructions on s390x/riscv64/aarch64 (x86_64 already has these
implemented).

This relies on adding a new method to test whether a `Value` is dead and
unused during lowering. While generally not useful this is applicable
for multi-result instructions such as `isplit` where one result may end
up not being used. This also is required because the egraph
optimizations can't currently handle multi-result instructions like
`isplit` so this can't be optimized to `umulhi` or `smulhi` in CLIF.

show more ...

Cranelift: add stack_switch CLIF instruction (#9078)

* stack_switch instruction

* Update cranelift/codegen/src/isa/x64/pcc.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

* cargo fmt

* on

Cranelift: add stack_switch CLIF instruction (#9078)

* stack_switch instruction

* Update cranelift/codegen/src/isa/x64/pcc.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

* cargo fmt

* only lower on linux

* give stack_switch the call() side effect

* add function in filetest doing switching only

* Extend documentation of new instruction

* better comments on how we handle the payloads

* Revert "only lower on linux"

This reverts commit 2af10f944186629de1615aa0ed999b7f49d13132.

* Add StackSwitchModel, use to compile stack_switch

* turn stack_switch_model into partial constructor

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

s390x: Support tail calls (#9052)

This adds support for a newly defined tail-call ABI for s390x
as well as supporting tail calls themselves.

We now use the tail-call ABI by default for Wasmtime, an

s390x: Support tail calls (#9052)

This adds support for a newly defined tail-call ABI for s390x
as well as supporting tail calls themselves.

We now use the tail-call ABI by default for Wasmtime, and
enable tail calls by default.

This also allows getting rid of a number of special case
and test exclusions for s390x.

Fixes: https://github.com/bytecodealliance/wasmtime/issues/6530

show more ...