Cranelift: Make `ir::{Constant,Immediate}` considered entities (#11207)

* Cranelift: Make `ir::{Constant,Immediate}` considered entities

They reference data in out-of-line pools rather than storing

Cranelift: Make `ir::{Constant,Immediate}` considered entities (#11207)

* Cranelift: Make `ir::{Constant,Immediate}` considered entities

They reference data in out-of-line pools rather than storing their data inline
in the instruction, and when an instruction containing them is moved from one
`ir::Function` to another, they need their indices updated
accordingly. Therefore, they really are entities rather than immediates.

This recategorization means that they will now be properly mapped in
`ir::InstructionData::map` calls.

* fix tests

show more ...

Convert `TrapCode` to a single byte (#9338)

* Convert `TrapCode` to a single byte

This commit refactors the representation of
`cranelift_codegen::ir::TrapCode` to be a single byte. The previous
enu

Convert `TrapCode` to a single byte (#9338)

* Convert `TrapCode` to a single byte

This commit refactors the representation of
`cranelift_codegen::ir::TrapCode` to be a single byte. The previous
enumeration is replaced with an opaque byte-sized structure. Previous
variants that Cranelift uses internally are now associated `const`
values on `TrapCode` itself. For example `TrapCode::IntegerOverflow` is
now `TrapCode::INTEGER_OVERFLOW`. All non-Cranelift traps are now
removed and exclusively live in the `wasmtime-cranelift` crate now.

The representation of a `TrapCode` is now:

* 0 - invalid, used in `MemFlags` for "no trap code"
* 1..256-N - user traps
* 256-N..256 - built-in Cranelift traps (it uses N of these)

This enables embedders to have 255-N trap codes which is more than
enough for Wasmtime for example. Cranelift reserves a few built-in codes
for itself which shouldn't eat too much into the trap space.
Additionally if Cranelift needs to grow a new trap it can do so pretty
easily too.

The overall intent of this commit is to reduce the coupling of Wasmtime
and Cranelift further and generally refactor Wasmtime to use user traps
more often. This additionally shrinks the size of `TrapCode` for storage
in various locations, notably it can now infallibly be represented
inside of a `MemFlags`.

Closes #9310

* Fix some more tests

* Fix more tests

* Fix even more tests

* Review comments

* Fix tests

* Fix rebase conflict

* Update test expectations

show more ...

cranelift: Add `f16const` and `f128const` instructions (#8893)

* cranelift: Add `f16const` and `f128const` instructions

* cranelift: Add constant propagation for `f16` and `f128`

riscv64: Refactor FRM and fcvt-to-int management (#7327)

* riscv64: Specify rounding modes in instructions

This commit updates how floating-point instructions specify their float
rounding mode (FRM

riscv64: Refactor FRM and fcvt-to-int management (#7327)

* riscv64: Specify rounding modes in instructions

This commit updates how floating-point instructions specify their float
rounding mode (FRM). Previously instructions stored `Option<FRM>` and
this would mostly be `None`. All floating-point instructions in RISC-V
have a 3-bit `rm` field, and most encode the FRM into this field but
some have a require encoding of this field. For example `fsgnj.s` uses
the `rm` field to differentiate between `fsgnj`, `fsgnjx`, and `fsgnjn`.
Instructions like `fadd` however use this field for a rounding mode.

All FPU instructions now store `FRM` directly. Instruction helpers like
`fadd` require this to be specified explicitly. Instructions helpers
like for `fsgnj` do not take this as an argument and hardcode the field
as necessary. This means that all lowerings of floating point
instructions, where relevant, now specify a rounding mode.

Previously the default rounding mode was to use the `fcsr` register,
meaning that the rounding mode would be determined dynamically at
runtime depending on the status of this register. Cranelift semantics,
however, are derivative of WebAssembly semantics which specify
round-to-nearest ties-to-even. This PR additionally fixes this
discrepancy by using `FRM::RNE` in all existing instructions instead of
`FRM::Fcsr`.

* riscv64: Refactor float-to-int conversions

This commit removes the `FcvtToInt` macro-instruction in the riscv64
backend in favor of decomposing it into individual operation for
`fcvt_to_{s,u}int*` instructions. This additionally provides a slightly
different lowering for the `*_sat` operations which doesn't use
branches. The non-saturating operations continue to have a number of
branches and their code has changed slightly due to how immediates are
loaded. Overall everything is in ISLE now instead of split a bit.

* riscv64: Clean up some dead code in the backend

Don't put `#![allow(dead_code)]` at the root, instead place it on some
smaller items.

* Fix emission tests

* Add regression tests and bless output

Closes #5992
Closes #5993

* Enable i8/i16 saturating float-to-int in fuzzgen

* Better `fcvt_*_bound` implementations

* Fix typo in match orderings

* Fix tests on x64

Where float-to-int isn't implemented for i8/i16

show more ...

Remove heaps from core Cranelift, push them into `cranelift-wasm` (#5386)

* cranelift-wasm: translate Wasm loads into lower-level CLIF operations

Rather than using `heap_{load,store,addr}`.

*

Remove heaps from core Cranelift, push them into `cranelift-wasm` (#5386)

* cranelift-wasm: translate Wasm loads into lower-level CLIF operations

Rather than using `heap_{load,store,addr}`.

* cranelift: Remove the `heap_{addr,load,store}` instructions

These are now legalized in the `cranelift-wasm` frontend.

* cranelift: Remove the `ir::Heap` entity from CLIF

* Port basic memory operation tests to .wat filetests

* Remove test for verifying CLIF heaps

* Remove `heap_addr` from replace_branching_instructions_and_cfg_predecessors.clif test

* Remove `heap_addr` from readonly.clif test

* Remove `heap_addr` from `table_addr.clif` test

* Remove `heap_addr` from the simd-fvpromote_low.clif test

* Remove `heap_addr` from simd-fvdemote.clif test

* Remove `heap_addr` from the load-op-store.clif test

* Remove the CLIF heap runtest

* Remove `heap_addr` from the global_value.clif test

* Remove `heap_addr` from fpromote.clif runtests

* Remove `heap_addr` from fdemote.clif runtests

* Remove `heap_addr` from memory.clif parser test

* Remove `heap_addr` from reject_load_readonly.clif test

* Remove `heap_addr` from reject_load_notrap.clif test

* Remove `heap_addr` from load_readonly_notrap.clif test

* Remove `static-heap-without-guard-pages.clif` test

Will be subsumed when we port `make-heap-load-store-tests.sh` to generating
`.wat` tests.

* Remove `static-heap-with-guard-pages.clif` test

Will be subsumed when we port `make-heap-load-store-tests.sh` over to `.wat`
tests.

* Remove more heap tests

These will be subsumed by porting `make-heap-load-store-tests.sh` over to `.wat`
tests.

* Remove `heap_addr` from `simple-alias.clif` test

* Remove `heap_addr` from partial-redundancy.clif test

* Remove `heap_addr` from multiple-blocks.clif test

* Remove `heap_addr` from fence.clif test

* Remove `heap_addr` from extends.clif test

* Remove runtests that rely on heaps

Heaps are not a thing in CLIF or the interpreter anymore

* Add generated load/store `.wat` tests

* Enable memory-related wasm features in `.wat` tests

* Remove CLIF heap from fcmp-mem-bug.clif test

* Add a mode for compiling `.wat` all the way to assembly in filetests

* Also generate WAT to assembly tests in `make-load-store-tests.sh`

* cargo fmt

* Reinstate `f{de,pro}mote.clif` tests without the heap bits

* Remove undefined doc link

* Remove outdated SVG and dot file from docs

* Add docs about `None` returns for base address computation helpers

* Factor out `env.heap_access_spectre_mitigation()` to a local

* Expand docs for `FuncEnvironment::heaps` trait method

* Restore f{de,pro}mote+load clif runtests with stack memory

show more ...

Cranelift: Add `heap_load` and `heap_store` instructions (#5300)

* Cranelift: Define `heap_load` and `heap_store` instructions

* Cranelift: Implement interpreter support for `heap_load` and `heap

Cranelift: Add `heap_load` and `heap_store` instructions (#5300)

* Cranelift: Define `heap_load` and `heap_store` instructions

* Cranelift: Implement interpreter support for `heap_load` and `heap_store`

* Cranelift: Add a suite runtests for `heap_{load,store}`

There are so many knobs we can twist for heaps and I wanted to exhaustively test
all of them, so I wrote a script to generate the tests. I've checked in the
script in case we want to make any changes in the future, but I don't think it
is worth adding this to CI to check that scripts are up to date or anything like
that.

* Review feedback

show more ...

cranelift: Remove booleans (#5031)

Remove the boolean types from cranelift, and the associated instructions breduce, bextend, bconst, and bint. Standardize on using 1/0 for the return value from ins

cranelift: Remove booleans (#5031)

Remove the boolean types from cranelift, and the associated instructions breduce, bextend, bconst, and bint. Standardize on using 1/0 for the return value from instructions that produce scalar boolean results, and -1/0 for boolean vector elements.

Fixes #3205

Co-authored-by: Afonso Bordado <[email protected]>
Co-authored-by: Ulrich Weigand <[email protected]>
Co-authored-by: Chris Fallin <[email protected]>

show more ...

cranelift: Remove of/nof overflow flags from icmp (#4879)

* cranelift: Remove of/nof overflow flags from icmp

Neither Wasmtime nor cg-clif use these flags under any circumstances.
From discussio

cranelift: Remove of/nof overflow flags from icmp (#4879)

* cranelift: Remove of/nof overflow flags from icmp

Neither Wasmtime nor cg-clif use these flags under any circumstances.
From discussion on #3060 I see it's long been unclear what purpose these
flags served.

Fixes #3060, fixes #4406, and fixes #4875... by deleting all the code
that could have been buggy.

This changes the cranelift-fuzzgen input format by removing some IntCC
options, so I've gone ahead and enabled I128 icmp tests at the same
time. Since only the of/nof cases were failing before, I expect these to
work.

* Restore trapif tests

It's still useful to validate that iadd_ifcout's iflags result can be
forwarded correctly to trapif, and for that purpose it doesn't really
matter what condition code is checked.

show more ...

Enforce all OperandKind have documentation

Remove more old backend ISA concepts (#3402)

This also paves the way for unifying TargetIsa and MachBackend, since now they map one to one. In theory the two traits could be merged, which would be n

Remove more old backend ISA concepts (#3402)

This also paves the way for unifying TargetIsa and MachBackend, since now they map one to one. In theory the two traits could be merged, which would be nice to limit the number of total concepts. Also they have quite different responsibilities, so it might be fine to keep them separate.

Interestingly, this PR started as removing RegInfo from the TargetIsa trait since the adapter returned a dummy value there. From the fallout, noticed that all Display implementations didn't needed an ISA anymore (since these were only used to render ISA specific registers). Also the whole family of RegInfo / ValueLoc / RegUnit was exclusively used for the old backend, and these could be removed. Notably, some IR instructions needed to be removed, because they were using RegUnit too: this was the oddball of regfill / regmove / regspill / copy_special, which were IR instructions inserted by the old regalloc. Fare thee well!

show more ...

More atomic ops

Implement Wasm Atomics for Cranelift/newBE/aarch64.

The implementation is pretty straightforward. Wasm atomic instructions fall
into 5 groups

* atomic read-modify-write
* atomic compare-and-swap
*

Implement Wasm Atomics for Cranelift/newBE/aarch64.

The implementation is pretty straightforward. Wasm atomic instructions fall
into 5 groups

* atomic read-modify-write
* atomic compare-and-swap
* atomic loads
* atomic stores
* fences

and the implementation mirrors that structure, at both the CLIF and AArch64
levels.

At the CLIF level, there are five new instructions, one for each group. Some
comments about these:

* for those that take addresses (all except fences), the address is contained
entirely in a single `Value`; there is no offset field as there is with
normal loads and stores. Wasm atomics require alignment checks, and
removing the offset makes implementation of those checks a bit simpler.

* atomic loads and stores get their own instructions, rather than reusing the
existing load and store instructions, for two reasons:

- per above comment, makes alignment checking simpler

- reuse of existing loads and stores would require extension of `MemFlags`
to indicate atomicity, which sounds semantically unclean. For example,
then *any* instruction carrying `MemFlags` could be marked as atomic, even
in cases where it is meaningless or ambiguous.

* I tried to specify, in comments, the behaviour of these instructions as
tightly as I could. Unfortunately there is no way (per my limited CLIF
knowledge) to enforce the constraint that they may only be used on I8, I16,
I32 and I64 types, and in particular not on floating point or vector types.

The translation from Wasm to CLIF, in `code_translator.rs` is unremarkable.

At the AArch64 level, there are also five new instructions, one for each
group. All of them except `::Fence` contain multiple real machine
instructions. Atomic r-m-w and atomic c-a-s are emitted as the usual
load-linked store-conditional loops, guarded at both ends by memory fences.
Atomic loads and stores are emitted as a load preceded by a fence, and a store
followed by a fence, respectively. The amount of fencing may be overkill, but
it reflects exactly what the SM Wasm baseline compiler for AArch64 does.

One reason to implement r-m-w and c-a-s as a single insn which is expanded
only at emission time is that we must be very careful what instructions we
allow in between the load-linked and store-conditional. In particular, we
cannot allow *any* extra memory transactions in there, since -- particularly
on low-end hardware -- that might cause the transaction to fail, hence
deadlocking the generated code. That implies that we can't present the LL/SC
loop to the register allocator as its constituent instructions, since it might
insert spills anywhere. Hence we must present it as a single indivisible
unit, as we do here. It also has the benefit of reducing the total amount of
work the RA has to do.

The only other notable feature of the r-m-w and c-a-s translations into
AArch64 code, is that they both need a scratch register internally. Rather
than faking one up by claiming, in `get_regs` that it modifies an extra
scratch register, and having to have a dummy initialisation of it, these new
instructions (`::LLSC` and `::CAS`) simply use fixed registers in the range
x24-x28. We rely on the RA's ability to coalesce V<-->R copies to make the
cost of the resulting extra copies zero or almost zero. x24-x28 are chosen so
as to be call-clobbered, hence their use is less likely to interfere with long
live ranges that span calls.

One subtlety regarding the use of completely fixed input and output registers
is that we must be careful how the surrounding copy from/to of the arg/result
registers is done. In particular, it is not safe to simply emit copies in
some arbitrary order if one of the arg registers is a real reg. For that
reason, the arguments are first moved into virtual regs if they are not
already there, using a new method `<LowerCtx for Lower>::ensure_in_vreg`.
Again, we rely on coalescing to turn them into no-ops in the common case.

There is also a ridealong fix for the AArch64 lowering case for
`Opcode::Trapif | Opcode::Trapff`, which removes a bug in which two trap insns
in a row were generated.

In the patch as submitted there are 6 "FIXME JRS" comments, which mark things
which I believe to be correct, but for which I would appreciate a second
opinion. Unless otherwise directed, I will remove them for the final commit
but leave the associated code/comments unchanged.

show more ...

[meta] Remove the OperandKindBuilder;

And replace it by constructors in OperandKind. There's a single optional
parameter function `set_doc` that remains, and didn't justify the whole
OperandKindBuil

[meta] Remove the OperandKindBuilder;

And replace it by constructors in OperandKind. There's a single optional
parameter function `set_doc` that remains, and didn't justify the whole
OperandKindBuilder concept to exist.

show more ...

[meta] Remove OperandKind::name field and explicitly pass rust_field_name/rust_type; (fixes #1177)

[meta] Rename OperandKind::default_member to format_field_name;

[codegen] add intcc conditions for reading overflow flag

Add conditions to IntCC for checking the overflow flag (Overflow,
NotOverflow).

Add x86 implementation of shuffle

[meta] Morph a few pub into pub(crate), and remove dead code;

[meta] Introduce the Immediates structure instead of using dynamic lookup;

Add operand kind and format for unsigned 128-bit immediates

[meta] Make Builders build() instead of finish();

[meta] Fix condition codes in immediates;

[meta] Port Formats and Operands to the Rust crate;