PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built

PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built verification of these facts after lowering to
machine instructions. We also added "memory types" describing layout
of memory and a "checked" flag on memory operations such that we could
verify that any checked memory operation accessed valid memory (as
defined by memory types attached to pointer values via
facts). Wasmtime's Cranelift backend then put appropriate memory types
and facts in its IR such that all accesses to memory (aspirationally)
could be checked, taking the whole mid-end and lowering backend of
Cranelift out of the trusted core that enforces SFI.

This basically worked, at the time, for static memories; but never for
dynamic memories, and then work on the feature lost
prioritization (aka I had to work on other things) and I wasn't able
to complete it and put it in fuzzing/enable it as a production option.

Unfortunately since then it has bit-rotted significantly -- as we add
new backend optimizations and instruction lowerings we haven't kept
the PCC framework up to date.

Inspired by the discussion in #12497 I think it's time to delete
it (hopefully just "for now"?) unless/until we can build it again. And
when we do that, we should probably get it to the point of validating
robust operation on all combinations of memory configurations before
merging. (That implies a big experiment branch rather than a bunch of
eager PRs in-tree, but so it goes.) I still believe it is possible to
build this (and I have ideas on how to do it!) but not right now.

show more ...

cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

Couple cleanups to the flags/settings handling in Cranelift (#11744)

* Remove unused shared flags

* Get rid of predicate settings

They were important in the old backend framework, but with the new

Couple cleanups to the flags/settings handling in Cranelift (#11744)

* Remove unused shared flags

* Get rid of predicate settings

They were important in the old backend framework, but with the new
backend framework if we need a combination of multiple settings, that
can just be done as a regular extractor doing &&. This simplifies the
settings implementation.

show more ...

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conver

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conversions for hashing the TargetIsa

* Remove func_body_len

It is identical to buffer.data.len()

* Introduce IsaFlagsHashKey

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Introduce log2_min_function_alignment flag (#10391)

* Remove function_alignment handling from cranelift-object and cranelift-jit

It is already handled by MachBuffer. The symbol_alignment could also

Introduce log2_min_function_alignment flag (#10391)

* Remove function_alignment handling from cranelift-object and cranelift-jit

It is already handled by MachBuffer. The symbol_alignment could also be
removed as no current backend has a symbol alignment bigger than the
function alignment, but keeping it around is a bit safer when new
backends are introduced.

* Introduce log2_min_function_alignment flag

This is required for cg_clif to implement -Zmin-function-alignment.

show more ...

Enable `impl-trait-overcaptures` 2024 transition lint (#9965)

* Enable `impl-trait-overcaptures` 2024 transition lint

This lint detects cases where returning `impl Trait` will work
differently in 2

Enable `impl-trait-overcaptures` 2024 transition lint (#9965)

* Enable `impl-trait-overcaptures` 2024 transition lint

This lint detects cases where returning `impl Trait` will work
differently in 2024 than in the current 2021 edition. Ambiguities are
resolved with `use<..>` syntax stabilized in Rust 1.82.0 to mean the
same thing in both editions.

* Fix some more `impl Trait` returns

* Tighten bounds on settings `iter`

* Fix build on 1.82.0

* Fix another capture on MSRV

show more ...

Cranelift: add option to use new single-pass register allocator. (#9611)

* Cranelift: add option to use new single-pass register allocator.

In bytecodealliance/regalloc2#181, @d-sonuga added a fast

Cranelift: add option to use new single-pass register allocator. (#9611)

* Cranelift: add option to use new single-pass register allocator.

In bytecodealliance/regalloc2#181, @d-sonuga added a fast single-pass
algorithm option to regalloc2, in addition to its existing backtracking
allocator. This produces code much more quickly, at the expense of code
quality. Sometimes this tradeoff is desirable (e.g. when performing a
debug build in a fast-iteration development situation, or in an initial
JIT tier).

This PR adds a Cranelift option to select the RA2 algorithm, plumbs it
through to a Wasmtime option, and adds the option to Wasmtime fuzzing as
well.

An initial compile-time measurement in Wasmtime: `spidermonkey.wasm`
builds in 1.383s with backtracking (existing algorithm), and 1.065s with
single-pass. The resulting binary runs a simple Fibonacci benchmark in
2.060s with backtracking vs. 3.455s with single-pass.

Hence, the single-pass algorithm yields a 23% compile-time reduction, at
the cost of a 67% runtime increase.

* cargo-vet audit for allocator-api2 0.2.18 -> 0.2.20.

show more ...

Gate support for implicit return area pointers behind an option (#9511)

* Remove couple of unused isle helpers

* Gate support for implicit return area pointers behind an option

It implements an in

Gate support for implicit return area pointers behind an option (#9511)

* Remove couple of unused isle helpers

* Gate support for implicit return area pointers behind an option

It implements an incorrect ABI and may be removed in the future due to
complexity reasons. By requiring to enable an option to use it, it
becomes harder to accidentally hit the ABI issue and allows a more
deprecation and eventual removal.

* Fix review comments

* Enable enable_multi_ret_implicit_sret for s390x tests that use i128

* Enable enable_multi_ret_implicit_sret for riscv tests that use vector types

* Enable enable_multi_ret_implicit_sret for more riscv tests

* Enable enable_multi_ret_implicit_sret for windows tests that use i128

show more ...

Update nightly in CI to latest (#9195)

Additionally address some new warnings that are cropping up throughout
the codebase.

Cranelift: add stack_switch CLIF instruction (#9078)

* stack_switch instruction

* Update cranelift/codegen/src/isa/x64/pcc.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

* cargo fmt

* on

Cranelift: add stack_switch CLIF instruction (#9078)

* stack_switch instruction

* Update cranelift/codegen/src/isa/x64/pcc.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

* cargo fmt

* only lower on linux

* give stack_switch the call() side effect

* add function in filetest doing switching only

* Extend documentation of new instruction

* better comments on how we handle the payloads

* Revert "only lower on linux"

This reverts commit 2af10f944186629de1615aa0ed999b7f49d13132.

* Add StackSwitchModel, use to compile stack_switch

* turn stack_switch_model into partial constructor

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

cranelift: Simplify checking whether probestack is needed (#8376)

`Callee::probestack_min_frame` was always set to the guard-page size.
That is the same as the `guard_size` local in `gen_prologue`,

cranelift: Simplify checking whether probestack is needed (#8376)

`Callee::probestack_min_frame` was always set to the guard-page size.
That is the same as the `guard_size` local in `gen_prologue`, which was
also the only place which used `probestack_min_frame`. So don't even
bother storing it, just compute it from the flags as needed.

Also, remove the long-obsolete `probestack_func_adjusts_sp` setting,
which hasn't been used since at least 2021. We may actually want to do
something like what that setting described, but even if we do, it should
be a choice the backend makes rather than a global setting.

show more ...

Use `<[_]>::join` instead of manual implementation (#8076)

Removes an old TODO comment.

Skeleton and initial support for proof-carrying code. (#7165)

* WIP veriwasm 2.0

Co-Authored-By: Chris Fallin <[email protected]>

* PCC: successfully parse some simple facts.

Co-authored-by: Nic

Skeleton and initial support for proof-carrying code. (#7165)

* WIP veriwasm 2.0

Co-Authored-By: Chris Fallin <[email protected]>

* PCC: successfully parse some simple facts.

Co-authored-by: Nick Fitzgerald <[email protected]>

* PCC: plumb facts through VCode and add framework on LowerBackend to check them.

Co-authored-by: Nick Fitzgerald <[email protected]>

* PCC: code is carrying some proofs! Very simple test-case.

Co-authored-by: Nick Fitzgerald <[email protected]>

* PCC: add a `safe` flag for checked memory accesses.

* PCC: add pretty-printing of facts to CLIF output.

* PCC: misc. cleanups.

* PCC: lots of cleanup.

* Post-rebase fixups and some misc. fixes.

* Add serde traits to facts.

* PCC: add succeed and fail tests.

* Review feedback: rename `safe` memflag to `checked`.

* Review feedback.

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

cranelift: Remove the `enable_simd` shared setting (#6631)

This commit removes a setting for Cranelift which I've found a bit
confusing historically and I think is no longer necessary. The setting
i

cranelift: Remove the `enable_simd` shared setting (#6631)

This commit removes a setting for Cranelift which I've found a bit
confusing historically and I think is no longer necessary. The setting
is currently documented as enabling SIMD instructions, but that only
sort of works for the x64 backend and none of the other backends look at
it. Historically this was used to flag to Cranelift that a higher x64
baseline feature set is required for codegen but as of #6625 that's no
longer necessary.

Otherwise it seems more Cranelift-like nowadays to say that vector
instructions generate SIMD instructions where non-vector instructions
probably don't, but may still depending on activated CPU features. In
that sense I'm not sure if a dedicated `enable_simd` setting is still
motivated, so this PR removes it.

This renames some features in the x86 backend such as `use_avx_simd` to
`use_avx` since the `_simd` part is no longer part of the computation
now that `enable_simd` is gone.

show more ...

aarch64: Fix Ldr19 relocations being unresolvable (#6384)

* Add a cranelift setting for padding between basic blocks

Various relocations, jumps, and such require special handling in
`MachBuffer` w

aarch64: Fix Ldr19 relocations being unresolvable (#6384)

* Add a cranelift setting for padding between basic blocks

Various relocations, jumps, and such require special handling in
`MachBuffer` with respect to islands to ensure that everything gets
emitted correctly. This commit adds a setting to synthetically insert
padding at the end of every basic block to help stress this logic with
more minimal test cases. The setting is disabled by default but is
something that we should be able to turn on during fuzzing, for example.

* aarch64: Fix out-of-range `Ldr19` relocations

This commit fixes a bug in the AArch64 backend, and possibly others,
where constants were unconditionally forced to be at the end of the
function when they sometimes couldn't be. For example the `Ldr19`
relocation has a 512k range meaning that if an instruction near the
beginning of a function accesses a constant at the end of a function and
the function is >1M, then the relocation cannot be resolved. This is all
handled internally with `MachBuffer`'s handling of islands but the
problem with constants is that the labels (and the constant values)
weren't defined until the end of the function.

The first attempt at fixing this was to move the calls to
`defer_constant` to the beginning of emission. This would enable the
constants to get deferred as necessary. This was problematic, however,
because it only solved the forwards case (aka your constant was forced
to the end of the function which is too far away). The backwards case,
aka your constant is way too far behind you, was a new problem that
arose.

To fix all of these issues constants are now handled differently inside
of the `MachBuffer`. Previously constants were all pre-assigned a
label-per-constant and all references to the constant would use that
single label. Instead a new heuristic has been added where constants
record their size/alignment at the start of emission and labels are
lazily deferred. When a label for a constant is requested then a label
is lazily allocated or a previously-allocated label for this constant is
returned. When an island is emitted then all emitted constants get
their labels cleared. This intends to balance the previous functionality
of multiple uses of a constant only emit the constant once with fixing
this issue with simplicity as well. This means that constants may get
emitted multiple times, since each reference to a constant after an
island is generated will be guaranteed to generate a new label, even if
it's in-range to access. This can perhaps be fixed in the future with a
more clever API where the `LabelUse` is passed into the function which
converts a constant to a label, but that's left as a refactoring for a
future date.

This commit also moves an `alignment: u32` field into the
`MachBufferFinalized` itself since that's now a function of whatever
constants actually got emitted. Additionally note that constant
emission in the middle of a function doesn't actually emit anything,
instead recording markers of where constants need to go. Then when a
buffer is finalized the constants are passed in to get access to the
data which fills in everything as it's referenced.

* Fuzz the `bb_padding_log2` setting

This commit hooks up the previously-added setting to Cranelift to
Wasmtime's fuzzing infrastructure. This will automatically configure the
setting based on the fuzz input to add a bit of "chaos" to the emitted
code. This should hopefully help expose the issue fixed previously via
fuzzing which otherwise won't generate massive functions.

* Realign back to an instruction boundary

Otherwise misaligned instructions were getting emitted and tripping
various asserts.

* Fix riscv64 testing

* Rename codegen setting to bb_padding_log2_minus_one

Allow for inserting one byte of padding.

* Doc updates

* Thread through shared flags differently

Don't use `EmitInfo`, instead pass in to vcode emission

* Fix s390x tests

* Combine island calculations during vcode emission

Fixes an off-by-just-a-few error if the two island checks are done
separately after a basic block.

show more ...

Cranelift: remove non-egraphs optimization pipeline and `use_egraphs` option. (#6167)

* Cranelift: remove non-egraphs optimization pipeline and `use_egraphs` option.

This PR removes the LICM, GVN,

Cranelift: remove non-egraphs optimization pipeline and `use_egraphs` option. (#6167)

* Cranelift: remove non-egraphs optimization pipeline and `use_egraphs` option.

This PR removes the LICM, GVN, and preopt passes, and associated support
pieces, from `cranelift-codegen`. Not to worry, we still have
optimizations: the egraph framework subsumes all of these, and has been
on by default since #5181.

A few decision points:

- Filetests for the legacy LICM, GVN and simple_preopt were removed too.
As we built optimizations in the egraph framework we wrote new tests
for the equivalent functionality, and many of the old tests were
testing specific behaviors in the old implementations that may not be
relevant anymore. However if folks prefer I could take a different
approach here and try to port over all of the tests.

- The corresponding filetest modes (commands) were deleted too. The
`test alias_analysis` mode remains, but no longer invokes a separate
GVN first (since there is no separate GVN that will not also do alias
analysis) so the tests were tweaked slightly to work with that. The
egrpah testsuite also covers alias analysis.

- The `divconst_magic_numbers` module is removed since it's unused
without `simple_preopt`, though this is the one remaining optimization
we still need to build in the egraphs framework, pending #5908. The
magic numbers will live forever in git history so removing this in the
meantime is not a major issue IMHO.

- The `use_egraphs` setting itself was removed at both the Cranelift and
Wasmtime levels. It has been marked deprecated for a few releases now
(Wasmtime 6.0, 7.0, upcoming 8.0, and corresponding Cranelift
versions) so I think this is probably OK. As an alternative if anyone
feels strongly, we could leave the setting and make it a no-op.

* Update test outputs for remaining test differences.

show more ...

x64: Take SIGFPE signals for divide traps (#6026)

* x64: Take SIGFPE signals for divide traps

Prior to this commit Wasmtime would configure `avoid_div_traps=true`
unconditionally for Cranelift. Thi

x64: Take SIGFPE signals for divide traps (#6026)

* x64: Take SIGFPE signals for divide traps

Prior to this commit Wasmtime would configure `avoid_div_traps=true`
unconditionally for Cranelift. This, for the division-based
instructions, would change emitted code to explicitly trap on trap
conditions instead of letting the `div` x86 instruction trap.

There's no specific reason for Wasmtime, however, to specifically avoid
traps in the `div` instruction. This means that the extra generated
branches on x86 aren't necessary since the `div` and `idiv` instructions
already trap for similar conditions as wasm requires.

This commit instead disables the `avoid_div_traps` setting for
Wasmtime's usage of Cranelift. Subsequently the codegen rules were
updated slightly:

* When `avoid_div_traps=true`, traps are no longer emitted for `div`
instructions.
* The `udiv`/`urem` instructions now list their trap as divide-by-zero
instead of integer overflow.
* The lowering for `sdiv` was updated to still explicitly check for zero
but the integer overflow case is deferred to the instruction itself.
* The lowering of `srem` no longer checks for zero and the listed trap
for the `div` instruction is a divide-by-zero.

This means that the codegen for `udiv` and `urem` no longer have any
branches. The codegen for `sdiv` removes one branch but keeps the
zero-check to differentiate the two kinds of traps. The codegen for
`srem` removes one branch but keeps the -1 check since the semantics of
`srem` mismatch with the semantics of `idiv` with a -1 divisor
(specifically for INT_MIN).

This is unlikely to have really all that much of a speedup but was
something I noticed during #6008 which seemed like it'd be good to clean
up. Plus Wasmtime's signal handling was already set up to catch
`SIGFPE`, it was just never firing.

* Remove the `avoid_div_traps` cranelift setting

With no known users currently removing this should be possible and helps
simplify the x64 backend.

* x64: GC more support for avoid_div_traps

Remove the `validate_sdiv_divisor*` pseudo-instructions and clean up
some of the ISLE rules now that `div` is allowed to itself trap
unconditionally.

* x64: Store div trap code in instruction itself

* Keep divisors in registers, not in memory

Don't accidentally fold multiple traps together

* Handle EXC_ARITHMETIC on macos

* Update emit tests

* Update winch and tests

show more ...

winch: Refactoring wasmtime compiler integration pieces to share more between Cranelift and Winch (#5944)

* Enable the native target by default in winch

Match cranelift-codegen's build script where

winch: Refactoring wasmtime compiler integration pieces to share more between Cranelift and Winch (#5944)

* Enable the native target by default in winch

Match cranelift-codegen's build script where if no architecture is
explicitly enabled then the host architecture is implicitly enabled.

* Refactor Cranelift's ISA builder to share more with Winch

This commit refactors the `Builder` type to have a type parameter
representing the finished ISA with Cranelift and Winch having their own
typedefs for `Builder` to represent their own builders. The intention is
to use this shared functionality to produce more shared code between the
two codegen backends.

* Moving compiler shared components to a separate crate

* Restore native flag inference in compiler building

This fixes an oversight from the previous commits to use
`cranelift-native` to infer flags for the native host when using default
settings with Wasmtime.

* Move `Compiler::page_size_align` into wasmtime-environ

The `cranelift-codegen` crate doesn't need this and winch wants the same
implementation, so shuffle it around so everyone has access to it.

* Fill out `Compiler::{flags, isa_flags}` for Winch

These are easy enough to plumb through with some shared code for
Wasmtime.

* Plumb the `is_branch_protection_enabled` flag for Winch

Just forwarding an isa-specific setting accessor.

* Moving executable creation to shared compiler crate

* Adding builder back in and removing from shared crate

* Refactoring the shared pieces for the `CompilerBuilder`

I decided to move a couple things around from Alex's initial changes.
Instead of having the shared builder do everything, I went back to
having each compiler have a distinct builder implementation. I
refactored most of the flag setting logic into a single shared location,
so we can still reduce the amount of code duplication.

With them being separate, we don't need to maintain things like
`LinkOpts` which Winch doesn't currently use. We also have an avenue to
error when certain flags are sent to Winch if we don't support them. I'm
hoping this will make things more maintainable as we build out Winch.

I'm still unsure about keeping everything shared in a single crate
(`cranelift_shared`). It's starting to feel like this crate is doing too
much, which makes it difficult to name. There does seem to be a need for
two distinct abstraction: creating the final executable and the handling
of shared/ISA flags when building the compiler. I could make them into
two separate crates, but there doesn't seem to be enough there yet to
justify it.

* Documentation updates, and renaming the finish method

* Adding back in a default temporarily to pass tests, and removing some unused imports

* Fixing winch tests with wrong method name

* Removing unused imports from codegen shared crate

* Apply documentation formatting updates

Co-authored-by: Saúl Cabrera <[email protected]>

* Adding back in cranelift_native flag inferring

* Adding new shared crate to publish list

* Adding write feature to pass cargo check

---------

Co-authored-by: Alex Crichton <[email protected]>
Co-authored-by: Saúl Cabrera <[email protected]>

show more ...

Enable egraph-based optimization by default. (#5587)

This PR follows up on #5382 and #5391, which rebuilt the egraph-based optimization framework to be more performant, by enabling it by default.

Enable egraph-based optimization by default. (#5587)

This PR follows up on #5382 and #5391, which rebuilt the egraph-based optimization framework to be more performant, by enabling it by default.

Based on performance results in #5382 (my measurements on SpiderMonkey and bjorn3's independent confirmation with cg_clif), it seems that this is reasonable to enable. Now that we have been fuzzing compiler configurations with egraph opts (#5388) for 6 weeks, having fixed a few fuzzbugs that came up (#5409, #5420, #5438) and subsequently received no further reports from OSS-Fuzz, I believe it is stable enough to rely on.

This PR enables `use_egraphs`, and also normalizes its meaning: previously it forced optimization (it basically meant "turn on the egraph optimization machinery"), now it runs egraph opts if the opt level indicates (it means "use egraphs to optimize if we are going to optimize"). The conditionals in the top-level pass driver are a little subtle, but will get simpler once we can remove the non-egraph path (which we plan to do eventually!).

Fixes #5181.

show more ...

Remove heaps from core Cranelift, push them into `cranelift-wasm` (#5386)

* cranelift-wasm: translate Wasm loads into lower-level CLIF operations

Rather than using `heap_{load,store,addr}`.

*

Remove heaps from core Cranelift, push them into `cranelift-wasm` (#5386)

* cranelift-wasm: translate Wasm loads into lower-level CLIF operations

Rather than using `heap_{load,store,addr}`.

* cranelift: Remove the `heap_{addr,load,store}` instructions

These are now legalized in the `cranelift-wasm` frontend.

* cranelift: Remove the `ir::Heap` entity from CLIF

* Port basic memory operation tests to .wat filetests

* Remove test for verifying CLIF heaps

* Remove `heap_addr` from replace_branching_instructions_and_cfg_predecessors.clif test

* Remove `heap_addr` from readonly.clif test

* Remove `heap_addr` from `table_addr.clif` test

* Remove `heap_addr` from the simd-fvpromote_low.clif test

* Remove `heap_addr` from simd-fvdemote.clif test

* Remove `heap_addr` from the load-op-store.clif test

* Remove the CLIF heap runtest

* Remove `heap_addr` from the global_value.clif test

* Remove `heap_addr` from fpromote.clif runtests

* Remove `heap_addr` from fdemote.clif runtests

* Remove `heap_addr` from memory.clif parser test

* Remove `heap_addr` from reject_load_readonly.clif test

* Remove `heap_addr` from reject_load_notrap.clif test

* Remove `heap_addr` from load_readonly_notrap.clif test

* Remove `static-heap-without-guard-pages.clif` test

Will be subsumed when we port `make-heap-load-store-tests.sh` to generating
`.wat` tests.

* Remove `static-heap-with-guard-pages.clif` test

Will be subsumed when we port `make-heap-load-store-tests.sh` over to `.wat`
tests.

* Remove more heap tests

These will be subsumed by porting `make-heap-load-store-tests.sh` over to `.wat`
tests.

* Remove `heap_addr` from `simple-alias.clif` test

* Remove `heap_addr` from partial-redundancy.clif test

* Remove `heap_addr` from multiple-blocks.clif test

* Remove `heap_addr` from fence.clif test

* Remove `heap_addr` from extends.clif test

* Remove runtests that rely on heaps

Heaps are not a thing in CLIF or the interpreter anymore

* Add generated load/store `.wat` tests

* Enable memory-related wasm features in `.wat` tests

* Remove CLIF heap from fcmp-mem-bug.clif test

* Add a mode for compiling `.wat` all the way to assembly in filetests

* Also generate WAT to assembly tests in `make-load-store-tests.sh`

* cargo fmt

* Reinstate `f{de,pro}mote.clif` tests without the heap bits

* Remove undefined doc link

* Remove outdated SVG and dot file from docs

* Add docs about `None` returns for base address computation helpers

* Factor out `env.heap_access_spectre_mitigation()` to a local

* Expand docs for `FuncEnvironment::heaps` trait method

* Restore f{de,pro}mote+load clif runtests with stack memory

show more ...

Implement inline stack probes for AArch64 (#5353)

* Turn off probestack by default in Cranelift

The probestack feature is not implemented for the aarch64 and s390x
backends and currently the on-

Implement inline stack probes for AArch64 (#5353)

* Turn off probestack by default in Cranelift

The probestack feature is not implemented for the aarch64 and s390x
backends and currently the on-by-default status requires the aarch64 and
s390x implementations to be a stub. Turning off probestack by default
allows the s390x and aarch64 backends to panic with an error message to
avoid providing a false sense of security. When the probestack option is
implemented for all backends, however, it may be reasonable to
re-enable.

* aarch64: Improve codegen for AMode fallback

Currently the final fallback for finalizing an `AMode` will generate
both a constant-loading instruction as well as an `add` instruction to
the base register into the same temporary. This commit improves the
codegen by removing the `add` instruction and folding the final add into
the finalized `AMode`. This changes the `extendop` used but both
registers are 64-bit so shouldn't be affected by the extending
operation.

* aarch64: Implement inline stack probes

This commit implements inline stack probes for the aarch64 backend in
Cranelift. The support here is modeled after the x64 support where
unrolled probes are used up to a particular threshold after which a loop
is generated. The instructions here are similar in spirit to x64 except
that unlike x64 the stack pointer isn't modified during the unrolled
loop to avoid needing to re-adjust it back up at the end of the loop.

* Enable inline probestack for AArch64 and Riscv64

This commit enables inline probestacks for the AArch64 and Riscv64
architectures in the same manner that x86_64 has it enabled now. Some
more testing was additionally added since on Unix platforms we should be
guaranteed that Rust's stack overflow message is now printed too.

* Enable probestack for aarch64 in cranelift-fuzzgen

* Address review comments

* Remove implicit stack overflow traps from x64 backend

This commit removes implicit `StackOverflow` traps inserted by the x64
backend for stack-based operations. This was historically required when
stack overflow was detected with page faults but Wasmtime no longer
requires that since it's not suitable for wasm modules which call host
functions. Additionally no other backend implements this form of
implicit trap-code additions so this is intended to synchronize the
behavior of all the backends.

This fixes a test added prior for aarch64 to properly abort the process
instead of accidentally being caught by Wasmtime.

* Fix a style issue

show more ...

egraph-based midend: draw the rest of the owl (productionized). (#4953)

* egraph-based midend: draw the rest of the owl.

* Rename `egg` submodule of cranelift-codegen to `egraph`.

* Apply some

egraph-based midend: draw the rest of the owl (productionized). (#4953)

* egraph-based midend: draw the rest of the owl.

* Rename `egg` submodule of cranelift-codegen to `egraph`.

* Apply some feedback from @jsharp during code walkthrough.

* Remove recursion from find_best_node by doing a single pass.

Rather than recursively computing the lowest-cost node for a given
eclass and memoizing the answer at each eclass node, we can do a single
forward pass; because every eclass node refers only to earlier nodes,
this is sufficient. The behavior may slightly differ from the earlier
behavior because we cannot short-circuit costs to zero once a node is
elaborated; but in practice this should not matter.

* Make elaboration non-recursive.

Use an explicit stack instead (with `ElabStackEntry` entries,
alongside a result stack).

* Make elaboration traversal of the domtree non-recursive/stack-safe.

* Work analysis logic in Cranelift-side egraph glue into a general analysis framework in cranelift-egraph.

* Apply static recursion limit to rule application.

* Fix aarch64 wrt dynamic-vector support -- broken rebase.

* Topo-sort cranelift-egraph before cranelift-codegen in publish script, like the comment instructs me to!

* Fix multi-result call testcase.

* Include `cranelift-egraph` in `PUBLISHED_CRATES`.

* Fix atomic_rmw: not really a load.

* Remove now-unnecessary PartialOrd/Ord derivations.

* Address some code-review comments.

* Review feedback.

* Review feedback.

* No overlap in mid-end rules, because we are defining a multi-constructor.

* rustfmt

* Review feedback.

* Review feedback.

* Review feedback.

* Review feedback.

* Remove redundant `mut`.

* Add comment noting what rules can do.

* Review feedback.

* Clarify comment wording.

* Update `has_memory_fence_semantics`.

* Apply @jameysharp's improved loop-level computation.

Co-authored-by: Jamey Sharp <[email protected]>

* Fix suggestion commit.

* Fix off-by-one in new loop-nest analysis.

* Review feedback.

* Review feedback.

* Review feedback.

* Use `Default`, not `std::default::Default`, as per @fitzgen

Co-authored-by: Nick Fitzgerald <[email protected]>

* Apply @fitzgen's comment elaboration to a doc-comment.

Co-authored-by: Nick Fitzgerald <[email protected]>

* Add stat for hitting the rewrite-depth limit.

* Some code motion in split prelude to make the diff a little clearer wrt `main`.

* Take @jameysharp's suggested `try_into()` usage for blockparam indices.

Co-authored-by: Jamey Sharp <[email protected]>

* Take @jameysharp's suggestion to avoid double-match on load op.

Co-authored-by: Jamey Sharp <[email protected]>

* Fix suggestion (add import).

* Review feedback.

* Fix stack_load handling.

* Remove redundant can_store case.

* Take @jameysharp's suggested improvement to FuncEGraph::build() logic

Co-authored-by: Jamey Sharp <[email protected]>

* Tweaks to FuncEGraph::build() on top of suggestion.

* Take @jameysharp's suggested clarified condition

Co-authored-by: Jamey Sharp <[email protected]>

* Clean up after suggestion (unused variable).

* Fix loop analysis.

* loop level asserts

* Revert constant-space loop analysis -- edge cases were incorrect, so let's go with the simple thing for now.

* Take @jameysharp's suggestion re: result_tys

Co-authored-by: Jamey Sharp <[email protected]>

* Fix up after suggestion

* Take @jameysharp's suggestion to use fold rather than reduce

Co-authored-by: Jamey Sharp <[email protected]>

* Fixup after suggestion

* Take @jameysharp's suggestion to remove elaborate_eclass_use's return value.

* Clarifying comment in terminator insts.

Co-authored-by: Jamey Sharp <[email protected]>
Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

cranelift: Add inline stack probing for x64 (#4747)

* cranelift: Add inline stack probe for x64

* cranelift: Cleanups comments

Thanks @jameysharp!