aarch64: Disable csdb emission by default (#12932)

* aarch64: Disable csdb emission by default

This has a massive performance penalty on macOS, for example, and peer
compilers are not emitting this

aarch64: Disable csdb emission by default (#12932)

* aarch64: Disable csdb emission by default

This has a massive performance penalty on macOS, for example, and peer
compilers are not emitting this as part of on-by-default mitigations.
This commit preserves the option to emit it with an aarch64-specific
`use_csdb` flag, but the default is now `false` meaning that this is not
emitted by default.

Closes #12789

* Fix tests

* Fix tests & review comments

* Use ISLE rule introduced

show more ...

PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built

PCC: remove proof-carrying code (for now?). (#12800)

In late 2023, we built out an experimental feature called
Proof-Carrying Code (PCC), where we attached "facts" to values in the
CLIF IR and built verification of these facts after lowering to
machine instructions. We also added "memory types" describing layout
of memory and a "checked" flag on memory operations such that we could
verify that any checked memory operation accessed valid memory (as
defined by memory types attached to pointer values via
facts). Wasmtime's Cranelift backend then put appropriate memory types
and facts in its IR such that all accesses to memory (aspirationally)
could be checked, taking the whole mid-end and lowering backend of
Cranelift out of the trusted core that enforces SFI.

This basically worked, at the time, for static memories; but never for
dynamic memories, and then work on the feature lost
prioritization (aka I had to work on other things) and I wasn't able
to complete it and put it in fuzzing/enable it as a production option.

Unfortunately since then it has bit-rotted significantly -- as we add
new backend optimizations and instruction lowerings we haven't kept
the PCC framework up to date.

Inspired by the discussion in #12497 I think it's time to delete
it (hopefully just "for now"?) unless/until we can build it again. And
when we do that, we should probably get it to the point of validating
robust operation on all combinations of memory configurations before
merging. (That implies a big experiment branch rather than a bunch of
eager PRs in-tree, but so it goes.) I still believe it is possible to
build this (and I have ideas on how to do it!) but not right now.

show more ...

CI: Add a check for cranelift-codegen (only x86 and aarch64) on no_std tagets (#12812)

* CI: Add a check for cranelift-codegen (only x86 and aarch64) on no_std tagets

* Change -F to --features

* F

CI: Add a check for cranelift-codegen (only x86 and aarch64) on no_std tagets (#12812)

* CI: Add a check for cranelift-codegen (only x86 and aarch64) on no_std tagets

* Change -F to --features

* Fix CI: remove unused imports when building without unwind

* Cargo fmt

show more ...

S390x: emit new instructions added in z17 (#12319)

* s390x: Emit instructions from MIE4 & VXRS_EXT3 on z17

This emits & tests a bunch of instructions:
* from Miscellaneous-Instruction-Extensions

S390x: emit new instructions added in z17 (#12319)

* s390x: Emit instructions from MIE4 & VXRS_EXT3 on z17

This emits & tests a bunch of instructions:
* from Miscellaneous-Instruction-Extensions Facility 4:
* CLZ, 64bit
* CTZ, 64bit
* from Vector-Enhancements Facility 3:
* 32x4, 64x2 & 128x1 variants of the following:
* Divide
* Remainder
* 64x2 & 128x1 multiply variants
* 128x1 vaiants of:
* Compare
* CLZ
* CTZ
* Max
* Min
* Average
* Negation
* Evaluate

Co-authored-by: Jimmy Brisson <[email protected]>

* s390x: Emit vector blend on z17

* Rename x86_blendv to blendv

Now that s390x implements blendv as well, we should refer to
the instruction without the x86 prefix.

---------

Co-authored-by: Ulrich Weigand <[email protected]>

show more ...

cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

Cranelift: add debug tag infrastructure. (#11768)

* Cranelift: add debug tag infrastructure.

This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCo

Cranelift: add debug tag infrastructure. (#11768)

* Cranelift: add debug tag infrastructure.

This PR adds *debug tags*, a kind of metadata that can attach to CLIF
instructions and be lowered to VCode instructions and as metadata on
the produced compiled code. It also adds opaque descriptor blobs
carried with stackslots. Together, these two features allow decorating
IR with first-class debug instrumentation that is properly preserved
by the compiler, including across optimizations and
inlining. (Wasmtime's use of these features will come in followup
PRs.)

The key idea of a "debug tag" is to allow the Cranelift embedder to
express whatever information it needs to, in a format that is opaque
to Cranelift itself, except for the parts that need translation during
lowering. In particular, the `DebugTag::StackSlot` variant gets
translated to a physical offset into the stackframe in the compiled
metadata output. So, for example, the embedder can emit a tag
referring to a stackslot, and another describing an offset in that
stackslot.

The debug tags exist as a *sequence* on any given instruction; the
meaning of the sequence is known only to the embedder, *except* that
during inlining, the tags for the inlining call instruction are
prepended to the tags of inlined instructions. In this way, a
canonical use-case of tags as describing original source-language
frames can preserve the source-language view even when multiple
functions are inlined into one.

The descriptor on a stackslot may look a little odd at first, but its
purpose is to allow serializing some description of
stackslot-contained runtime user-program data, in a way that is firmly
attached to the stackslot. In particular, in the face of inlining,
this descriptor is copied into the inlining (parent) function from the
inlined function when the stackslot entity is copied; no other
metadata outside Cranelift needs to track the identity of stackslots
and know about that motion. This fits nicely with the ability of tags
to refer to stackslots; together, the embedder can annotate
instructions as having certain state in stackslots, and describe the
format of that state per stackslot.

This infrastructure is tested with some compile-tests now;
testing of the interpretation of the metadata output will come with
end-to-end debug instrumentation tests in a followup PR.

* Review feedback: add back sequence points and enforce tags only on sequence points or calls.

* Use Vecs for debug metadata in MachBuffer to avoid SmallVec size penalty in not-used case.

* Review feedback: switch from inlined stackslot descriptor blobs to u64 keys.

show more ...

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conver

Couple of optimizations to the Cranelift incremental cache (#11186)

* Fix a couple of comments

* Remove flags.predicate_view()

It is a remenant of the old backend framework.

* Avoid string conversions for hashing the TargetIsa

* Remove func_body_len

It is identical to buffer.data.len()

* Introduce IsaFlagsHashKey

show more ...

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as wel

Update the `log` dependency (#11197)

* Update the `log` dependency

This enables getting warnings about formatting strings in the `log`
crate directives which are then additionally fixed here as well.

* Update dependency directive in `Cargo.toml`

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Switch to using Wasmtime-style builtins for ceil, floor, etc. (#10657)

* Switch to using Wasmtime-style builtins for ceil, floor, etc.

With this patch, all emitted calls are Wasmtime-style builtins

Switch to using Wasmtime-style builtins for ceil, floor, etc. (#10657)

* Switch to using Wasmtime-style builtins for ceil, floor, etc.

With this patch, all emitted calls are Wasmtime-style builtins, rather
than Cranelift-style libcalls. This ensures that all calls from
Cranelift-generated code into Wasmtime host code use the same mechanism,
and eliminates the relocation handling code for the libcall mechanism.

* Update tests.

* Avoid using x86_pshufb on non-x86 platforms.

* Revert unneeded f32/f64 changes in Pulley.

* Define i8x16 as an unconstructible type if sse isn't available.

* Delete the setters too.

* Fix f32/f64 setters.

* Test with prtest:full.

prtest:full

* Support fma.

* Return true for `has_native_fma` on pulley.

This works because pulley already has code implementing fma. This avoids
needing to marshal f32x4 values into builtin function calls on pulley.

* Update tests.

show more ...

Fix argument extension on riscv64 for Wasmtime builtins (#10069)

* Fix argument extension on riscv64 for Wasmtime builtins

This commit fixes a crash found in the CI of #10040. That PR itself
isn't

Fix argument extension on riscv64 for Wasmtime builtins (#10069)

* Fix argument extension on riscv64 for Wasmtime builtins

This commit fixes a crash found in the CI of #10040. That PR itself
isn't the fault per-se but rather uncovered a preexisting issue on
riscv64. According to riscv64's ABI docs it looks like arguments are all
expected to be sign-extended, whereas currently in Wasmtime all host
signatures are zero-extended on all platforms. This commit applies two
changes to fix this:

* A new `TargetIsa::default_argument_extension` method was added which
is now used instead of unconditionally using `uext` to apply to all
arguments/results.

* While I was here I went ahead and split apart things where the wasm
signature of a builtin doesn't use `uext` or `sext`. The host
signature still does, however, which means that any extension
necessary happens in the trampoline we generate per-libcall as opposed
to at all callsites.

I'm not certain that all platforms require `uext` but I've left the
`TargetIsa` implementation as `uext` for now with a comment explaining
why. Currently the only non-`uext` platforms are riscv64, which is
`sext` to fix the issue from #10040, and Pulley which is "none" as
things work differently there.

* Update test expectations

show more ...

[DWARF] Add logging to range building (#9969)

Update to target-lexicon 0.13.0 (#9752)

* Update to target-lexicon 0.13.0

Pulling in bytecodealliance/target-lexicon#115 to lay the foundation for
some big-endian work in Pulley

* Fix more compile

Update to target-lexicon 0.13.0 (#9752)

* Update to target-lexicon 0.13.0

Pulling in bytecodealliance/target-lexicon#115 to lay the foundation for
some big-endian work in Pulley

* Fix more compile errors

show more ...

Implement unwind support for ARM64 Windows (#9266)

Wasmtime: Implement the custom-page-sizes proposal (#8763)

* Wasmtime: Implement the custom-page-sizes proposal

This commit adds support for the custom-page-sizes proposal to Wasmtime:
https://gith

Wasmtime: Implement the custom-page-sizes proposal (#8763)

* Wasmtime: Implement the custom-page-sizes proposal

This commit adds support for the custom-page-sizes proposal to Wasmtime:
https://github.com/WebAssembly/custom-page-sizes

I've migrated, fixed some bugs within, and extended the `*.wast` tests for this
proposal from the `wasm-tools` repository. I intend to upstream them into the
proposal shortly.

There is a new `wasmtime::Config::wasm_custom_page_sizes_proposal` method to
enable or disable the proposal. It is disabled by default.

Our fuzzing config has been updated to turn this feature on/off as dictated by
the arbitrary input given to us from the fuzzer.

Additionally, there were getting to be so many constructors for
`wasmtime::MemoryType` that I added a builder rather than add yet another
constructor.

In general, we store the `log2(page_size)` rather than the page size
directly. This helps cut down on invalid states and properties we need to
assert.

I've also intentionally written this code such that supporting any power of two
page size (rather than just the exact values `1` and `65536` that are currently
valid) will essentially just involve updating `wasmparser`'s validation and
removing some debug asserts in Wasmtime.

* Update error string expectation

* Remove debug logging

* Use a right shift instead of a division

* fix error message expectation again

* remove page size from VMMemoryDefinition

* fix size of VMMemoryDefinition again

* Only dynamically check for `-1` sentinel for 1-byte page sizes

* Import functions that are used a few times

* Better handle overflows when rounding up to the host page size

Propagate errors instead of returning a value that is not actually a rounded up
version of the input.

Delay rounding up various config sizes until runtime instead of eagerly doing it
at config time (which isn't even guaranteed to work, so we already had to have a
backup plan to round up at runtime, since we might be cross-compiling wasm or
not have the runtime feature enabled).

* Fix some anyhow and nostd errors

* Add missing rounding up to host page size at runtime

* Add validate feature to wasmparser dep

* Add some new rounding in a few places, due to no longer rounding in config methods

* Avoid actually trying to allocate the whole address space in the `massive_64_bit_still_limited` test

The point of the test is to ensure that we hit the limiter, so just cancel the
allocation from the limiter, and otherwise avoid MIRI attempting to allocate a
bunch of memory after we hit the limiter.

* prtest:full

* Revert "Avoid actually trying to allocate the whole address space in the `massive_64_bit_still_limited` test"

This reverts commit ccfa34a78dd3d53e49a6158ca03077d42ce8bcd7.

* miri: don't attempt to allocate more than 4GiB of memory

It seems that rather than returning a null pointer from `std::alloc::alloc`,
miri will sometimes choose to simply crash the whole program.

* remove duplicate prelude import after rebasing

show more ...

Migrate all Winch filetests to `tests/disas` (#8243)

* Switch Winch tests to ATT syntax

* Update all test expectations

* Move all winch tests to `disas` folder

* Add `test = "winch"` to `disas`

Migrate all Winch filetests to `tests/disas` (#8243)

* Switch Winch tests to ATT syntax

* Update all test expectations

* Move all winch tests to `disas` folder

* Add `test = "winch"` to `disas`

* Add `test = "winch"` to all winch test files

* Stub out bits to get AArch64 Winch tests working

* Update expectations for all aarch64 winch tests

* Update flags in Winch tests

Use CLI syntax as that's what `flags` was repurposes as in the new test
suite.

* Update all test expectations for x64 winch

* Omit more offsets by default

* Delete now-dead code

* Update an error message

* Update non-winch test expectations

show more ...

Skeleton and initial support for proof-carrying code. (#7165)

* WIP veriwasm 2.0

Co-Authored-By: Chris Fallin <[email protected]>

* PCC: successfully parse some simple facts.

Co-authored-by: Nic

Skeleton and initial support for proof-carrying code. (#7165)

* WIP veriwasm 2.0

Co-Authored-By: Chris Fallin <[email protected]>

* PCC: successfully parse some simple facts.

Co-authored-by: Nick Fitzgerald <[email protected]>

* PCC: plumb facts through VCode and add framework on LowerBackend to check them.

Co-authored-by: Nick Fitzgerald <[email protected]>

* PCC: code is carrying some proofs! Very simple test-case.

Co-authored-by: Nick Fitzgerald <[email protected]>

* PCC: add a `safe` flag for checked memory accesses.

* PCC: add pretty-printing of facts to CLIF output.

* PCC: misc. cleanups.

* PCC: lots of cleanup.

* Post-rebase fixups and some misc. fixes.

* Add serde traits to facts.

* PCC: add succeed and fail tests.

* Review feedback: rename `safe` memflag to `checked`.

* Review feedback.

---------

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Make MachineEnv a per-ABI property (#6957)

The MachineEnv structure contains the allocatable and preferred
register sets. This is currently fixed per TargetIsa - however,
conceptually these registe

Make MachineEnv a per-ABI property (#6957)

The MachineEnv structure contains the allocatable and preferred
register sets. This is currently fixed per TargetIsa - however,
conceptually these register sets can differ between ABIs on the
same ISA.

To allow for this, replace the TargetIsa machine_env routine with
an ABIMachineSpec get_machine_env routine. To ensure the structure
is still only allocated once, cache it via static OnceLock variables.

No functional change intended.

show more ...

Support referencing stack slots in the DWARF debug info (#6960)

* Add a test

* Disable test

* Add support for specifying stack locations in debug info

Always emit SysV-style CFI unwind info if we

Support referencing stack slots in the DWARF debug info (#6960)

* Add a test

* Disable test

* Add support for specifying stack locations in debug info

Always emit SysV-style CFI unwind info if we need debug info,
and reference it in the debug info using DW_OP_call_frame_cfa.

* Add toolchain comment to the test

* Add a comment and assert

show more ...

x64: Fix ISA requirement for `pmaddubsw` (#6629)

* x64: Fix ISA requirement for `pmaddubsw`

This was erroneously listed as SSE2 but this is actually an SSSE3
instruction. This commit updates the IS

x64: Fix ISA requirement for `pmaddubsw` (#6629)

* x64: Fix ISA requirement for `pmaddubsw`

This was erroneously listed as SSE2 but this is actually an SSSE3
instruction. This commit updates the ISA requirement as well as related
lowerings to have everything work with an SSE2 baseline as well which is
mostly plumbing to make sure the relaxed-simd dot product instructions
use it when it's available but otherwise fall-back to the deterministic
lowering.

* Add require ssse3 feature

* Fix preexisting wat test

show more ...

x64: Add non-SSSE3 lowering for `sqmul_round_sat` (#6607)

Additionally avoid the use of the `x86_pmulhrsw` instruction if SSSE3
isn't enabled.

x64: Add non-SSSE3 lowerings of `pshufb` (#6606)

* x64: Add non-SSSE3 lowerings of `pshufb`

Or, more accurately, add lowerings which don't use `pshufb`'s
functionality at all where possible or othe

x64: Add non-SSSE3 lowerings of `pshufb` (#6606)

* x64: Add non-SSSE3 lowerings of `pshufb`

Or, more accurately, add lowerings which don't use `pshufb`'s
functionality at all where possible or otherwise fall back to a new
libcall. This particular instruction seemed uniquely difficult to
implement in the backend so I decided to "cop out" and use libcall
instead. The libcall will be used for `popcnt`, `shuffle`, and
`swizzle` instructions when SSSE3 isn't available.

* Implemente SSE2 popcnt with Hacker's Delight

* x64: Implement passing vector arguments in the fastcall convention

Windows says that vector arguments are passed indirectly so handle that
here through the `ABIArg::ImplicitPtrArg` variant. Some additional
handling is added to the general machinst backend.

* Update `gen_load_base_offset` for x64

* Fill out remaining bits of fastcall and vector parameters

* Remove now-unnecessary `Clone` bound

show more ...

x64: Add non-SSE4.1 fallback for blend instructions (#6399)

This commit avoids the usage of blend-related instructions when the
SSE4.1 feature is not available. This means that `bitselect` uses the

x64: Add non-SSE4.1 fallback for blend instructions (#6399)

This commit avoids the usage of blend-related instructions when the
SSE4.1 feature is not available. This means that `bitselect` uses the
less optimal lowering and additionally the `x86_blendv` instruction is
disabled for relaxed simd lowerings based on whether SSE4.1 is available
or not (falling back to the deterministic lowering which is supported if
it's not available).

Additionally the `iabs` implementation for i64x2 was updated to have a
non-blend-based lowering, modeled after LLVM.

show more ...

aarch64: Fix Ldr19 relocations being unresolvable (#6384)

* Add a cranelift setting for padding between basic blocks

Various relocations, jumps, and such require special handling in
`MachBuffer` w

aarch64: Fix Ldr19 relocations being unresolvable (#6384)

* Add a cranelift setting for padding between basic blocks

Various relocations, jumps, and such require special handling in
`MachBuffer` with respect to islands to ensure that everything gets
emitted correctly. This commit adds a setting to synthetically insert
padding at the end of every basic block to help stress this logic with
more minimal test cases. The setting is disabled by default but is
something that we should be able to turn on during fuzzing, for example.

* aarch64: Fix out-of-range `Ldr19` relocations

This commit fixes a bug in the AArch64 backend, and possibly others,
where constants were unconditionally forced to be at the end of the
function when they sometimes couldn't be. For example the `Ldr19`
relocation has a 512k range meaning that if an instruction near the
beginning of a function accesses a constant at the end of a function and
the function is >1M, then the relocation cannot be resolved. This is all
handled internally with `MachBuffer`'s handling of islands but the
problem with constants is that the labels (and the constant values)
weren't defined until the end of the function.

The first attempt at fixing this was to move the calls to
`defer_constant` to the beginning of emission. This would enable the
constants to get deferred as necessary. This was problematic, however,
because it only solved the forwards case (aka your constant was forced
to the end of the function which is too far away). The backwards case,
aka your constant is way too far behind you, was a new problem that
arose.

To fix all of these issues constants are now handled differently inside
of the `MachBuffer`. Previously constants were all pre-assigned a
label-per-constant and all references to the constant would use that
single label. Instead a new heuristic has been added where constants
record their size/alignment at the start of emission and labels are
lazily deferred. When a label for a constant is requested then a label
is lazily allocated or a previously-allocated label for this constant is
returned. When an island is emitted then all emitted constants get
their labels cleared. This intends to balance the previous functionality
of multiple uses of a constant only emit the constant once with fixing
this issue with simplicity as well. This means that constants may get
emitted multiple times, since each reference to a constant after an
island is generated will be guaranteed to generate a new label, even if
it's in-range to access. This can perhaps be fixed in the future with a
more clever API where the `LabelUse` is passed into the function which
converts a constant to a label, but that's left as a refactoring for a
future date.

This commit also moves an `alignment: u32` field into the
`MachBufferFinalized` itself since that's now a function of whatever
constants actually got emitted. Additionally note that constant
emission in the middle of a function doesn't actually emit anything,
instead recording markers of where constants need to go. Then when a
buffer is finalized the constants are passed in to get access to the
data which fills in everything as it's referenced.

* Fuzz the `bb_padding_log2` setting

This commit hooks up the previously-added setting to Cranelift to
Wasmtime's fuzzing infrastructure. This will automatically configure the
setting based on the fuzz input to add a bit of "chaos" to the emitted
code. This should hopefully help expose the issue fixed previously via
fuzzing which otherwise won't generate massive functions.

* Realign back to an instruction boundary

Otherwise misaligned instructions were getting emitted and tripping
various asserts.

* Fix riscv64 testing

* Rename codegen setting to bb_padding_log2_minus_one

Allow for inserting one byte of padding.

* Doc updates

* Thread through shared flags differently

Don't use `EmitInfo`, instead pass in to vcode emission

* Fix s390x tests

* Combine island calculations during vcode emission

Fixes an off-by-just-a-few error if the two island checks are done
separately after a basic block.

show more ...

fuzz: randomize block lowering order (#6254)

* fuzz: randomize block lowering order

Co-authored-by: Moritz Waser <[email protected]>
Co-authored-by: Remo Senekowitsch <[email protected]>

* fix block

fuzz: randomize block lowering order (#6254)

* fuzz: randomize block lowering order

Co-authored-by: Moritz Waser <[email protected]>
Co-authored-by: Remo Senekowitsch <[email protected]>

* fix block lowering order randomization

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* simplify control plane internals

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* avoid unnecessary allocations

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* remove unused change_order function

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* add arbitrary 1.3.0 to cargo vet imports lock

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* optimize ControlPlane::shuffle

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* clarify shuffle being a noop without chaos mode

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* reorder only direct successors of a block

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

* rename get_permutation -> shuffled

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

---------

Co-authored-by: Falk Zwimpfer <[email protected]>
Co-authored-by: Moritz Waser <[email protected]>

show more ...