wasmtime-cranelift: Use `wasmtime_environ::error` instead of `anyhow` (#12204)

Debug: implement breakpoints and single-stepping. (#12133)

* Debug: implement breakpoints and single-stepping.

This is a PR that puts together a bunch of earlier pieces (patchable
calls in #12061 a

Debug: implement breakpoints and single-stepping. (#12133)

* Debug: implement breakpoints and single-stepping.

This is a PR that puts together a bunch of earlier pieces (patchable
calls in #12061 and #12101, private copies of code in #12051, and all
the prior debug event and instrumentation infrastructure) to implement
breakpoints in the guest debugger.

These are implemented in the way we have planned in #11964: each
sequence point (location prior to a Wasm opcode) is now a patchable call
instruction, patched out (replaced with NOPs) by default. When patched
in, the breakpoint callsite calls a trampoline with the `patchable` ABI
which then invokes the `breakpoint` hostcall. That hostcall emits the
debug event and nothing else.

A few of the interesting bits in this PR include:
- Implementations of "unpublish" (switch permissions back to read/write
from read/execute) for mmap'd code memory on all our platforms.
- Infrastructure in the frame-tables (debug info) metadata producer and
parser to record "breakpoint patches".
- A tweak to the NOP metadata packaged with the `MachBuffer` to allow
multiple NOP sizes. This lets us use one 5-byte NOP on x86-64, for
example (did you know x86-64 had these?!) rather than five 1-byte
NOPs.

This PR also implements single-stepping with a global-per-`Store` flag,
because at this point why not; it's a small additional bit of logic to
do *all* patches in all modules registered in the `Store` when that flag
is enabled.

A few realizations for future work:
- The need for an introspection API available to a debugger to see the
modules within a component is starting to become clear; either that,
or the "module and PC" location identifier for a breakpoint switches
to a "module or component" sum type. Right now, the tests for this
feature use only core modules. Extending to components should not
actually be hard at all, we just need to build the API for it.
- The interaction between inlining and `patchable_call` is interesting:
what happens if we inline a `patchable_call` at a `try_call` callsite?
Right now, we do *not* update the `patchable_call` to a `try_call`,
because there is no `patchable_try_call`; this is fine in the Wasmtime
embedding in practice because we never (today!) throw exceptions from
a breakpoint handler. This does suggest to me that maybe we should
make patchability a property of any callsite, and allow try-calls to
be patchable too (with the same restriction about no return values as
the only restriction); but happy to discuss that one further.

* Add missing debug.wat disas test.

* Review feedback.

* Fix comment on `CodeMemory::text_mut`.

* Review feedback.

* Review feedback: abort process on failure to re-apply executable permissions.

* Implement icache flush for aarch64.

This appears to be necessary as we otherwise see a failure in CI on
macOS/aarch64 that is consistent with patched-in breakpoint calls still
being incorrectly cached after we remove them and republish the code.

There is a longstanding issue in #3310 tracking proper icache coherence
handling on aarch64. We implemented this for Linux with the `membarrier`
syscall but never did so for macOS. Maybe this is the first point at
which it matters, because code was always loaded at new addresses (hence
did not have coherence issues because nothing would have been cached)
previously.

prtest:full

* Review feedback: use `next_multiple_of`.

show more ...

Make Wasmtime's `FuncKey` one-to-one with Cranelift's `ir::UserExternalName` (#11415)

* Make Wasmtime's `FuncKey` one-to-one with Cranelift's `ir::UserExternalName`

`FuncKey`, which used to be call

Make Wasmtime's `FuncKey` one-to-one with Cranelift's `ir::UserExternalName` (#11415)

* Make Wasmtime's `FuncKey` one-to-one with Cranelift's `ir::UserExternalName`

`FuncKey`, which used to be called `CompileKey`, is now one-to-one with
`cranelift_codegen::ir::UserExternalName`, and is used for not just identifying
compilation objects but also relocations and call-graph edges. This allows us to
determine the `StaticModuleIndex` and `DefinedFuncIndex` pair for any
`cranelift_codegen::ir::FuncRef`, regardless of inlining depth, which fixes some
fuzz bugs on OSS-Fuzz.

This continues pushing on the idea that Wasmtime's compilation orchestration and
linking should be relatively agnostic to the kinds of things it is actually
compiling and linking, allowing us to tweak, add, and remove new kinds of
`FuncKey`s more easily. Adding a new `FuncKey` should not require modifying
relocation resolution, for example, just a little bit of code to run the
associated compilation and optionally some code to extract metadata into our
final artifacts for querying at runtime. Everything in between should Just
Continue Working. We still aren't all the way there yet, but this does bring us
a little bit closer.

Finally, in Cranelift's inlining pass, this adds a check that a block is
inserted in the layout before attempting to remove it from the layout, which
would otherwise cause panics. This was triggered by multi-level inlining and
now-unreachable blocks in the inner callees.

I'll note that this does update basically all of the disas tests, or at least
nearly all of them that make function calls. This is because the namespace/index
numbering pair changed slightly to align with `FuncKey`, but that should pretty
much be the only changes.

* remove debug info from panic message, it is only available in some `cfg`s

* fill out module doc comment

* Fix compilation without `component-model` feature

* Fix some more cfg compilations

* cargo fmt

* fix a wrong `&dyn Any` auto coercion; add helpful debug logging and assertions for this kind of thing

show more ...

Remove `cranelift_entity::{Signed, Unsigned}` (#11400)

Use `*::cast_{un,}signed` in the Rust standard library stabilized in
1.87.

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Switch to using Wasmtime-style builtins for ceil, floor, etc. (#10657)

* Switch to using Wasmtime-style builtins for ceil, floor, etc.

With this patch, all emitted calls are Wasmtime-style builtins

Switch to using Wasmtime-style builtins for ceil, floor, etc. (#10657)

* Switch to using Wasmtime-style builtins for ceil, floor, etc.

With this patch, all emitted calls are Wasmtime-style builtins, rather
than Cranelift-style libcalls. This ensures that all calls from
Cranelift-generated code into Wasmtime host code use the same mechanism,
and eliminates the relocation handling code for the libcall mechanism.

* Update tests.

* Avoid using x86_pshufb on non-x86 platforms.

* Revert unneeded f32/f64 changes in Pulley.

* Define i8x16 as an unconstructible type if sse isn't available.

* Delete the setters too.

* Fix f32/f64 setters.

* Test with prtest:full.

prtest:full

* Support fma.

* Return true for `has_native_fma` on pulley.

This works because pulley already has code implementing fma. This avoids
needing to marshal f32x4 values into builtin function calls on pulley.

* Update tests.

show more ...

Don't pad .text to page size for pulley (#10285)

* Don't pad .text to page size for pulley

When targeting pulley we aren't directly emitting executable
code in the .text section and we don't have a

Don't pad .text to page size for pulley (#10285)

* Don't pad .text to page size for pulley

When targeting pulley we aren't directly emitting executable
code in the .text section and we don't have a good idea of the
true target page size so we end up with ELF files that can
have a significant amount of extra padding around the .text
section with no benefit to the consumer.

The change here aligns with the already present section flag
SH_WASMTIME_NOT_EXECUTED. Padding elimination for the .rodata.wasm
section is already handled by the presence/absence of the
memory-init-on-cow configuration.

For a basic wasip1 hello-world rust program with the combination
of this patch and `-O memory-init-cow=n` I saw a reduction in
the compiled wasm ELF from 421KB to 189KB with the sections no
longer showing as being padded out significantly:

```
$ objdump --section-headers target/wasm32-wasip1/release/hello-wasm-world-0x00.cwasm

target/wasm32-wasip1/release/hello-wasm-world-0x00.cwasm: file format elf64-littleriscv

Sections:
Idx Name Size VMA Type
0 00000000 0000000000000000
1 .wasmtime.engine 00000353 0000000000000000 DATA
2 .wasmtime.bti 00000001 0000000000000000 DATA
3 .text 00011bdc 0000000000000000
4 .wasmtime.addrmap 00011c5c 0000000000000000 DATA
5 .wasmtime.traps 00000ae0 0000000000000000 DATA
6 .rodata.wasm 000019e8 0000000000000000 DATA
7 .name.wasm 000027a6 0000000000000000 DATA
8 .wasmtime.info 000010f9 0000000000000000 DATA
9 .symtab 00001788 0000000000000000
10 .strtab 000040f0 0000000000000000
11 .shstrtab 00000089 0000000000000000
```

* code_memory: don't unpublish non-published memory

For the case that we construct a CodeMemory with a custom_code_memory
implementation but we don't end up publishing memory, do not try
to unpublish the memory.

This could happen in a few cases:
- The .text didn't require being made executable
- There's an error or other circumstances cause the CodeMemory
to be dropped prior to publish being called.

Within the context of the .text size optimizations for pulley,
this caused CI failures for a default pulley target as the
errant unpublish was now being called on non-aligned memory.
Prior to the .text optimizations there was still an mprotect
being called to change bits on the .text region which just happened
to be benign.

* In addition to published, check needs_executable before custom unpublishing

Changing the placement of published or introducing new state could
be other ways of addressing the problem of doing a custom
unpublish but those approaches would potentially create other issues
with avoiding duplicate relocations.

Given where this code would be used and the ability for the
`custom_code_memory` to provide their own protections, this
seems very much so Good Enough and gets the tests passing
again while providing more reasonable behavior for user code.

show more ...

Run the full test suite on 32-bit platforms (#9837)

* Run the full test suite on 32-bit platforms

This commit switches to running the full test suite in its entirety
(`./ci/run-tests.sh`) on 32-bit

Run the full test suite on 32-bit platforms (#9837)

* Run the full test suite on 32-bit platforms

This commit switches to running the full test suite in its entirety
(`./ci/run-tests.sh`) on 32-bit platforms in CI in addition to 64-bit
platforms. This notably adds i686 and armv7 as architectures that are
tested in CI.

Lots of little fixes here and there were applied to a number of tests.
Many tests just don't run on 32-bit platforms or a platform without
Cranelift support, and they've been annotated as such where necessary.
Other tests were adjusted to run on all platforms a few minor bug fixes
are here as well.

prtest:full

* Fix clippy warning

* Get wasm code working by default on 32-bit

Don't require the `pulley` feature opt-in on 32-bit platforms to get
wasm code running.

* Fix dead code warning

* Fix build on armv7

* Fix test assertion on armv7

* Review comments

* Update how tests are skipped

* Change how Pulley is defaulted

Default to pulley in `build.rs` rather than in `Cargo.toml` to make it
easier to write down the condition and comment what's happening. This
means that the `pulley-interpreter` crate and pulley support in
Cranelift is always compiled in now and cannot be removed. This should
hopefully be ok though as the `pulley-interpreter` crate is still
conditionally used (meaning it can get GC'd) and the code-size of
Cranelift is not as important as the runtime itself.

* pulley: Save/restore callee-save state on traps

* Fewer clippy warnings about casts

* Use wrapping_add in `g32_addr`, fixing arm test

show more ...

fix typo (#9906)

Signed-off-by: Jack Drogon <[email protected]>

Support executing Pulley in Wasmtime (#9744)

* Support executing Pulley in Wasmtime

This commit is the initial implementation of executing the Pulley
interpreter from the `wasmtime` crate. This gi

Support executing Pulley in Wasmtime (#9744)

* Support executing Pulley in Wasmtime

This commit is the initial implementation of executing the Pulley
interpreter from the `wasmtime` crate. This gives access to all of the
`wasmtime` crate's runtime APIs backed by execution of bytecode in
Pulley. This builds on the previous PRs I've been making for support in
Pulley to culminate in testing on CI in this PR. This PR handles some
final tidbits related to producing a runnable image that can be
interpreted by the `wasmtime` crate such as:

* Pulley compilation artifacts are no longer marked as natively
executable, just read-only.
* Pulley compilation artifacts include wasm-to-array trampolines like
normal platforms (removes a pulley special-case).
* Dispatch of host calls from Pulley to the Wasmtime runtime are
implemented.
* Pulley's list of panicking wasm features is slimmed down as most are
covered by "this lowering isn't supported" errors.
* Execution of wasm code now has an `if` to see whether Pulley is
enabled within a `Store` or not.
* Traps and basic "unwinding" of the pulley stack are implemented (e.g.
a "pulley version" of `setjmp` and `longjmp`, sort of)
* Halting the interpreter has been refactored to help shrink the size of
`ControlFlow<Done>` and handle metadata with each done state.

Some minor refactorings are also included here and there along with a
few fixes here and there necessary to get tests passing.

The next major part of this commit is updates to our `wast` test suite
and executing all `*.wast` files. Pulley is now executed by default for
all files as a new execution engine. This means that all platforms in CI
will start executing Pulley tests. At this time almost all tests are
flagged as "expected to fail" but there are a small handful of
allow-listed tests which are expected to pass. This exact list will
change over time as CLIF lowerings are implemented and the interpreter
is extended.

Follow-up PRs will extend the testing strategy further such as:

* Extending `#[wasmtime_test]` to include Pulley in addition to Winch.
* Getting testing set up on CI for 32-bit platforms.

prtest:full

* Fix pulley fuzz build

* Fix clippy lints

* Shuffle around some `#[cfg]`'d code

* Remove unused imports

* Update feature sets testing MIRI

Enable pulley for wasmtime/wasmtime-cli and also enable all features for
wasmtime-environ

* Round up pulley's page size to 64k

* Skip pulley tests on s390x for now

* Add a safety rail for matching a pulley target to the host

* Fix more pulley tests on s390x

* Review comments

* Fix fuzz build

show more ...

pulley: Clarify documentation around `CallIndirectHost` (#9746)

Handling review comments from #9665 and fully updating documentation to
reflect the mid-pr design shift to the currently-landed state.

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint

Start using `#[expect]` instead of `#[allow]` (#9696)

* Start using `#[expect]` instead of `#[allow]`

In Rust 1.81, our new MSRV, a new feature was added to Rust to use
`#[expect]` to control lint levels. This new lint annotation will
silence a lint but will itself cause a lint if it doesn't actually
silence anything. This is quite useful to ensure that annotations don't
get stale over time.

Another feature is the ability to use a `reason` directive on the
attribute with a string explaining why the attribute is there. This
string is then rendered in compiler messages if a warning or error
happens.

This commit migrates applies a few changes across the workspace:

* Some `#[allow]` are changed to `#[expect]` with a `reason`.
* Some `#[allow]` have a `reason` added if the lint conditionally fires
(mostly related to macros).
* Some `#[allow]` are removed since the lint doesn't actually fire.
* The workspace configures `clippy::allow_attributes_without_reason = 'warn'`
as a "ratchet" to prevent future regressions.
* Many crates are annotated to allow `allow_attributes_without_reason`
during this transitionary period.

The end-state is that all crates should use
`#[expect(..., reason = "...")]` for any lint that unconditionally fires
but is expected. The `#[allow(..., reason = "...")]` lint should be used
for conditionally firing lints, primarily in macro-related code.
The `allow_attributes_without_reason = 'warn'` level is intended to be
permanent but the transitionary
`#[expect(clippy::allow_attributes_without_reason)]` crate annotations
to go away over time.

* Fix adapter build

prtest:full

* Fix one-core build of icache coherence

* Use `allow` for missing_docs

Work around rust-lang/rust#130021 which was fixed in Rust 1.83 and isn't
fixed for our MSRV at this time.

* More MSRV compat

show more ...

pulley: Implement interpreter-to-host calls (#9665)

* pulley: Implement interpreter-to-host calls

This commit is an initial stab at implementing interpreter-to-host
communication in Pulley. The bas

pulley: Implement interpreter-to-host calls (#9665)

* pulley: Implement interpreter-to-host calls

This commit is an initial stab at implementing interpreter-to-host
communication in Pulley. The basic problem is that Pulley needs the
ability to call back into Wasmtime to implement tasks such as
`memory.grow`, imported functions, etc. For native platforms this is a
simple `call_indirect` operation in Cranelift but the story for Pulley
must be different because it's effectively switching from interpreted
code to native code.

The initial idea for this in #9651 is replaced here and looks mostly
similar but with a few changes. The overall structure of how this works
is:

* A new `call_indirect_host` opcode is added to Pulley.
* Function signatures that can be called from Pulley bytecode are
statically enumerated at build-time.
* This enables the implementation of `call_indirect_host` to take an
immediate of which signature is being used and cast the function
pointer to the right type.
* A new pulley-specific relocation is added to Cranelift for this opcode.
* `RelocDistance::Far` calls to a name trigger the use of
`call_indirect_host`.
* The relocation is filled in by Wasmtime after compilation where the
signature number is inserted.
* A new `NS_*` value for user-function namespaces is reserved in
`wasmtime-cranelift` for this new namespace of functions.
* Code generation for Pulley in `wasmtime-cranelift` now has
Pulley-specific handling of the wasm-to-host transition where all
previous `call_indirect` instructions are replaced with a call to a
"backend intrinsic" which gets lowered to a `call_indirect_host`.

Note that most of this still isn't hooked up everywhere in Wasmtime.
That means that the testing here is pretty light at this time. It'll
require a fair bit more work to get everything fully integrated from
Wasmtime in Pulley. This is expected to be one of the significant
remaining chunks of work and should help unblock future testing (or make
those diffs smaller ideally).

* Review comments

show more ...

Enable some Clippy conversion lints for `wasmtime-cranelift` (#9536)

Similar to how `wasmtime::runtime` has a few off-by-default lints turned
on for it do the same for the compilation phase of `wasm

Enable some Clippy conversion lints for `wasmtime-cranelift` (#9536)

Similar to how `wasmtime::runtime` has a few off-by-default lints turned
on for it do the same for the compilation phase of `wasmtime-cranelift`.
This is intended to help weed out lossy `as` casts and instead steer
users to `from` or `try_from` conversions.

show more ...

Implement unwind support for ARM64 Windows (#9266)

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

Fix some typos (#8641)

* occurred

* winch typos

* tests typos

* cli typos

* fuzz typos

* examples typos

* docs typos

* crates/wasmtime typos

* crates/environ typos

* crates/cranelift typos

* crates/test-programs typos

* crates/c-api typos

* crates/cache typos

* crates other typos

* cranelift/codegen/src/isa typos

* cranelift/codegen/src other typos

* cranelift/codegen other typos

* cranelift other typos

* ci js typo

* .github workflows typo

* RELEASES typo

* Fix clang-format documentation line

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

Bad relocation type generated (#8255) (#8264)

* Bad relocation type generated
When disabling sse on x86_64 architecture machines and generating float
libcall, incorrect relocation type R_X86_64_8 ma

Bad relocation type generated (#8255) (#8264)

* Bad relocation type generated
When disabling sse on x86_64 architecture machines and generating float
libcall, incorrect relocation type R_X86_64_8 may be obtained, with the
correct type being R_X86_64_64.

show more ...

Migrate all Winch filetests to `tests/disas` (#8243)

* Switch Winch tests to ATT syntax

* Update all test expectations

* Move all winch tests to `disas` folder

* Add `test = "winch"` to `disas`

Migrate all Winch filetests to `tests/disas` (#8243)

* Switch Winch tests to ATT syntax

* Update all test expectations

* Move all winch tests to `disas` folder

* Add `test = "winch"` to `disas`

* Add `test = "winch"` to all winch test files

* Stub out bits to get AArch64 Winch tests working

* Update expectations for all aarch64 winch tests

* Update flags in Winch tests

Use CLI syntax as that's what `flags` was repurposes as in the new test
suite.

* Update all test expectations for x64 winch

* Omit more offsets by default

* Delete now-dead code

* Update an error message

* Update non-winch test expectations

show more ...

Exit through Cranelift-generated trampolines for builtins (#8152)

* Exit through Cranelift-generated trampolines for builtins

This commit changes how builtin functions in Wasmtime (think
`memory.gr

Exit through Cranelift-generated trampolines for builtins (#8152)

* Exit through Cranelift-generated trampolines for builtins

This commit changes how builtin functions in Wasmtime (think
`memory.grow`) are implemented. These functions are required to exit
through some manner of trampoline to handle runtime requirements for
backtracing right now. Currently this is done via inline assembly for
each architecture (or external assembly for s390x). This is a bit
unfortunate as it's a lot of hand-coding and making sure everything is
right, and it's not easy to update as it's multiple platforms to update.

The change in this commit is to instead use Cranelift-generated
trampolines for this purpose instead. The path for invoking a builtin
function now looks like:

* Wasm code calls a statically known symbol for each builtin.
* The statically known symbol will perform exit trampoline duties (e.g.
pc/fp/etc) and then load a function pointer to the host
implementation.
* The host implementation is invoked and then proceeds as usual.

The main new piece for this PR is that all wasm modules and functions
are compiled in parallel but an output of this compilation phase is what
builtin functions are required. All builtin functions are then unioned
together into one set and then anything required is generated just
afterwards. That means that only one builtin-trampoline per-module is
generated per-builtin.

This work is inspired by #8135 and my own personal desire to have as
much about our ABI details flowing through Cranelift as we can. This in
theory makes it more flexible to deal with future improvements to our
ABI.

prtest:full

* Fix some build issues

* Update winch test expectations

* Update Winch to use new builtin shims.

This commit refactors the Winch compiler to use the new trampolines for
all Wasmtime builtins created in the previous commits. This required a
fair bit of refactoring to handle plumbing through a new kind of
relocation and function call.

Winch's `FuncEnv` now contains a `PrimaryMap` from `UserExternalNameRef`
to `UserExternalName`. This is because there's now more than one kind of
name than just wasm function relocations, so the raw index space of
`UserExternalNameRef` is no longer applicable. This required threading
`FuncEnv` to more locations along with some refactorings to ensure that
lifetimes work out ok.

The `CompiledFunction` no longer stores a trait object of how to map
name refs to names and now directly has a `Primarymap`. This also means
that Winch's return value from its `TargetIsa` is a `CompiledFunction`
as opposed to the previous just-a-`MachBuffer` so it can also package up
all the relocation information. This ends up having `winch-codegen`
depend on `wasmtime-cranelift-shared` as a new dependency.

* Review feedback

show more ...

Remove the wasmtime-cranelift-shared crate (#8222)

The wasmtime-cranelift-shared crate is not as useful as it once was, as
it's no longer possible to build wasmtime with only winch; winch uses
the t

Remove the wasmtime-cranelift-shared crate (#8222)

The wasmtime-cranelift-shared crate is not as useful as it once was, as
it's no longer possible to build wasmtime with only winch; winch uses
the trampolines generated by cranelift now.

show more ...

Implement the relaxed SIMD proposal (#5892)

* Initial support for the Relaxed SIMD proposal

This commit adds initial scaffolding and support for the Relaxed SIMD
proposal for WebAssembly. Codegen s

Implement the relaxed SIMD proposal (#5892)

* Initial support for the Relaxed SIMD proposal

This commit adds initial scaffolding and support for the Relaxed SIMD
proposal for WebAssembly. Codegen support is supported on the x64 and
AArch64 backends on this time.

The purpose of this commit is to get all the boilerplate out of the way
in terms of plumbing through a new feature, adding tests, etc. The tests
are copied from the upstream repository at this time while the
WebAssembly/testsuite repository hasn't been updated.

A summary of changes made in this commit are:

* Lowerings for all relaxed simd opcodes have been added, currently all
exhibiting deterministic behavior. This means that few lowerings are
optimal on the x86 backend, but on the AArch64 backend, for example,
all lowerings should be optimal.

* Support is added to codegen to, eventually, conditionally generate
different code based on input codegen flags. This is intended to
enable codegen to more efficient instructions on x86 by default, for
example, while still allowing embedders to force
architecture-independent semantics and behavior. One good example of
this is the `f32x4.relaxed_fmadd` instruction which when deterministic
forces the `fma` instruction, but otherwise if the backend doesn't
have support for `fma` then intermediate operations are performed
instead.

* Lowerings of `iadd_pairwise` for `i16x8` and `i32x4` were added to the
x86 backend as they're now exercised by the deterministic lowerings of
relaxed simd instructions.

* Sample codegen tests for added for x86 and aarch64 for some relaxed
simd instructions.

* Wasmtime embedder support for the relaxed-simd proposal and forcing
determinism have been added to `Config` and the CLI.

* Support has been added to the `*.wast` runtime execution for the
`(either ...)` matcher used in the relaxed-simd proposal.

* Tests for relaxed-simd are run both with a default `Engine` as well as
a "force deterministic" `Engine` to test both configurations.

* All tests from the upstream repository were copied into Wasmtime.
These tests should be deleted when WebAssembly/testsuite is updated.

* x64: Add x86-specific lowerings for relaxed simd

This commit builds on the prior commit and adds an array of `x86_*`
instructions to Cranelift which have semantics that match their
corresponding x86 equivalents. Translation for relaxed simd is then
additionally updated to conditionally generate different CLIF for
relaxed simd instructions depending on whether the target is x86 or not.
This means that for AArch64 no changes are made but for x86 most relaxed
instructions now lower to some x86-equivalent with slightly different
semantics than the "deterministic" lowering.

* Add libcall support for fma to Wasmtime

This will be required to implement the `f32x4.relaxed_madd` instruction
(and others) when an x86 host doesn't specify the `has_fma` feature.

* Ignore relaxed-simd tests on s390x and riscv64

* Enable relaxed-simd tests on s390x

* Update cranelift/codegen/meta/src/shared/instructions.rs

Co-authored-by: Andrew Brown <[email protected]>

* Add a FIXME from review

* Add notes about deterministic semantics

* Don't default `has_native_fma` to `true`

* Review comments and rebase fixes

---------

Co-authored-by: Andrew Brown <[email protected]>

show more ...

Resolve libcall relocations for older CPUs (#5567)

* Resolve libcall relocations for older CPUs

Long ago Wasmtime used to have logic for resolving relocations
post-compilation for libcalls which

Resolve libcall relocations for older CPUs (#5567)

* Resolve libcall relocations for older CPUs

Long ago Wasmtime used to have logic for resolving relocations
post-compilation for libcalls which I ended up removing during
refactorings last year. As #5563 points out, however, it's possible to
get Wasmtime to panic by disabling SSE features which forces Cranelift
to use libcalls for some floating-point operations instead. Note that
this also requires disabling SIMD because SIMD support has a baseline of
SSE 4.2.

This commit pulls back the old implementations of various libcalls and
reimplements logic necessary to have them work on CPUs without SSE 4.2

Closes #5563

* Fix log message in `wast` support

* Fix offset listed in relocations

Be sure to factor in the offset of the function itself

* Review comments

show more ...

Implement AOT compilation for components (#5160)

* Pull `Module` out of `ModuleTextBuilder`

This commit is the first in what will likely be a number towards
preparing for serializing a compiled

Implement AOT compilation for components (#5160)

* Pull `Module` out of `ModuleTextBuilder`

This commit is the first in what will likely be a number towards
preparing for serializing a compiled component to bytes, a precompiled
artifact. To that end my rough plan is to merge all of the compiled
artifacts for a component into one large object file instead of having
lots of separate object files and lots of separate mmaps to manage. To
that end I plan on eventually using `ModuleTextBuilder` to build one
large text section for all core wasm modules and trampolines, meaning
that `ModuleTextBuilder` is no longer specific to one module. I've
extracted out functionality such as function name calculation as well as
relocation resolving (now a closure passed in) in preparation for this.

For now this just keeps tests passing, and the trajectory for this
should become more clear over the following commits.

* Remove component-specific object emission

This commit removes the `ComponentCompiler::emit_obj` function in favor
of `Compiler::emit_obj`, now renamed `append_code`. This involved
significantly refactoring code emission to take a flat list of functions
into `append_code` and the caller is responsible for weaving together
various "families" of functions and un-weaving them afterwards.

* Consolidate ELF parsing in `CodeMemory`

This commit moves the ELF file parsing and section iteration from
`CompiledModule` into `CodeMemory` so one location keeps track of
section ranges and such. This is in preparation for sharing much of this
code with components which needs all the same sections to get tracked
but won't be using `CompiledModule`. A small side benefit from this is
that the section parsing done in `CodeMemory` and `CompiledModule` is no
longer duplicated.

* Remove separately tracked traps in components

Previously components would generate an "always trapping" function
and the metadata around which pc was allowed to trap was handled
manually for components. With recent refactorings the Wasmtime-standard
trap section in object files is now being generated for components as
well which means that can be reused instead of custom-tracking this
metadata. This commit removes the manual tracking for the `always_trap`
functions and plumbs the necessary bits around to make components look
more like modules.

* Remove a now-unnecessary `Arc` in `Module`

Not expected to have any measurable impact on performance, but
complexity-wise this should make it a bit easier to understand the
internals since there's no longer any need to store this somewhere else
than its owner's location.

* Merge compilation artifacts of components

This commit is a large refactoring of the component compilation process
to produce a single artifact instead of multiple binary artifacts. The
core wasm compilation process is refactored as well to share as much
code as necessary with the component compilation process.

This method of representing a compiled component necessitated a few
medium-sized changes internally within Wasmtime:

* A new data structure was created, `CodeObject`, which represents
metadata about a single compiled artifact. This is then stored as an
`Arc` within a component and a module. For `Module` this is always
uniquely owned and represents a shuffling around of data from one
owner to another. For a `Component`, however, this is shared amongst
all loaded modules and the top-level component.

* The "module registry" which is used for symbolicating backtraces and
for trap information has been updated to account for a single region
of loaded code holding possibly multiple modules. This involved adding
a second-level `BTreeMap` for now. This will likely slow down
instantiation slightly but if it poses an issue in the future this
should be able to be represented with a more clever data structure.

This commit additionally solves a number of longstanding issues with
components such as compiling only one host-to-wasm trampoline per
signature instead of possibly once-per-module. Additionally the
`SignatureCollection` registration now happens once-per-component
instead of once-per-module-within-a-component.

* Fix compile errors from prior commits

* Support AOT-compiling components

This commit adds support for AOT-compiled components in the same manner
as `Module`, specifically adding:

* `Engine::precompile_component`
* `Component::serialize`
* `Component::deserialize`
* `Component::deserialize_file`

Internally the support for components looks quite similar to `Module`.
All the prior commits to this made adding the support here
(unsurprisingly) easy. Components are represented as a single object
file as are modules, and the functions for each module are all piled
into the same object file next to each other (as are areas such as data
sections). Support was also added here to quickly differentiate compiled
components vs compiled modules via the `e_flags` field in the ELF
header.

* Prevent serializing exported modules on components

The current representation of a module within a component means that the
implementation of `Module::serialize` will not work if the module is
exported from a component. The reason for this is that `serialize`
doesn't actually do anything and simply returns the underlying mmap as a
list of bytes. The mmap, however, has `.wasmtime.info` describing
component metadata as opposed to this module's metadata. While rewriting
this section could be implemented it's not so easy to do so and is
otherwise seen as not super important of a feature right now anyway.

* Fix windows build

* Fix an unused function warning

* Update crates/environ/src/compilation.rs

Co-authored-by: Nick Fitzgerald <[email protected]>

Co-authored-by: Nick Fitzgerald <[email protected]>

show more ...

Align functions according to their ISA's requirements (#4826)

Add a function_alignment function to the TargetIsa trait, and use it to align functions when generating objects. Additionally, collect t

Align functions according to their ISA's requirements (#4826)

Add a function_alignment function to the TargetIsa trait, and use it to align functions when generating objects. Additionally, collect the maximum alignment required for pc-relative constants in functions and pass that value out. Use the max of these two values when padding functions for alignment.

This fixes a bug on x86_64 where rip-relative loads to sse registers could cause a segfault, as functions weren't always guaranteed to be aligned to 16-byte addresses.

Fixes #4812

show more ...

Implement `canon lower` of a `canon lift` function in the same component (#4347)

* Implement `canon lower` of a `canon lift` function in the same component

This commit implements the "degenerate"

Implement `canon lower` of a `canon lift` function in the same component (#4347)

* Implement `canon lower` of a `canon lift` function in the same component

This commit implements the "degenerate" logic for implementing a
function within a component that is lifted and then immediately lowered
again. In this situation the lowered function will immediately generate
a trap and doesn't need to implement anything else.

The implementation in this commit is somewhat heavyweight but I think is
probably justified moreso in future additions to the component model
rather than what exactly is here right now. It's not expected that this
"always trap" functionality will really be used all that often since it
would generally mean a buggy component, but the functionality plumbed
through here is hopefully going to be useful for implementing
component-to-component adapter trampolines.

Specifically this commit implements a strategy where the `canon.lower`'d
function is generated by Cranelift and simply has a single trap
instruction when called, doing nothing else. The main complexity comes
from juggling around all the data associated with these functions,
primarily plumbing through the traps into the `ModuleRegistry` to
ensure that the global `is_wasm_trap_pc` function returns `true` and at
runtime when we lookup information about the trap it's all readily
available (e.g. translating the trapping pc to a `TrapCode`).

* Fix non-component build

* Fix some offset calculations

* Only create one "always trap" per signature

Use an internal map to deduplicate during compilation.

show more ...