Miscellaneous feature-related build fixes. (#12719)

Most found with cargo-all-features.

Signed-off-by: Piotr Sikora <[email protected]>

Cranelift: add patchable call instructions. (#12101)

* Cranelift: add patchable call instructions.

The new `patchable_call` CLIF instruction pairs with the `patchable`
ABI, and emits a callsite wit

Cranelift: add patchable call instructions. (#12101)

* Cranelift: add patchable call instructions.

The new `patchable_call` CLIF instruction pairs with the `patchable`
ABI, and emits a callsite with one new key property: the MachBuffer
carries metadata that describes exactly which byte range to "NOP out"
(overwrite with NOP instructions) to disable that callsite. Doing so is
semantically valid and explicitly supported.

This enables patching of code at runtime to dynamically turn on and off
features such as instrumentation or debugging hooks. We plan to use this
to implement breakpoints in Wasmtime's guest debugging support.

As part of this change, I added a notion of "unit of NOP bytes" to the
MachBuffer so that the consumer (e.g., Wasmtime's Cranelift-based code
compilation pipeline and metadata-producing logic) can handle patchable
callsites without any other special knowledge of the ISA.

For the "real metal" ISAs there are perfectly well-defined NOPs to use,
but for Pulley, where all opcodes are assigned at compile time by macro
magic, I explicitly defined NOP as opcode byte 0 by moving `Nop`'s
definition to the top of the list and adding a unit test asserting its
encoding.

A design note: in principle it would be possible, as an alternative, to
treat "patchability" as an orthogonal dimension of all callsites, and
emit the metadata describing the instruction-offset range for any
callsite with the flag set. The only truly necessary semantic
restriction is that there are no return values (because if we turn the
callsite off, nothing writes to them); we could support patchability for
other ABIs and for the other kinds of call instructions. The `patchable`
ABI would then be better described as something like the "no clobbers
ABI". I opted not to generalize in this way because it creates some
less-tested corners and the generalized form, at least at the MachInst
level, is not really much simpler in the end.

A testing note: I opted not to implement actual code patching in the
`cranelift-tools` filetest runner and test patching callsites in/out via
some actuation (e.g. a magic hostcall, like we do for throws) because
(i) that's a lot of new plumbing and (ii) we are going to test this very
shortly in Wasmtime anyway and (iii) the correctness (or not) of the
location-and-length metadata is easy enough to verify in the
disassemblies in the compile-tests.

* Review feedback: remove dependence on (and test for) NOP being the literal byte 0.

show more ...

Update nightly rust used in CI (#11856)

* Update nightly rust used in CI

A few more warnings are cropping up so squash them.

* Frob the `unreachable_code` lint

Looks like nightly rust has gotten

Update nightly rust used in CI (#11856)

* Update nightly rust used in CI

A few more warnings are cropping up so squash them.

* Frob the `unreachable_code` lint

Looks like nightly rust has gotten much more aggressive about linting on
unreachable code.

prtest:full

* Allow some more warnings...

show more ...

Update nightly Rust used in CI (#11755)

* Update nightly Rust used in CI

Keeping it up-to-date

prtest:full

* Fix unused warnings on nightly

* Rename rustdoc feature

* Adjust some removals

pulley: Implement support for `symbol_value` (#11576)

This commit fills out the Pulley lowerings of the `symbol_value` and
`func_addr` CLIF instructions. Additionally handling of relocations on
Pull

pulley: Implement support for `symbol_value` (#11576)

This commit fills out the Pulley lowerings of the `symbol_value` and
`func_addr` CLIF instructions. Additionally handling of relocations on
Pulley is improved to be more "formal" as opposed to just blindly using
an x64 relocation and assuming it works out. The intention here is to
make Pulley behave more similarly to other platforms in all these
respects while also enabling usage of Wasmtime to eventually use
`symbol_addr` to calculate an address relative to the current PC
(similar to x64).

This includes Cranelift golden tests as well as more filetests being
run, but the actual integration into Wasmtime will be deferred to a
future commit.

show more ...

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stor

pulley: Reimplement wasm loads/stores & memory opcodes (#10154)

* pulley: Reimplement wasm loads/stores & memory opcodes

This commit is a large refactoring to reimplement how WebAssembly
loads/stores are translated to Pulley opcodes when using the
interpreter. Additionally the functionality related to memory support
has changed quite a bit with the interpreter as well. This is all based
off comments on #10102 with the end goal of folding the two Pulley
opcodes today of "do the bounds check" and "do the load" into one
opcode. This is intended to reduce the number of opcodes and overall
improve interpreter throughput by minimizing turns of the interpreter
loop.

The basic idea behind this PR is that a new basic suite of loads/stores
are added to Pulley which trap if the address is zero. This provides a
route to translate trapping loads/stores in CLIF to Pulley bytecode
without actually causing segfaults at runtime. WebAssembly translation
to CLIF is then updated to use the `select` trick for wasm loads/stores
where either 0 is loaded from or the actual address is loaded from.
Basic support for translation and such is added for this everywhere, and
this ensures that all loads/stores for wasm will be translated
successfully with Pulley.

The next step was to extend the "g32" addressing mode preexisting in
Pulley to support a bounds check as well. New pattern-matches were added
to ISLE to search for a bounds check in the address of a trapping
load/store. If found then the entire chain of operations necessary to
compute the address are folded into a single "g32" opcode which ends up
being a fallible load/store at runtime.

To fit all this into Pulley this commit contains a number of
refactorings to shuffle around existing opcodes related to memory and
extend various pieces of functionality here and there:

* Pulley now uses a `AddrFoo` types to represent addressing modes as a
single immediate rather than splitting it up into pieces for each
method. For example `AddrO32` represents "base + offset32". `AddrZ`
represents the same thing but traps if the address is zero. The
`AddrG32` mode represents a bounds-checked 32-bit linear memory access
on behalf of wasm.

* Pulley loads/stores were reduced to always using an `AddrFoo`
immediate. This means that the old `offset8` addressing mode was
removed without replacement here (to be added in the future if
necessary). Additionally the suite of sign-extension modes supported
were trimmed down to remove 8-to-64, 16-to-64, and 32-to-64 extensions
folded as part of the opcode. These can of course always be re-added
later but probably want to be added just for the `G32` addressing mode
as opposed to all addressing modes.

* The interpreter itself was refactored to have an `AddressingMode`
trait to ensure that all memory accesses, regardless of addressing
modes, are largely just copy/pastes of each other. In the future it
might make sense to implement these methods with a macro, but for now
it's copy/paste.

* In ISLE the `XLoad` generic instruction removed its `ext` field to
have extensions handled exclusively in ISLE instead of partly in
`emit.rs`.

* Float/vector loads/stores now have "g32" addressing (in addition to
the "z" that's required for wasm) since it was easy to add them.

* Translation of 1-byte accesses on Pulley from WebAssembly to CLIF no
longer has a special case for using `a >= b` instead of `a > b - 1` to
ensure that the same bounds-check instruction can be used for all
sizes of loads/stores.

* The bounds-check which folded a load-of-the-bound into the opcode is
now present as a "g32bne" addressing mode. with its of suite of
instructions to boo.

Overall this PR is not a 1:1 replacement of all previous opcodes with
exactly one opcode. For example loading 8 bits sign-extended to 64-bits
is now two opcodes instead of one. Additionally some previous opcodes
have expanded in size where for example the 8-bit offset mode was remove
in favor of only having 32-bit offsets. The goal of this PR is to reboot
how memory is handled in Pulley. All loads/stores now use a specific
addressing mode and currently all operations supported across addressing
modes are consistently supported. In the future it's expected that some
features will be added to some addressing modes and not others as
necessary, for example extending the "g32" addressing mode only instead
of all addressing modes.

For an evaluation of this PR:

* Code size: `spidermonkey.cwasm` file is reduced from 19M to 16M.
* Sightglass: `pulldown-cmark` is improved by 15%
* Sightglass: `bz2` is improved by 20%
* Sightglass: `spidermonkey` is improved by 22%
* Coremark: score improved by 40%

Overall this PR and new design looks to be a large win. This is all
driven by the reduction in opcodes both for compiled code size and
execution speed by minimizing turns of the interpreter loop. In the end
I'm also pretty happy with how this turned out and I think the
refactorings are well worth it.

* Use new `is_pulley` helper more

* Improve `addrz` helper, tighten up `memory-inbounds.wat` a bit

* Improve codegen in a few `memory-inbounds.wat` cases

* Fix test expectation

show more ...

Require lint reasons in `pulley-interpreter` (#10173)

* Require lint reasons in `pulley-interpreter`

Continuing work originally started in #9696

* Add more pulley #[cfg]

pulley: Optimize bounds-checks for 1-byte loads/stores (#10100)

These have a different pattern than N-byte loads/stores where the
condition being tested is `a >= b` which doesn't match the pattern f

pulley: Optimize bounds-checks for 1-byte loads/stores (#10100)

These have a different pattern than N-byte loads/stores where the
condition being tested is `a >= b` which doesn't match the pattern for
N-byte loads/stores with `a > b - N`. This commit adds dedicated opcodes
to Pulley for this pattern to help optimize single-byte loads/stores.

show more ...

pulley: Slightly optimize bounds checks (#10080)

* pulley: Slightly optimize bounds checks

In profiling a module I was noticing that the previous
`xbc32_bound_trap` instruction wasn't being used wh

pulley: Slightly optimize bounds checks (#10080)

* pulley: Slightly optimize bounds checks

In profiling a module I was noticing that the previous
`xbc32_bound_trap` instruction wasn't being used when I expected.
Investigation revealed that the load of the bound itself was GVN'd and
deduplicated (yay!) but it meant that the load was used in two locations
meaning it didn't pass checks for `sinkable_load`. This commit fixes
this by repurposing `xbc32_bound_trap` for "the bound is in a register"
and renaming the previous instruction to `xbc32_boundne_trap`. This
helps cut down on the number of opcodes in this benchmark and improves
performance slightly.

At the same time this tightens up "sinkable loads" to require native
endianness since that's what the bound of memory is stored as.
Additionally in addition to testing for `a < b` and optimizing that this
also now optimizes `b > a`, the same condition just having the arguments
swapped.

* Fix some copy/paste typos

show more ...

pulley: Add a multiply-and-add macro instruction (#10081)

This is present in riscv64 and aarch64 native ISAs and was found in a
benchmark I was looking at so let's add a macro-op as well to help cas

pulley: Add a multiply-and-add macro instruction (#10081)

This is present in riscv64 and aarch64 native ISAs and was found in a
benchmark I was looking at so let's add a macro-op as well to help cases
where this crops up in the wild.

show more ...

Add basic support for profiling Pulley (#10034)

* Add basic support for profiling Pulley

This commit adds basic support for profiling the Pulley interpreter.
This is partially achievable previously

Add basic support for profiling Pulley (#10034)

* Add basic support for profiling Pulley

This commit adds basic support for profiling the Pulley interpreter.
This is partially achievable previously through the use of native
profilers, but the downside of that approach is that you can find hot
instructions but it's not clear in what context the hot instructions are
being executed nor what functions are hot. The goal of this profiler is
to show pulley bytecode and time spent in bytecode itself to better
understand the shape of code around a hot instruction to identify new
macro opcodes for example.

The general structure of this new profiler is:

* There is a compile-time feature for Pulley which is off-by-default
where, when enabled, Pulley will record its current program counter
into an `AtomicUsize` before each instruction.

* When the CLI has `--profile pulley` Wasmtime will spawn a sampling
thread in the same process which will periodically read from this
`AtomicUsize` to record where the program is currently executing.

* The Pulley profiler additionally records all bytecode through the use
of the `ProfilingAgent` trait to ensure that the recording has access
to all bytecode as well.

* Samples are taken throughout the process and emitted to a
`pulley-$pid.data` file. This file is then interpreted and printed by
an "example" program `profiler-html.rs` in the `pulley/examples`
directory.

The end result is that hot functions of Pulley bytecode can be seen and
instructions are annotated with how frequently they were executed. This
enables finding hot loops and understanding more about the whole loop,
bytecodes that were selected, and such.

* Add missing source file

* Check the profile-pulley feature in CI

* Miscellaneous fixes for CI

* Fix type-checking of `become` on nightly Rust

* Fix more misc CI issues

* Fix dispatch in tail loop

* Update test expectations

* Review comments

* Fix a feature combo

show more ...

pulley: Shrink frame save/restore instructions (#9999)

* pulley: Shrink frame save/restore instructions

This commit shrinks the size of the `PushFrameSave` and
`PopFrameRestore` functions which are

pulley: Shrink frame save/restore instructions (#9999)

* pulley: Shrink frame save/restore instructions

This commit shrinks the size of the `PushFrameSave` and
`PopFrameRestore` functions which are used in almost all wasm functions.
Previously these instructions allowed for 32-bits of stack space in
addition to saving/restoring all 32 X-registers. In reality though it's
quite uncommon to need more than 16-bits of stack space and ABI-wise the
most commonly saved registers are the upper 16 registers of the X
register set.

This commit therefore shrinks the frame size to 16 bits and only has the
ability to save/restore the upper 16 X-registers. Note that any
clobbered registers and frame sizes are still supported, they'll just
use more pessimal encodings which aren't a single opcode. If a function
uses >64KiB of stack space though it's probably not too important what
the dispatch cost is at the beginning.

The overall result of this change is that each instruction shaves of 4
bytes (2 from the frame size and 2 from the registers being
saved/restored). This results in a 4% faster execution time on the bz2
Sightglass benchmark, ~1% on pulldown-cmark, and while it shrinks
`spidermonkey.cwasm` slightly it's not significant.

* Remove no-longer-applicable test

* Fix clippy error

* Update test expectations

show more ...

pulley: Add more addressing modes for loads/stores (#9994)

* pulley: Add more addressing modes for loads/stores

This commit adds a new "g32" addressing mode to Pulley that matches the
pattern emitt

pulley: Add more addressing modes for loads/stores (#9994)

* pulley: Add more addressing modes for loads/stores

This commit adds a new "g32" addressing mode to Pulley that matches the
pattern emitted by Cranelift for 32-bit wasm guests running on hosts.
The general idea here is that this addressing mode encompasses an
addition of a host-width value to a zero-extended (optionally) 32-bit
value. On 32-bit hosts there's no zero-extension but on 64-bit hosts
there's a zero-extension. The wasm address is always 32-bits though
which enables using a single instruction for both 32 and 64-bit hosts.

New "g32" loads and stores are added to Pulley with varying sizes and
options according to what seems to be common in wasm. The `disas` test
suite was updated to showcase using these instructions for wasm
loads/stores on 32 and 64-bit hosts.

An additional change in this commit is to deduplicate the 32/64-bit
bounds-check macro-ops. The trick in this commit works for those as well
meaning that only a single instruction is needed instead of
one-per-host-pointer-width. Additionally the load of the bound from the
`VMContext` is folded into the bounds check itself as it was found that
this was always present anyway before the bounds check.

Overall this shrinks the size of `spidermonkey.cwasm` from 21M to 20M
and the runtime of `pulldown-cmark`, `bz2`, and `spidermonkey` on
Sightglass have all been reduced by 10%. Not as big wins as I was hoping
for but alas.

* Fix debug assertions on 32-bit

show more ...

Fix comment about `xbc32_bound32_trap` pulley macro-op (#9998)

Unlike the `bound64` version, this instruction does not zero-extend the address.

pulley: Finish `simd` proposal implementation (#9935)

* pulley: Finish `simd` proposal implementation

This commit fills out the final and miscellaneous set of opcodes for
Pulley to have a complete

pulley: Finish `simd` proposal implementation (#9935)

* pulley: Finish `simd` proposal implementation

This commit fills out the final and miscellaneous set of opcodes for
Pulley to have a complete implementation of the `simd` proposal for
WebAssembly. All spec tests are now enabled and the Pulley-specific
exceptions for `*.wast` tests are all gone.

Closes #9783

* Remove stray build script

show more ...

pulley: Add special instructions for `dst = 0` and 1 (#9986)

* pulley: Add special instructions for `dst = 0` and 1

This commit adds a special instruction for setting a register to the
value 0 or t

pulley: Add special instructions for `dst = 0` and 1 (#9986)

* pulley: Add special instructions for `dst = 0` and 1

This commit adds a special instruction for setting a register to the
value 0 or the value 1. This extends to the full width of the register
and accounts for the majority of all `xconst` instructions found in
`spidermonkey.cwasm`. It's not a major size decrease, but helps a bit.

* Fix CLIF test expectations

* Fix more test expectations

show more ...

pulley: Implement the wide-arithmetic proposal (#9944)

* pulley: Implement the wide-arithmetic proposal

Add a few minor instructions/lowerings for the new operations added as
part of the wide-arith

pulley: Implement the wide-arithmetic proposal (#9944)

* pulley: Implement the wide-arithmetic proposal

Add a few minor instructions/lowerings for the new operations added as
part of the wide-arithmetic proposal. These are all part of the
"extended" opcode set since they shouldn't be common and if they're
performance critical you probably want a native backend instead.

* Review comments

show more ...

pulley: Add some macro-instructions related to bounds-checks (#9943)

* pulley: Add some macro-instructions related to bounds-checks

This commit starts down the path of optimizing wasm loads/stores

pulley: Add some macro-instructions related to bounds-checks (#9943)

* pulley: Add some macro-instructions related to bounds-checks

This commit starts down the path of optimizing wasm loads/stores in
Pulley with macro-instructions. It's expected that these instructions
are so common that it's worth putting them in the 1-byte namespace of
opcodes.

Locally this gets a 10% speedup on the sightglass bz2 benchmark.

* Remove incorrect lowering rule

show more ...

pulley: Fill out most remaining simd float ops (#9884)

* pulley: Fill out most remaining simd float ops

Get most simd/float-related tests passing. Mostly reusing preexisting
scalar ops for the simd

pulley: Fill out most remaining simd float ops (#9884)

* pulley: Fill out most remaining simd float ops

Get most simd/float-related tests passing. Mostly reusing preexisting
scalar ops for the simd implementation.

* Fix fma test on MinGW

prtest:full

* More MinGW fixes

show more ...

Make Pulley pass `simd_f32x4_arith.wast` (#9897)

* Add f32x4 arithmetic instructions to Pulley:

Adds float SIMD instructions on 4 lanes (f32x4) for subtraction, multiplication, and negation.
`vtrun

Make Pulley pass `simd_f32x4_arith.wast` (#9897)

* Add f32x4 arithmetic instructions to Pulley:

Adds float SIMD instructions on 4 lanes (f32x4) for subtraction, multiplication, and negation.
`vtrunc32x4` and `vmuli32x4` were used as basis on how to organize things.

* Mark `simd_f32x4_arith.wast` as passing for Pulley:

To be exact: `spec_testsuite/simd_f32x4_arith.wast` has been removed from the should fail list for Pulley.

* Rename 2 f32x4 arithmatic instructions to contain "f32x4":

Specifically:
- "vsub32x4" -> "vsubf32x4"
- "vmul32x4" -> "vmulf32x4"

show more ...

feat: Implement simd_f64x2_arith for pulley (#9917)

pulley: Implement iadd_pairwise (#9912)

* pulley: Implement iadd_pairwise

* access by index

Co-authored-by: Xuanwo <[email protected]>

---------

Co-authored-by: Xuanwo <[email protected]>

pulley: implement vector shuffle (#9910)

* impl shuffle

* raise inst size limit

pulley: fill out more int vector ops (#9908)

* get simd_i8x16_arith2 passing

* get simd_i64x2_arith2 passing

* get simd_i8x16_sat_arith passing

* get simd_i16x8_sat_arith passing

pulley: Implement vector sqmul_round_sat (#9911)

* pulley: Implement vector sqmul_round_sat

* parenthesize to bring out op. order