cranelift-codegen: rename most uses of std to core and alloc (#12237)

* rename most std uses to core and alloc

* cargo fmt

Cranelift: implement an "unwinder" crate and exception throws in filetests. (#10919)

This commit introduces the next major piece of machinery (after the
previously-landed `try_call` support) that we

Cranelift: implement an "unwinder" crate and exception throws in filetests. (#10919)

This commit introduces the next major piece of machinery (after the
previously-landed `try_call` support) that we will eventually use to
implement Wasm exceptions in Wasmtime. In particular, it implements a
generic unwinder as a new crate that supports (i) walking a stack
produced by Cranelift code, (ii) serializing Cranelift exception
metadata to compact tables (in a way very similar to address maps in
Wasmtime, so they will be mappable directly from disk), (iii) using
these serialized tables to find handlers during a stack-walk, and (iv)
jumping to handlers (i.e., actually unwinding). This crate is currently
used in the filetests runner, and will next be used in Wasmtime.

The commit first performs code-motion: it moves stack-walking code from
Wasmtime to `cranelift-unwinder`. This itself has no functional effect,
but isolates the code that understands contiguous sequences of Cranelift
frames ("activations") from that which is specific to Wasmtime's
activation delimiters and metadata.

It then implements a compact exception-table format. This format uses
the `object` crate's mechanisms for directly referencing in-memory
arrays of little-endian `u32`s in a way that will allow us to find
handlers when mapping exception metadata directly from an ELF section in
a `.cwasm` (for example). The format consists of four sorted `u32`
arrays in a way that allows us to look up a callsite first, then search
its sorted array of handler offsets by tags.

It next implements the actual unwind control flow: it contains an
assembly stub for each supported architecture that transfers control to
a PC, SP, and FP value "up the stack", with payload values placed in the
payload registers we have defined per our exception ABI in Cranelift.

Finally, it puts these pieces together in the filetest runner. Note that
the runtest does a lot "by hand": we don't have entry and exit
trampolines as we do in Wasmtime, so the filetest contains three
functions, with the middle one invoking the "throw hostcall" and entry
and exit trampolines around it grabbing the appropriate entry/exit FPs
and exit PC. The dance to call back to host code is also somewhat
delicate, as we haven't done this before. The `JITModule`'s linking +
relocation support does not seem sufficient to properly define a symbol,
so instead we scan for `func_addr` instructions referencing a well-known
name (`__cranelift_throw`) and replace them with `iconst`s with the
function address at runtime, baking it in. This is somewhat ugly, but it
works. All of these filetest-specific details will be handled much more
nicely in the Wasmtime version of this functionality, as we have proper
abstractions for entry/exit trampolines and hostcalls.

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

Enforce `uninlined_format_args` for the workspace (#9065)

* Enforce `uninlined_format_args` for the workspace

* fix: failing `Monolith Checks` job

* fix: formatting

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parall

Decouple `serde` from its `derive` crate (#6917)

By not activating the `derive` feature on `serde`, the compilation speed
can be improved by a lot. This is because `serde` can then compile in
parallel to `serde_derive`, allowing it to finish compilation possibly
even before `serde_derive`, unblocking all the crates waiting for
`serde` to start compiling much sooner.

As it turns out the main deciding factor for how long the compile time of a
project is, is primarly determined by the depth of dependencies rather
than the width. In other words, a crate's compile times aren't affected
by how many crates it depends on, but rather by the longest chain of
dependencies that it needs to wait on. In many cases `serde` is part of
that long chain, as it is part of a long chain if the `derive` feature
is active:

`proc-macro2` compile build script > `proc-macro2` run build script >
`proc-macro2` > `quote` > `syn` > `serde_derive` > `serde` >
`serde_json` (or any crate that depends on serde)

By decoupling it from `serde_derive`, the chain is shortened and compile
times get much better.

Check this issue for a deeper elaboration:
https://github.com/serde-rs/serde/issues/2584

For `wasmtime` I'm seeing a reduction from 24.75s to 22.45s when
compiling in `release` mode. This is because wasmtime through `gimli`
has a dependency on `indexmap` which can only start compiling when
`serde` is finished, which you want to happen as early as possible so
some of wasmtime's dependencies can start compiling.

To measure the full effect, the dependencies can't by themselves
activate the `derive` feature. I've upstreamed a patch for
`fxprof-processed-profile` which was the only dependency that activated
it for `wasmtime` (not yet published to crates.io). `wasmtime-cli` and
co. may need patches for their dependencies to see a similar
improvement.

show more ...

fuzzgen: Generate multiple functions per testcase (#5765)

* fuzzgen: Generate multiple functions per testcase

* fuzzgen: Fix typo

Co-authored-by: Jamey Sharp <[email protected]>

---------

Co-aut

fuzzgen: Generate multiple functions per testcase (#5765)

* fuzzgen: Generate multiple functions per testcase

* fuzzgen: Fix typo

Co-authored-by: Jamey Sharp <[email protected]>

---------

Co-authored-by: Jamey Sharp <[email protected]>

show more ...

cranelift: Allow `call` and `call_indirect` in runtests (#4667)

* cranelift: Change test runner order

Changes the ordering of runtests to run per target and then per function.

This change does

cranelift: Allow `call` and `call_indirect` in runtests (#4667)

* cranelift: Change test runner order

Changes the ordering of runtests to run per target and then per function.

This change doesn't do a lot by itself, but helps future refactorings of runtests.

* cranelift: Rename SingleFunctionCompiler to TestCaseCompiler

* cranelift: Skip runtests per target instead of per run

* cranelift: Deduplicate test names

With the upcoming changes to the runtest infrastructure we require unique ExtNames for all tests.

Note that for test names we have a 16 character limit on test names, and must be unique within those 16 characters.

* cranelift: Add TestFileCompiler to runtests

TestFileCompiler allows us to compile the entire file once, and then call the trampolines for each test.

The previous code was compiling the function for each invocation of a test.

* cranelift: Deduplicate ExtName for avg_round tests

* cranelift: Rename functions as they are defined.

The JIT internally only deals with User functions, and cannot link test name funcs.

This also caches trampolines by signature.

* cranelift: Preserve original name when reporting errors.

* cranelift: Rename aarch64 test functions

* cranelift: Add `call` and `call_indirect` tests!

* cranelift: Add pauth runtests for aarch64

* cranelift: Rename duplicate s390x tests

* cranelift: Delete `i128_bricmp_of` function from i128-bricmp

It looks like we forgot to delete it when it was moved to
`i128-bricmp-overflow`, and since it didn't have a run invocation
it was never compiled.

However, s390x does not support this, and panics when lowering.

* cranelift: Add `colocated` call tests

* cranelift: Rename *more* `s390x` tests

* cranelift: Add pauth + sign_return_address call tests

* cranelift: Undeduplicate test names

With the latest main changes we now support *unlimited* length test names.

This commit reverts:
52274676ff631c630f9879dd32e756566d3e700f
7989edc172493547cdf63e180bb58365e8a43a42
25c8a8395527d98976be6a34baa3b0b214776739
792e8cfa8f748077f9d80fe7ee5e958b7124e83b

* cranelift: Add LibCall tests

* cranelift: Revert more test names

These weren't auto reverted by the previous revert.

* cranelift: Disable libcall tests for aarch64

* cranelift: Runtest fibonacci tests

* cranelift: Misc cleanup

show more ...

Don't limit ExternalName::TestName length (#4764)

In order to keep the `ExternalName` enum small, the `TestcaseName`
struct was limited to 17 bytes: a 1 byte length and a 16 byte buffer.
Due to al

Don't limit ExternalName::TestName length (#4764)

In order to keep the `ExternalName` enum small, the `TestcaseName`
struct was limited to 17 bytes: a 1 byte length and a 16 byte buffer.
Due to alignment, that made `ExternalName` 20 bytes.

That fixed-size buffer means that the names of functions in Cranelift
filetests are truncated to fit, which limits our ability to give tests
meaningful names. And I think meaningful names are important in tests.

This patch replaces the inline `TestcaseName` buffer with a
heap-allocated slice. We don't care about performance for test names, so
an indirection out to the heap is fine in that case. But we do care
somewhat about the size of `ExternalName` when it's used during
compiles.

On 64-bit systems, `Box<[u8]>` is 16 bytes, so `TestcaseName` gets one
byte smaller. Unfortunately, its alignment is 8 bytes, so `ExternalName`
grows from 20 to 24 bytes.

According to `valgrind --tool=dhat`, this change has very little effect
on compiler performance. Building wasmtime with `--no-default-features
--release`, and compiling the pulldown-cmark benchmark from Sightglass,
I measured these differences between `main` and this patch:

- total number of allocations didn't change (`ExternalName::TestCase` is
not used in normal compiles)
- 592 more bytes allocated over the process lifetime, out of 171.5MiB
- 320 more bytes allocated at peak heap size, out of 12MiB
- 0.24% more instructions executed
- 16,987 more bytes written
- 12,120 _fewer_ bytes read

show more ...

Implement an incremental compilation cache for Cranelift (#4551)

This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested b

Implement an incremental compilation cache for Cranelift (#4551)

This is the implementation of https://github.com/bytecodealliance/wasmtime/issues/4155, using the "inverted API" approach suggested by @cfallin (thanks!) in Cranelift, and trait object to provide a backend for an all-included experience in Wasmtime.

After the suggestion of Chris, `Function` has been split into mostly two parts:

- on the one hand, `FunctionStencil` contains all the fields required during compilation, and that act as a compilation cache key: if two function stencils are the same, then the result of their compilation (`CompiledCodeBase<Stencil>`) will be the same. This makes caching trivial, as the only thing to cache is the `FunctionStencil`.
- on the other hand, `FunctionParameters` contain the... function parameters that are required to finalize the result of compilation into a `CompiledCode` (aka `CompiledCodeBase<Final>`) with proper final relocations etc., by applying fixups and so on.

Most changes are here to accomodate those requirements, in particular that `FunctionStencil` should be `Hash`able to be used as a key in the cache:

- most source locations are now relative to a base source location in the function, and as such they're encoded as `RelSourceLoc` in the `FunctionStencil`. This required changes so that there's no need to explicitly mark a `SourceLoc` as the base source location, it's automatically detected instead the first time a non-default `SourceLoc` is set.
- user-defined external names in the `FunctionStencil` (aka before this patch `ExternalName::User { namespace, index }`) are now references into an external table of `UserExternalNameRef -> UserExternalName`, present in the `FunctionParameters`, and must be explicitly declared using `Function::declare_imported_user_function`.
- some refactorings have been made for function names:
- `ExternalName` was used as the type for a `Function`'s name; while it thus allowed `ExternalName::Libcall` in this place, this would have been quite confusing to use it there. Instead, a new enum `UserFuncName` is introduced for this name, that's either a user-defined function name (the above `UserExternalName`) or a test case name.
- The future of `ExternalName` is likely to become a full reference into the `FunctionParameters`'s mapping, instead of being "either a handle for user-defined external names, or the thing itself for other variants". I'm running out of time to do this, and this is not trivial as it implies touching ISLE which I'm less familiar with.

The cache computes a sha256 hash of the `FunctionStencil`, and uses this as the cache key. No equality check (using `PartialEq`) is performed in addition to the hash being the same, as we hope that this is sufficient data to avoid collisions.

A basic fuzz target has been introduced that tries to do the bare minimum:

- check that a function successfully compiled and cached will be also successfully reloaded from the cache, and returns the exact same function.
- check that a trivial modification in the external mapping of `UserExternalNameRef -> UserExternalName` hits the cache, and that other modifications don't hit the cache.
- This last check is less efficient and less likely to happen, so probably should be rethought a bit.

Thanks to both @alexcrichton and @cfallin for your very useful feedback on Zulip.

Some numbers show that for a large wasm module we're using internally, this is a 20% compile-time speedup, because so many `FunctionStencil`s are the same, even within a single module. For a group of modules that have a lot of code in common, we get hit rates up to 70% when they're used together. When a single function changes in a wasm module, every other function is reloaded; that's still slower than I expect (between 10% and 50% of the overall compile time), so there's likely room for improvement.

Fixes #4155.

show more ...

s390x: Implement tls_value (#4616)

Implement the tls_value for s390 in the ELF general-dynamic mode.

Notable differences to the x86_64 implementation are:
- We use a __tls_get_offset libcall ins

s390x: Implement tls_value (#4616)

Implement the tls_value for s390 in the ELF general-dynamic mode.

Notable differences to the x86_64 implementation are:
- We use a __tls_get_offset libcall instead of __tls_get_addr.
- The current thread pointer (stored in a pair of access registers)
needs to be added to the result of __tls_get_offset.
- __tls_get_offset has a variant ABI that requires the address of
the GOT (global offset table) is passed in %r12.

This means we need a new libcall entries for __tls_get_offset.
In addition, we also need a way to access _GLOBAL_OFFSET_TABLE_.
The latter is a "magic" symbol with a well-known name defined
by the ABI and recognized by the linker. This patch introduces
a new ExternalName::KnownSymbol variant to support such names
(originally due to @afonso360).

We also need to emit a relocation on a symbol placed in a
constant pool, as well as an extra relocation on the call
to __tls_get_offset required for TLS linker optimization.

Needed by the cg_clif frontend.

show more ...

Add serde serialization support for the full clif ir

Mass rename Ebb and relatives to Block (#1365)

* Manually rename BasicBlock to BlockPredecessor

BasicBlock is a pair of (Ebb, Inst) that is used to represent the
basic block subcomponent of an E

Mass rename Ebb and relatives to Block (#1365)

* Manually rename BasicBlock to BlockPredecessor

BasicBlock is a pair of (Ebb, Inst) that is used to represent the
basic block subcomponent of an Ebb that is a predecessor to an Ebb.

Eventually we will be able to remove this struct, but for now it
makes sense to give it a non-conflicting name so that we can start
to transition Ebb to represent a basic block.

I have not updated any comments that refer to BasicBlock, as
eventually we will remove BlockPredecessor and replace with Block,
which is a basic block, so the comments will become correct.

* Manually rename SSABuilder block types to avoid conflict

SSABuilder has its own Block and BlockData types. These along with
associated identifier will cause conflicts in a later commit, so
they are renamed to be more verbose here.

* Automatically rename 'Ebb' to 'Block' in *.rs

* Automatically rename 'EBB' to 'block' in *.rs

* Automatically rename 'ebb' to 'block' in *.rs

* Automatically rename 'extended basic block' to 'basic block' in *.rs

* Automatically rename 'an basic block' to 'a basic block' in *.rs

* Manually update comment for `Block`

`Block`'s wikipedia article required an update.

* Automatically rename 'an `Block`' to 'a `Block`' in *.rs

* Automatically rename 'extended_basic_block' to 'basic_block' in *.rs

* Automatically rename 'ebb' to 'block' in *.clif

* Manually rename clif constant that contains 'ebb' as substring to avoid conflict

* Automatically rename filecheck uses of 'EBB' to 'BB'

'regex: EBB' -> 'regex: BB'
'$EBB' -> '$BB'

* Automatically rename 'EBB' 'Ebb' to 'block' in *.clif

* Automatically rename 'an block' to 'a block' in *.clif

* Fix broken testcase when function name length increases

Test function names are limited to 16 characters. This causes
the new longer name to be truncated and fail a filecheck test. An
outdated comment was also fixed.

show more ...

Fix clippy warnings.

This commit fixes the current set of (stable) clippy warnings in the repo.

Rustfmt

Always use extern crate std in cranelift-codegen

moved crates in lib/ to src/, renamed crates, modified some files' text (#660)

moved crates in lib/ to src/, renamed crates, modified some files' text (#660)