Consolidate building guest programs for tests (#10654)

* Consolidate building guest programs for tests

This commit folds all C/C++ tests for debuginfo into the same location
as Rust programs compil

Consolidate building guest programs for tests (#10654)

* Consolidate building guest programs for tests

This commit folds all C/C++ tests for debuginfo into the same location
as Rust programs compiled to wasm. That means that all of
`tests/all/debug/testsuite/*` is moved to
`crates/test-programs/src/bin/*`. Additionally the hardcoded per-test
configuration in `crates/test-programs/artifacts/build.rs` was removed
in favor of in-test configuration at the top of the file. This means
that all precompiled binaries are removed from the tree for DWARF
testing.

Various offsets were updated, a test was dropped as it wasn't used, and
some commands/directives were updated to account for this adjustment.

prtest:debug

* Adjust a lint setting

show more ...

Update some dependencies on their major version tracks (#10425)

* Update rustix to 1.0.x

* Bump itertools to its latest version

* Update base64 to its latest version

* Update wit-bindgen to its l

Update some dependencies on their major version tracks (#10425)

* Update rustix to 1.0.x

* Bump itertools to its latest version

* Update base64 to its latest version

* Update wit-bindgen to its latest version

* Update v8 to its latest version on crates.io

Just keeping up-to-date

* Update capstone dependency to its latest version

* Update libtest-mimic to its latest version

* Update cargo-metadata dependency

* Update thiserror dependency to latest

* Update bytesize dependency

* Drop getrandom dependency from test-programs

Favor using `wasi::random_get` instead for now.

* Fix deny.toml syntax

* Fix merge conflict

* Downgrade v8 to respect MSRV

* Fix compile on windows

show more ...

Add `rust-version.workspace = true` to all crates (#9112)

Right now this is only on some crates such as `wasmtime` itself and
`wasmtime-cli`, but by applying it to all crates it helps with version
s

Add `rust-version.workspace = true` to all crates (#9112)

Right now this is only on some crates such as `wasmtime` itself and
`wasmtime-cli`, but by applying it to all crates it helps with version
selection of those using just Cranelift for example.

show more ...

Revert "wasi-adapter: Implement provider crate that embeds the adapter binaries (#8792)" (#8856)

* Revert "wasi-adapter: Implement provider crate that embeds the adapter binaries (#8792)"

This reve

Revert "wasi-adapter: Implement provider crate that embeds the adapter binaries (#8792)" (#8856)

* Revert "wasi-adapter: Implement provider crate that embeds the adapter binaries (#8792)"

This reverts commit 2dbf8f15b3c07b51d090782d9be7172358799563.

* Enable requisite feature for cranelift-frontend testing

show more ...

wasi-adapter: Implement provider crate that embeds the adapter binaries (#8792)

* wasi-adapter: Implement provider crate that embeds the adapter binaries

* Upgrade wasi adapters to the latest versi

wasi-adapter: Implement provider crate that embeds the adapter binaries (#8792)

* wasi-adapter: Implement provider crate that embeds the adapter binaries

* Upgrade wasi adapters to the latest version

* Update adapter docs

* Recompile asi adapters with 1.78

* Recompile wasi adapters with 1.79

* Add some debugging to adapter build script

* Fix script debugging

* Compute wasi adapter version based on latest adapter commit hash

* Try to bless wasi adapters again

* Try to work around CI auto-merges

* Revert to just using workspace version

* Add the wasi adapter provider to the crate publication list

* Use wasi adapter provider in artifacts test + explicit MSRV in CI

* Explicit adapter crate version

* Small fix

* Remove version info from adapter metadata

* Check but don't install rust toolchain in build script

* Bless after rebase

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Remove the borrow checking from `wiggle` entirely (#8702)

* Remove the borrow checking from `wiggle` entirely

This commit is a refactoring of the `wiggle` crate which powers the
`*.witx`-based bind

Remove the borrow checking from `wiggle` entirely (#8702)

* Remove the borrow checking from `wiggle` entirely

This commit is a refactoring of the `wiggle` crate which powers the
`*.witx`-based bindings generation of Wasmtime for wasip1 support.
Originally `wiggle` had a full-blown runtime borrow checker which
verified that borrows were disjoint when appropriate. In #8277 this was
removed in favor of a more coarse "either all shared or all mutable"
guarantee. It turns out that this exactly matches what the Rust type
system guarantees at compile time as well.

This commit removes all runtime borrow checking in favor of compile-time
borrow checking instead. This means that there is no longer the
possibility of a runtime error arising from borrowing errors. Current
bindings in Wasmtime needed no restructuring to work with this new API.

The source of the refactors here are all in the `wiggle` crate. Changes
include:

* The `GuestPtr` type lost its type parameter. Additionally it only
contains a `u32` pointer now instead.
* The `GuestMemory` trait is replaced with a simple `enum` of
possibilities.
* Helper methods on `GuestPtr` are all moved to `GuestMemory`.
* A number of abstractions were simplified now that borrow checking is
no longer necessary.
* Generated trait methods now all take `&mut GuestMemory<'_>` as an
argument.

These changes were then propagated to the `wasmtime-wasi` and
`wasi-common` crates in their preview0 and preview1 implementations of
WASI. All changes are just general refactors, no functional change is
intended here.

* Review comments

* Fix publishing of wiggle-macro crate

* Fix wiggle docs

show more ...

Use an incremental cache when testing WASI (#8354)

Currently we've got a good number of WASI tests and they're all
relatively large. We also can run a single test in up to three
configurations:

* A

Use an incremental cache when testing WASI (#8354)

Currently we've got a good number of WASI tests and they're all
relatively large. We also can run a single test in up to three
configurations:

* As-is with a module
* As a component in "sync" mode
* As a component in "async" mode

In debug mode compilation of all these modules can take a significant
chunk of time (20-30s in total for test suites) This commit updates
these test suites to use an in-memory per-process incremental cache
backed by a simple `Mutex<HashMap>`. This gives some good speedups in
debug mode, locally the wasi-common, wasmtime-wasi, and
wasmtime-wasi-http test suites were reduced from 32 to 17 seconds. I'd
expect larger speedups on less-parallel machines such as our CI.

show more ...

Update a few crates on their major version tracks (#7908)

Found via `cargo outdated` and the vets here were easy enough.

Configure workspace lints, enable running some Clippy lints on CI (#7561)

* Configure Rust lints at the workspace level

This commit adds necessary configuration knobs to have lints configured
at th

Configure workspace lints, enable running some Clippy lints on CI (#7561)

* Configure Rust lints at the workspace level

This commit adds necessary configuration knobs to have lints configured
at the workspace level in Wasmtime rather than the crate level. This
uses a feature of Cargo first released with 1.74.0 (last week) of the
`[workspace.lints]` table. This should help create a more consistent set
of lints applied across all crates in our workspace in addition to
possibly running select clippy lints on CI as well.

* Move `unused_extern_crates` to the workspace level

This commit configures a `deny` lint level for the
`unused_extern_crates` lint to the workspace level rather than the
previous configuration at the individual crate level.

* Move `trivial_numeric_casts` to workspace level

* Change workspace lint levels to `warn`

CI will ensure that these don't get checked into the codebase and
otherwise provide fewer speed bumps for in-process development.

* Move `unstable_features` lint to workspace level

* Move `unused_import_braces` lint to workspace level

* Start running Clippy on CI

This commit configures our CI to run `cargo clippy --workspace` for all
merged PRs. Historically this hasn't been all the feasible due to the
amount of configuration required to control the number of warnings on
CI, but with Cargo's new `[lint]` table it's possible to have a
one-liner to silence all lints from Clippy by default. This commit by
default sets the `all` lint in Clippy to `allow` to by-default disable
warnings from Clippy. The goal of this PR is to enable selective access
to Clippy lints for Wasmtime on CI.

* Selectively enable `clippy::cast_sign_loss`

This would have fixed #7558 so try to head off future issues with that
by warning against this situation in a few crates. This lint is still
quite noisy though for Cranelift for example so it's not worthwhile at
this time to enable it for the whole workspace.

* Fix CI error

prtest:full

show more ...

Refactor the test-programs test suite (#7182)

* Refactor the test-programs test suite

This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often w

Refactor the test-programs test suite (#7182)

* Refactor the test-programs test suite

This commit is a large refactoring that reorganizes `test-programs` and
how we tests wasms in Wasmtime. Often writing tests requires complicated
interactions with the guest which can't be done via hand-written `*.wat`
syntax and requires a compiler to get engaged. For this purpose Wasmtime
currently has the `crates/test-programs/*` test suite which builds files
from source and then runs the tests. This has been somewhat cumbersome
in the past though and it's not been easy to extend this over time, so
this commit attempts to address this.

The scheme implemented in this PR looks like:

* All wasm test programs live in `crates/test-programs/src/bin/*.rs`.
All of them, no exceptions.

* Wasm tests have shared support located at
`crates/test-programs/src/lib.rs` and its submodules, such as bindings
generation for WASI.

* Wasm tests are built by a new `crates/test-programs/artifacts` crate.
This crate compiles modules and additionally creates components for
all test programs. The crate itself only records the path to these
outputs and a small amount of testing support, but otherwise doesn't
interact with `wasmtime`-the-crate itself.

* All tests in `crates/test-programs/tests/*.rs` have moved. For example
wasi-http tests now live at `crates/wasi-http/tests/*.rs`. Legacy
tests of wasi-common now live at `crates/wasi-common/tests/*.rs`.
Modern tests for preview2 live at `crates/wasi/tests/*.rs`.

* Wasm tests are bucketed based on their filename prefix. For example
`preview1_*` is tested in wasi-common and wasmtime-wasi. The
`preview2_*` prefix is only tested with wasmtime-wasi, however.

* A new `cli_*` prefix is used to execute tests as part of
`tests/all/main.rs`. This is a new submodule in
`tests/all/cli_tests.rs` which executes these components on the
command line. Many old "command" tests were migrated here.

* Helper macros are generated to assert that a test suite is run in its
entirety. This way if a `preview1_*` test is added it's asserted to
get added to both wasi-common and wasmtime-wasi in the various modes
they run tests.

Overall this moved a number of tests around and refactored some edges of
the tests, but this should not lose any tests (except one that wasn't
actually testing anything). Additionally the hope is that it's much
easier to add tests in the future. The process is to add a new file in
`crates/test-programs/src/bin/*.rs` named appropriately. For example a
preview2 executable is `preview2_*` and a CLI tests is `cli_*`. When
building the test suite an error is generated in the appropriate module
then of "please write a test here", and then a test is written in the
same manner as the other tests in the module.

* Remove no-longer-needed fetches

prtest:full

* I'm worried wasi is running low on semicolons

* Add the WASI target in all CI actions

* Add unknown-unknown target on all CI builders too

* Fix building test artifacts under miri

Need to avoid wrappers for these cross-compiled targets

* Break circular dependency for packaging

Don't use the workspace dep for `wasmtime-wasi` since it injects a
version, instead use a `path = '..'` dependency to fool Cargo into
dropping the dependency during the package phase.

* Fix some merge conflicts with tests

* Fix rebase for new tests

* Remove stray comment

* Fix some flaky tests

* Fix network tests in synchronous mode

This commit is an attempt to fix some networking tests in synchronous
mode in our test suite. Currently networking tests don't actually run in
synchronous mode on CI which is why no failures have been surfaced yet,
but the refactoring in #7182 is going to start doing this.

Currently the `udp_sample_application.rs` test blocks infinitely in
synchronous mode for me locally, most of the time. This appears to be an
interaction between how Tokio handles readiness and how we're
entering the event loop. We're effectively entering the Tokio event loop
with a future that's always ready which ends up starving Tokio of
otherwise performing its background work such as updating flags for
readiness of reading/writing.

The fix here is to add a yield at the start of an `in_tokio` block which
is used in synchronous mode. This is a kludge fix but the intention is
to enable Tokio to have a chance to update readiness flags and process
events from epoll/kqueue/etc.

An additional fix to this issue is WebAssembly/wasi-sockets#64 where the
test is waiting on `READABLE` or `WRITABLE`, but in this specific case
it should only wait on `READABLE`. If it waited on just this then that
would also fix this issue. Nevertheless having a `yield_now` is expected
to have little-to-no overhead and otherwise fix this edge case of an
always-ready future.

* Fix passing empty arguments on the CLI

* Add another blocking accept

* Update crates/test-programs/src/bin/api_proxy.rs

Co-authored-by: Trevor Elliott <[email protected]>

---------

Co-authored-by: Trevor Elliott <[email protected]>

show more ...