1# Testing
2
3This section describes how to run Wasmtime's tests and add new tests.
4
5Before continuing, make sure you can [build
6Wasmtime](./contributing-building.md) successfully. Can't run the tests if you
7can't build it!
8
9## Installing `wasm32` Targets
10
11To compile the tests, you'll need the `wasm32-wasip1` and
12`wasm32-unknown-unknown` targets installed, which, assuming you're using
13[rustup.rs](https://rustup.rs) to manage your Rust versions, can be done as
14follows:
15
16```console
17rustup target add wasm32-wasip1 wasm32-unknown-unknown
18```
19
20## Running Tests
21
22Depending on what you're modifying there's a few commands you may be the most
23interested:
24
25* `cargo test` - used to run the `tests/*` folder at the top-level. This tests
26  the CLI and contains most tests for the `wasmtime` crate itself. This will
27  also run all spec tests. Note that this does not run all tests in the
28  repository, but it's generally a good starting point.
29* `cargo test -p cranelift-tools` - used if you're working on Cranelift and this
30  will run all the tests at `cranelift/filetests/filetests`. You can also,
31  within the `cranelift` folder, run `cargo run test ./filetests` to run these
32  tests.
33* `cargo test -p wasmtime-wasi` - this will run all WASI tests for the
34  `wasmtime-wasi` crate.
35
36At this time not all of the crates in the Wasmtime workspace can be tested, so
37running all tests is a little non-standard. To match what CI does and run all
38tests you'll need to execute
39
40```console
41./ci/run-tests.py
42```
43
44## Testing a Specific Crate
45
46You can test a particular Wasmtime crate with `cargo test -p
47wasmtime-whatever`. For example, to test the `wasmtime-environ` crate, execute
48this command:
49
50```console
51cargo test -p wasmtime-environ
52```
53
54Alternatively, you can `cd` into the crate's directory, and run `cargo test`
55there, without needing to supply the `-p` flag:
56
57```console
58cd crates/environ/
59cargo test
60```
61
62## Running the Wasm Spec Tests
63
64The spec testsuite itself is in a git submodule, so make sure you've
65checked it out and initialized its submodule:
66
67```console
68git submodule update --init
69```
70
71When the submodule is checked out, Wasmtime runs the Wasm spec testsuite as part
72of testing the `wasmtime-cli` crate at the crate root, meaning in the root of
73the repository you can execute:
74
75```console
76cargo test --test wast
77```
78
79You can pass an additional CLI argument to act as a filter on which tests to
80run. For example to only run the spec tests themselves (excluding handwritten
81Wasmtime-specific tests) and only in Cranelift you can run:
82
83```console
84cargo test --test wast Cranelift/tests/spec
85```
86
87Note that in general spec tests are executed regardless of whether they pass
88or not. In `tests/wast.rs` there's a `should_fail` function which indicates the
89expected result of the test. When adding new spec tests or implementing features
90this function will need to be updated as tests change from failing to passing.
91
92## Running WASI Integration Tests
93
94WASI integration tests can be run separately from all other tests which
95can be useful when working on the `wasmtime-wasi` crate. This can be done by
96executing this command:
97
98```console
99cargo test -p wasmtime-wasi
100```
101
102Similarly if you're testing HTTP-related functionality you can execute:
103
104```console
105cargo test -p wasmtime-wasi-http
106```
107
108Note that these tests will compile programs in `crates/test-programs` to run.
109
110## Adding New Tests
111
112### Adding Rust's `#[test]`-Style Tests
113
114For very "unit-y" tests, we add `test` modules in the same `.rs` file as the
115code that is being tested. These `test` modules are configured to only get
116compiled during testing with `#[cfg(test)]`.
117
118```rust
119// some code...
120
121#[cfg(test)]
122mod tests {
123    use super::*;
124
125    #[test]
126    fn some_test_for_that_code() {
127        // ...
128    }
129}
130```
131
132If you're writing a unit test and a `test` module doesn't already exist, you can
133create one.
134
135For more "integration-y" tests, each crate supports a separate `tests` directory
136within the crate, and put the tests inside there. Most integration tests in
137Wasmtime are located in the root `tests/*.rs` location, notably
138`tests/all/*.rs`. This tests much of the `wasmtime` crate for example and
139facilitates `cargo test` at the repository root running most tests.
140
141Some tests make more sense to live per-crate, though.  For example, many WASI
142tests are at `crates/wasi/tests/*.rs`. For adding a test feel free to add it
143wherever feels best, there's not really a strong reason to put it in one place
144over another. While it's easiest to add to existing tests it's ok to add a new
145`tests` directory with tests too.
146
147### Adding Specification-Style Wast Tests
148
149We use the spec testsuite as-is and without custom patches or a forked
150version via a submodule at `tests/spec_testsuite`. This probably isn't what you
151want to modify when adding a new Wasmtime test!
152
153When you have a Wasmtime-specific test that you'd like to write in Wast and use
154the Wast-style assertions, you can add it to our "misc testsuite". The misc
155testsuite uses the same syntax and assertions as the spec testsuite, but lives
156in `tests/misc_testsuite`. Feel free to add new tests to existing
157`tests/misc_testsuite/*.wast` files or create new ones as needed. These tests
158are run from the crate root:
159
160```console
161cargo test --test wast
162```
163
164If you have a new test that you think really belongs in the spec testsuite, make
165sure it makes sense for every Wasm implementation to run your test (i.e. it
166isn't Wasmtime-specific) and send a pull request
167[upstream](https://github.com/WebAssembly/spec). Once it is accepted in the
168upstream repo, it'll make its way to the test-specific mirror at
169[WebAssembly/testsuite](https://github.com/WebAssembly/testsuite) and then we
170can update our git submodule and we'll start running the new tests.
171
172### Adding WASI Integration Tests
173
174When you have a WASI-specific test program that you'd like to include as a
175test case to run against our WASI implementation, you can add it to our
176`test-programs` crate. In particular, you should drop a main-style Rust source
177file into `crates/test-programs/src/bin/PREFIX_some_new_test.rs`. Here the
178`PREFIX` indicates what test suite it's going to run as. For example
179`preview2_*` tests are run as part of `wasmtime-wasi` crate tests. The `cli_*`
180tests are run as part of `tests/all/cli_tests.rs`. It's probably easiest to use
181a preexisting prefix. The `some_new_test` name is arbitrary and is selected as
182appropriate by you.
183
184One a test file is added you'll need to add some code to execute the tests as
185well. For example if you added a new test
186`crates/test-programs/src/bin/cli_my_test.rs` then you'll need to add a new
187function to `tests/all/cli_tests.rs` such as:
188
189```rust
190#[test]
191fn my_test() {
192    // ...
193}
194```
195
196The path to the compiled WebAssembly of your test case will be available in a
197Rust-level `const` named `CLI_MY_TEST`. There is also a component version at
198`CLI_MY_TEST_COMPONENT`. These are used to pass as arguments to
199`wasmtime`-the-CLI for example or you can use `Module::from_file`.
200
201When in doubt feel free to copy existing tests and then modify them to suit your
202needs.
203