1# test-programs
2
3This crate contains many different binaries which are built into Wasm test
4programs for other crates in the Wasmtime workspace.
5
6Each file in `./src/bin/` is built into a distinct test binary. By convention,
7the prefix before the first underscore in each binary is the name of the test
8suite: `api`, `cli`, `http`, `nn`, `piped`, `preview1`, and `preview2`.
9
10## Test suites
11
12### api
13
14This suite is used for exercising `wasmtime`'s Rust APIs. Each test program in
15this suite has particular behavior which interacts with the tests themselves.
16
17These test programs are executed in `/crates/wasi/tests/api.rs`.
18
19### cli
20
21This suite is used for exercising the `wasmtime-cli` crate, which produces the
22`wasmtime` native executable. Each test program in this suite has particular
23behavior which interacts with the commands and arguments passed to the
24wasmtime CLI. Some use the `wasmtime run` command and others the `wasmtime
25serve` command.
26
27These test programs are executed in `/tests/all/cli_tests.rs`.
28
29### http
30
31This test suite is used for exercising the `wasmtime-wasi-http` crate. Each
32test program in this suite is a component which exports the wasi-cli command
33world's run interface (i.e. its entrypoint is defined by `fn main() {...}`),
34and is given the cli command world imports as well as the wasi-http
35outgoing-handler and types imports - in short, this is a basic executable
36which can make http requests using wasi-http.
37
38Each test program executor spawns a fresh tokio runtime on a unique local
39socket which responds to the requests made by a single test program. This
40socket address is passed to the test program using the `HTTP_SERVER`
41environment variable.
42
43These test programs are executed in `/crates/wasi-http/tests/all/sync.rs`
44and `/crates/wasi-http/tests/all/async_.rs`, which test the synchronous and
45async rust interfaces of the `wasmtime` and `wasmtime-wasi-http` crates,
46respectively.
47
48### nn
49
50This test suite is used for exercising the `wasmtime-wasi-nn` crate. It only
51tests the witx ABI of the wasi-nn proposal.
52
53These test programs are executed in `crates/wasi-nn/tests/all.rs`.
54
55### piped
56
57This suite is used for exercising the wasmtime-cli and, in particular,
58examining the behavior when the standard output of one instance is piped to
59the input of another instance. The test programs in this suite are passed
60the environment variable `PIPED_SIDE` with the value `PRODUCER` or `CONSUMER`
61to select the behavior of producing output to stdout, or consuming input
62on stdin.
63
64These test programs are executed in `/tests/all/cli_tests.rs`, which creates
65two separate `wasmtime` host processes, with the consumer process stdin file
66descriptor set to the producer process stdout file descriptor. The test
67harness asserts that both processes exit successfully.
68
69### preview1
70
71This suite is used for exercising the Wasi Preview 1 (aka Wasi 0.1) interface,
72and, additionally, provides test coverage of Wasi Preview 2 (aka Wasi 0.2)
73when used with the component adapter, as described below.
74
75Some of these tests require the stdin stream provided is always empty and
76closed, others require that the stdin stream is open and pending forever. The
77test execution provides this by inheriting the host process stdio in some test
78program invocations, and providing the default null stdio in others.
79
80All of these tests require an empty directory be provided as a preopen, and
81the path of that preopen passed as the first argument.
82
83The Wasmtime tree has three distinct ways to implement the Wasi 0.1 interface,
84which should all provide identical functionality:
85* The Wasm Module is executed against the host implementation of Wasi 0.1
86  provided by the `wasi-common` crate. The test programs are executed
87  in `/crates/wasi-common/tests/all/{sync,async_}.rs` against the synchronous
88  and async Rust interfaces of the `wasmtime` and `wasi-common` crate.
89* The Wasm Module is executed against the host implementation of Wasi 0.1
90  provided by the `wasmtime-wasi` crate. This Wasi 0.1 interface is implemented
91  using the Wasi 0.2 implementation defined in that same crate. The test
92  programs are executed in `/crates/wasi/tests/all/preview1.rs` against the
93  synchronous Rust interfaces of the `wasmtime` and `wasmtime-wasi` crates.
94* The Wasm Component created by composing the Module with the
95  `wasi-preview1-component-adapter`, which provides a pure WebAssembly
96  implementation of the Wasi 0.1 import functions in terms of the Wasi 0.2
97  import functions. When run in the host implementation of Wasi 0.2 provided
98  by the `wasmtime-wasi` crate, this mode of execution exercises the Wasi 0.2
99  interfaces, and ensures that a faithful implementation of Wasi 0.1 can be
100  provided in terms of them, without needing any support for Wasi 0.1 in the
101  host. The test programs are executed in
102  `crates/wasi/tests/all/{sync,async_}.rs` against the synchronous and async
103  Rust interfaces of the `wasmtime` and `wasmtime-wasi` crates.
104
105### preview2
106
107This suite is used for exercising the portions of Wasi Preview 2 (aka Wasi
1080.2) which are not covered by the Wasi 0.1 + component adapter, as described
109in the preview1 suite above.
110
111These test programs use `wit-bindgen` to generate bindings to Wasi 0.2
112(invoked at the root of the `test-programs` crate), and then invoke those
113bindings directly in each test. The modules created from these test programs
114still have imports of Wasi 0.1 thanks to the Rust `std` library, so those
115imports are implemented by the same `wasi-preview1-component-adapter` as in
116the preview1 suite. (When Rust stabilizes the `wasm32-wasip2` target, this
117will finally get to be removed!) However, the tests themselves should be
118exercising the Wasi 0.2 interfaces directly - in some cases, such as the
119socket tests, its exercising functionality that Wasi 0.1 lacked.
120
121These tests are executed much like the component adapted preview1 test suite
122is, in `crates/wasi/tests/all/{sync,async_}.rs`.
123
124## Building and executing these test programs
125
126These tests are built and consumed by way of the `test-programs-artifacts`
127crate. `test-programs-artifacts` has a `build.rs` which compiles this crate
128and translates some of the resulting Wasm Modules to Wasm Components. It then
129exposes the filesystem paths of these binaries as string constants, and as
130macros which iterate over the complete set of binary paths in a given test
131suite.
132
133The filesystem path to each Wasm Module built by `test-programs-artifacts`
134is provided by a `pub const <TESTNAME>: &'static str = ...;` at the top
135level of that crate, where `<TESTNAME>` is the file stem of the source file
136in `./src/bin`, in shouty snake case (all caps and underscores).
137
138The filesystem path to each Wasm Component built by `test-programs-artifacts`
139is provided by a `pub const <TESTNAME>_COMPONENT: &'static str = ...;` at the top
140level of that crate, where `<TESTNAME>` is the file stem of the source file
141in `./src/bin`, in shouty snake case (all caps and underscores), and then the
142suffix `_COMPONENT` is added to the identifier to distinguish it from the
143module.
144
145Wasm Components are not built for the `nn` test suite. All other components
146are created, using the `wasi-preview1-component-adapter` to translate Wasi 0.1
147module imports to Wasi 0.2 component imports.
148
149Each test suite is given an iterator macro in `test-programs-artifacts` named
150`foreach_<suitename>`.
151
152The iterator macro is passed the identifier of another macro which is passed
153the path of each test in the suite. You could do something very clever with
154this if you desire, but as of this writing, all use sites in the workspace
155pass it a macro `assert_test_exists` which emits a `use <testname> as _;`
156statement. This ensures that, in the rust mod where `foreach_<suitename>` is
157invoked, a `fn <testname>() {...}` is defined, which in turn makes sure that,
158if a wasmtime developer adds a new test program, they also add a test which
159executes it in the appropriate location.
160
161By convention, the expectation is that each mod that invokes
162`foreach_<suitename>!(assert_test_exists);` also defines something like:
163
164```
165#[test]
166fn <testname>() {
167    // do something which executes <TESTNAME> or
168    // <TESTNAME>_COMPONENT here.
169}
170```
171