1# Reducing Test Cases
2
3When reporting a bug, or investing a bug report, in Wasmtime it is easier for
4everyone involved when there is a test case that reproduces the bug. It is even
5better when that test case is as small as possible, so that developers don't
6need to wade through megabytes of unrelated Wasm that isn't necessary to
7showcase the bug. The process of taking a large test case and stripping out the
8unnecessary bits is called *test case reduction*.
9
10[The `wasm-tools shrink` tool](https://github.com/bytecodealliance/wasm-tools) can
11automatically reduce Wasm test cases when given
12
131. the original, unreduced test case, and
142. a predicate script to determine whether the bug reproduces on a given reduced
15   test case candidate.
16
17If the test case causes Wasmtime to segfault, the script can run Wasmtime and
18check its exit code. If the test case produces a different result in Wasmtime vs
19another Wasm engine, the script can run both engines and compare their
20results. It is also often useful to `grep` through the candidate's WAT
21disassembly to make sure that relevant features and instructions are present.
22
23Note that there are also a few other test-case reducers that can operate on
24Wasm. All of them, including `wasm-shrink`, work fairly similarly at a high
25level, but often if one reducer gets stuck in a local minimum, another reducer
26can pick up from there and reduce the test case further due to differences in
27the details of their implementations. Therefore, if you find that `wasm-shrink`
28isn't very effective on a particular test case, you can try continuing reduction
29with one of the following:
30
31* [Binaryen's `wasm-reduce`
32  tool](https://github.com/WebAssembly/binaryen?tab=readme-ov-file#tools)
33* [`creduce`, which can be effective at reducing Wasm test cases when
34  disassembled into their `.wat` text
35  format](https://github.com/csmith-project/creduce)
36
37## Case Study: [Issue #7779](https://github.com/bytecodealliance/wasmtime/issues/7779)
38
39A bug was reported involving the `memory.init` instruction. The attached test
40case was larger than it needed to be and contained a bunch of functions and
41other things that were irrelevant. A perfect use case for `wasm-tools shrink`!
42
43First, we needed a predicate script to identify the reported buggy behavior. The
44script is given the candidate test case as its first argument and must exit zero
45if the candidate exhibits the bug and non-zero otherwise.
46
47```bash
48#!/usr/bin/env bash
49
50# Propagate failure: exit non-zero if any individual command exits non-zero.
51set -e
52
53# Disassembly the candidate into WAT. Make sure the `memory.init` instruction
54# is present, since the bug report is about that instruction. Additionally, make
55# sure it is referencing the same data segment.
56wasm-tools print $1 | grep -q 'memory.init 2'
57
58# Make sure that the data segment in question remains unchanged, as mutating its
59# length can change the semantics of the `memory.init` instruction.
60wasm-tools print $1 | grep -Eq '\(data \(;2;\) \(i32\.const 32\) "\\01\\02\\03\\04\\05\\06\\07\\08\\ff"\)'
61
62# Make sure that the `_start` function that contains the `memory.init` is still
63# exported, so that running the Wasm will run the `memory.init` instruction.
64wasm-tools print $1 | grep -Eq '\(export "_start" \(func 0\)\)'
65
66# Run the testcase in Wasmtime and make sure that it traps the same way as the
67# original test case.
68cargo run --manifest-path ~/wasmtime/Cargo.toml -- run $1 2>&1 \
69    | grep -q 'wasm trap: out of bounds memory access'
70```
71
72Note that this script is a little fuzzy! It just checks for `memory.init` and a
73particular trap. That trap can correctly occur according to Wasm semantics when
74`memory.init` is given certain inputs! This means we need to double check that
75the reduced test case actually exhibits a real bug and its inputs haven't been
76transformed into something that Wasm semantics specify should indeed
77trap. Sometimes writing very precise predicate scripts is difficult, but we do
78the best we can and usually it works out fine.
79
80With the predicate script in hand, we can automatically reduce the original test
81case:
82
83```shell-session
84$ wasm-tools shrink predicate.sh test-case.wasm
85369 bytes (1.07% smaller)
86359 bytes (3.75% smaller)
87357 bytes (4.29% smaller)
88354 bytes (5.09% smaller)
89344 bytes (7.77% smaller)
90...
91118 bytes (68.36% smaller)
92106 bytes (71.58% smaller)
9394 bytes (74.80% smaller)
9491 bytes (75.60% smaller)
9590 bytes (75.87% smaller)
96
97test-case.shrunken.wasm :: 90 bytes (75.87% smaller)
98================================================================================
99(module
100  (type (;0;) (func))
101  (func (;0;) (type 0)
102    (local i32 f32 i64 f64)
103    i32.const 0
104    i32.const 9
105    i32.const 0
106    memory.init 2
107  )
108  (memory (;0;) 1 5)
109  (export "_start" (func 0))
110  (data (;0;) (i32.const 8) "")
111  (data (;1;) (i32.const 16) "")
112  (data (;2;) (i32.const 32) "\01\02\03\04\05\06\07\08\ff")
113)
114================================================================================
115```
116
117In this case, the arguments to the original `memory.init` instruction haven't
118changed, and neither has the relevant data segment, so the reduced test case
119should exhibit the same behavior as the original.
120
121In the end, it was [determined that Wasmtime was behaving as
122expected](https://github.com/bytecodealliance/wasmtime/issues/7779#issuecomment-1894350625),
123but the presence of the reduced test case makes it much easier to make that
124determination.
125