1# Reducing Test Cases
2
3When reporting a bug, or investing a bug report, in Wasmtime it is easier for
4everyone involved when there is a test case that reproduces the bug. It is even
5better when that test case is as small as possible, so that developers don't
6need to wade through megabytes of unrelated Wasm that isn't necessary to
7showcase the bug. The process of taking a large test case and stripping out the
8unnecessary bits is called *test case reduction*.
9
10[The `wasm-tools shrink` tool](github.com/bytecodealliance/wasm-tools) can
11automatically reduce Wasm test cases when given
12
131. the original, unreduced test case, and
142. a predicate script to determine whether the bug reproduces on a given reduced
15   test case candidate.
16
17If the test case causes Wasmtime to segfault, the script can run Wasmtime and
18check its exit code. If the test case produces a different result in Wasmtime vs
19another Wasm engine, the script can run both engines and compare their
20results. It is also often useful to `grep` through the candidate's WAT
21disassembly to make sure that relevant features and instructions are present.
22
23## Case Study: [Issue #7779](https://github.com/bytecodealliance/wasmtime/issues/7779)
24
25A bug was reported involving the `memory.init` instruction. The attached test
26case was larger than it needed to be and contained a bunch of functions and
27other things that were irrelevant. A perfect use case for `wasm-tools shrink`!
28
29First, we needed a predicate script to identify the reported buggy behavior. The
30script is given the candidate test case as its first argument and must exit zero
31if the candidate exhibits the bug and non-zero otherwise.
32
33```bash
34#!/usr/bin/env bash
35
36# Propagate failure: exit non-zero if any individual command exits non-zero.
37set -e
38
39# Disassembly the candidate into WAT. Make sure the `memory.init` instruction
40# is present, since the bug report is about that instruction. Additionally, make
41# sure it is referencing the same data segment.
42wasm-tools print $1 | grep -q 'memory.init 2'
43
44# Make sure that the data segment in question remains unchanged, as mutating its
45# length can change the semantics of the `memory.init` instruction.
46wasm-tools print $1 | grep -Eq '\(data \(;2;\) \(i32\.const 32\) "\\01\\02\\03\\04\\05\\06\\07\\08\\ff"\)'
47
48# Make sure that the `_start` function that contains the `memory.init` is still
49# exported, so that running the Wasm will run the `memory.init` instruction.
50wasm-tools print $1 | grep -Eq '\(export "_start" \(func 0\)\)'
51
52# Run the testcase in Wasmtime and make sure that it traps the same way as the
53# original test case.
54cargo run --manifest-path ~/wasmtime/Cargo.toml -- run $1 2>&1 \
55    | grep -q 'wasm trap: out of bounds memory access'
56```
57
58Note that this script is a little fuzzy! It just checks for `memory.init` and a
59particular trap. That trap can correctly occur according to Wasm semantics when
60`memory.init` is given certain inputs! This means we need to double check that
61the reduced test case actually exhibits a real bug and its inputs haven't been
62transformed into something that Wasm semantics specify should indeed
63trap. Sometimes writing very precise predicate scripts is difficult, but we do
64the best we can and usually it works out fine.
65
66With the predicate script in hand, we can automatically reduce the original test
67case:
68
69```shell-session
70$ wasm-tools shrink predicate.sh test-case.wasm
71369 bytes (1.07% smaller)
72359 bytes (3.75% smaller)
73357 bytes (4.29% smaller)
74354 bytes (5.09% smaller)
75344 bytes (7.77% smaller)
76...
77118 bytes (68.36% smaller)
78106 bytes (71.58% smaller)
7994 bytes (74.80% smaller)
8091 bytes (75.60% smaller)
8190 bytes (75.87% smaller)
82
83test-case.shrunken.wasm :: 90 bytes (75.87% smaller)
84================================================================================
85(module
86  (type (;0;) (func))
87  (func (;0;) (type 0)
88    (local i32 f32 i64 f64)
89    i32.const 0
90    i32.const 9
91    i32.const 0
92    memory.init 2
93  )
94  (memory (;0;) 1 5)
95  (export "_start" (func 0))
96  (data (;0;) (i32.const 8) "")
97  (data (;1;) (i32.const 16) "")
98  (data (;2;) (i32.const 32) "\01\02\03\04\05\06\07\08\ff")
99)
100================================================================================
101```
102
103In this case, the arguments to the original `memory.init` instruction haven't
104changed, and neither has the relevant data segment, so the reduced test case
105should exhibit the same behavior as the original.
106
107In the end, it was [determined that Wasmtime was behaving as
108expected](https://github.com/bytecodealliance/wasmtime/issues/7779#issuecomment-1894350625),
109but the presence of the reduced test case makes it much easier to make that
110determination.
111