1# Fuzzing 2 3## External Fuzzing Campaigns 4 5The Wasmtime maintainers appreciate bug reports from external fuzzing campaigns 6— when done thoughtfully and responsibly. Triaging and diagnosing bug reports, 7particularly reports for bugs found by fuzzers, takes a lot of time and effort 8on the part of Wasmtime maintainers. We ask that you match that effort by 9following the guidelines below. 10 11### Talk To Us First 12 13We would love to collaborate and help you find bugs in Wasmtime! We do lots of 14fuzzing already (see the docs below about our internal fuzzing infrastructure) 15but we always have ideas for new kinds of generators for directed fuzzing, new 16oracles that we wish we had, areas of code that we wish were better fuzzed, and 17etc... We can share which of Wasmtime's properties are most important to us and 18what kinds of bugs we value discovering the most. It is also good for us to know 19that an external fuzzing campaign is spinning up and that we should be on the 20look out for new issues being filed. 21 22[Come say hello on our Zulip and introduce yourself 23:)](https://bytecodealliance.zulipchat.com/#narrow/stream/217126-wasmtime) 24 25### If a Bug Might be a Security Vulnerability, Do Not File a Public Issue 26 27When you find a new bug, first evaluate it against our 28[guidelines](./security-what-is-considered-a-security-vulnerability.md) for what 29constitutes a security vulnerability. 30 31If you determine that the bug is not a security vulnerability, then [file an 32issue on our public 33tracker](https://github.com/bytecodealliance/wasmtime/issues/new/choose). 34 35If you think the bug might be considered a security vulnerability, **do not open 36a public issue detailing the bug!** Instead, follow the vulnerability reporting 37process documented [here](https://bytecodealliance.org/security). 38 39### Write Good Bug Reports 40 41The Wasmtime maintainers appreciate bug reports with the following: 42 431. **A minimal test case:** You should integrate [automatic test-case 44 reduction](./contributing-reducing-test-cases.md) into your fuzzing campaign. 45 46* **Steps to reproduce:** Simple, unambiguous steps that can be performed to 47 reproduce the buggy behavior with the test case. These steps should include 48 all options you configured Wasmtime with, such as CLI flags and 49 `wasmtime::Config` method calls. Ideally these steps are as simple for 50 maintainers to execute as running `wasmtime [OPTIONS] testcase.wasm` or a Rust 51 `#[test]` that can be run via `cargo test`. 52 53* **Expected behavior:** A description of the expected, non-buggy behavior, as 54 well as your rationale for *why* that behavior is expected. For example, just 55 because another Wasm engine or an alternative Wasmtime execution strategy 56 produces a different result from default Wasmtime, that is not necessarily a 57 bug. See the [documentation 58 below](#divergent-webassembly-behavior-across-runtimes) for examples of known divergent 59 behavior of one module in two runtimes. If applicable, make sure to account 60 for this in your rationale and analysis of the bug. 61 62* **Actual behavior:** A description of the actual, buggy behavior. This should 63 include things various things like incorrect computation results, assertion 64 failure messages, stack traces, signals raised, and etc... when applicable. 65 66* **Wasmtime version and system information:** Include the version of Wasmtime 67 you are using (either the exact release or the git commit hash) and your ISA, 68 operating system, distribution, and kernel version in the bug report. 69 70Including the above information is extra important for bugs discovered 71mechanically, whether by fuzzing or other means, since the associated test cases 72will often be pseudo-random or otherwise unintuitive to debug. 73 74### Divergent WebAssembly behavior across runtimes 75 76WebAssembly has a variety of sources of [non-determinism] which means that the 77exact same module is allowed to behave differently under the same inputs 78across multiple runtimes. These specifics don't often arise in "real world" 79modules but can quickly arise during fuzzing. Some example behaviors are: 80 81* **NaN bit patterns** - floating-point operations which produce NaN as a result 82 are allowed to produce any one of a set of patterns of NaN. This means that 83 the exact bit-representation of the result of a floating-point operation may 84 diverge across engines. When fuzzing you can update your source-generation to 85 automatically canonicalize NaN values after all floating point operations. 86 Wasmtime has built-in options to curb this [non-determinism] as well. 87 88* **Relaxed SIMD** - the `relaxed-simd` proposal to WebAssembly explicitly has 89 multiple allowed results for instructions given particular inputs. These 90 instructions are inherently non-deterministic across implementations. When 91 fuzzing you can avoid these instructions entirely, canonicalize the results, 92 or use Wasmtime's built-in options to curb the [non-determinism]. 93 94* **Call stack exhaustion** - the WebAssembly specification requires that all 95 function calls consume a nonzero-amount of some resource which can eventually 96 be exhausted. This means that infinite recursion is not allowed in any 97 WebAssembly engine. Bounded, but very large, recursion is allowed in 98 WebAssembly but is not guaranteed to work across WebAssembly engines. One 99 engine may have different stack settings than another engine and/or runtime 100 parameters may tune how much stack space is taken (e.g. optimizations on/off). 101 If one engine stack overflows and another doesn't then that's not necessarily 102 a bug in either engine. Short of banning recursion there's no known great way 103 to handle this apart from throwing out fuzz test cases that stack overflow. 104 105* **Memory exhaustion** - the `memory.grow` and `table.grow` instructions in 106 WebAssembly are not always guaranteed to either fail or succeed. This means 107 that growth may succeed in one engine but fail in another depending on various 108 settings. To handle this in fuzzing it's recommended to generate memories with 109 a maximum size and ensure that each engine being fuzzed can grow memory all 110 the way to the maximum size. 111 112* **WASIp1 API behavior** - the initial specification of WASI, WASIp1 or 113 `wasi_snapshot_preview1`, effectively is not suitable for differential fuzzing 114 across engines. The APIs are not thoroughly specified enough nor is there a 115 rigorous enough test suite to codify what exactly should happen in all 116 situations on all platforms. This means that exactly what kind of error arises 117 or various other edge cases may behave differently across engines. The lack of 118 specificity of WASIp1 means that there is no great oracle as to whether an 119 engine is right or wrong. Development of WASIp1 has ceased and the Component 120 Model is being worked on instead (e.g. WASIp2 and beyond) which is more 121 suitable for differential fuzzing. 122 123[non-determinism]: ./examples-deterministic-wasm-execution.md 124 125### Do Not Report the Same Bug Multiple Times 126 127Fuzzers will often trigger the same bug multiple times in multiple different 128ways. Do not just file an issue for every single test case where the fuzzer 129triggers an assertion failure. Many, or even most, of those test cases will 130trigger the exact same assertion failure, but perhaps with a slightly different 131stack trace. Spend some amount of effort deduplicating bugs before reporting 132them. 133 134### Do Not Report Too Many Issues At Once 135 136Please do not clog up our issue tracker by filing dozens and dozens of bug 137reports all at the same time. Choose a handful of the bugs your fuzzer has 138discovered, prioritizing the ones that seem most serious, and file issues for 139those bugs first. As those issues are resolved, then file a few more issues, and 140so on. 141 142### Further Reading 143 144Here are some more helpful resources to help your external fuzzing efforts 145succeed: 146 147* Blog post: [Responsible and Effective Bugfinding by John 148 Regehr](https://blog.regehr.org/archives/2037) 149 150## Wasmtime's Internal Fuzzing Infrastructure 151 152The Wasmtime project leverages extensive fuzzing for its safety and correctness 153assurances, and therefore already has a fairly large amount of fuzzing 154infrastructure. [Our fuzzers run continuously on 155OSS-Fuzz.](https://github.com/google/oss-fuzz/tree/master/projects/wasmtime) 156 157### Test Case Generators and Oracles 158 159Test case generators and oracles live in the `wasmtime-fuzzing` crate, located 160in the `crates/fuzzing` directory. 161 162A *test case generator* takes raw, unstructured input from a fuzzer and 163translates that into a test case. This might involve interpreting the raw input 164as "DNA" or pre-determined choices through a decision tree and using it to 165generate an in-memory data structure, or it might be a no-op where we interpret 166the raw bytes as if they were Wasm. 167 168An *oracle* takes a test case and determines whether we have a bug. For example, 169one of the simplest oracles is to take a Wasm binary as an input test case, 170validate and instantiate it, and (implicitly) check that no assertions failed or 171segfaults happened. A more complicated oracle might compare the result of 172executing a Wasm file with and without optimizations enabled, and make sure that 173the two executions are observably identical. 174 175Our test case generators and oracles strive to be fuzzer-agnostic: they can be 176reused with libFuzzer or AFL or any other fuzzing engine or driver. 177 178### libFuzzer and `cargo fuzz` Fuzz Targets 179 180We combine a test case generator and one more oracles into a *fuzz 181target*. Because the target needs to pipe the raw input from a fuzzer into the 182test case generator, it is specific to a particular fuzzer. This is generally 183fine, since they're only a couple of lines of glue code. 184 185Currently, all of our fuzz targets are written for 186[libFuzzer](https://www.llvm.org/docs/LibFuzzer.html) and [`cargo 187fuzz`](https://rust-fuzz.github.io/book/cargo-fuzz.html). They are defined in 188the `fuzz` subdirectory. 189 190See 191[`fuzz/README.md`](https://github.com/bytecodealliance/wasmtime/blob/main/fuzz/README.md) 192for details on how to run these fuzz targets and set up a corpus of seed inputs. 193