1# Building a Minimal Wasmtime embedding
2
3Wasmtime embeddings may wish to optimize for binary size and runtime footprint
4to fit on a small system. This documentation is intended to guide some features
5of Wasmtime and how to best produce a minimal build of Wasmtime.
6
7## Building a minimal CLI
8
9> *Note*: the exact numbers in this section were last updated on 2023-10-18 on a
10> macOS aarch64 host. For up-to-date numbers consult the artifacts in the [`dev`
11> release of Wasmtime][dev] where the `wasmtime-min` executable represents the
12> culmination of these steps.
13
14[dev]: https://github.com/bytecodealliance/wasmtime/releases/tag/dev
15
16Many Wasmtime embeddings go through the `wasmtime` crate as opposed to the
17`wasmtime` CLI executable, but to start out let's take a look at minimizing the
18command line executable. By default the wasmtime command line executable is
19relatively large:
20
21```shell
22$ cargo build
23$ ls -l ./target/debug/wasmtime
24-rwxr-xr-x@ 1 root  root    140M Oct 18 08:33 target/debug/wasmtime
25```
26
27The easiest size optimization is to compile with optimizations. This will strip
28lots of dead code and additionally generate much less debug information by
29default
30
31```shell
32$ cargo build --release
33$ ls -l ./target/release/wasmtime
34-rwxr-xr-x@ 1 root  root     33M Oct 18 08:34 target/release/wasmtime
35```
36
37Much better, but still relatively large! The next thing that can be done is to
38disable the default features of the `wasmtime-cli` crate. This will remove all
39optional functionality from the crate and strip it down to the bare bones
40functionality.
41
42```shell
43$ cargo build --release --no-default-features
44$ ls -l ./target/release/wasmtime
45-rwxr-xr-x@ 1 root  root    6.7M Oct 18 08:37 target/release/wasmtime
46```
47
48Note that this executable is stripped to the bare minimum of functionality which
49notably means it does not have a compiler for WebAssembly files. This means that
50`wasmtime compile` is no longer supported meaning that `*.cwasm` files must be
51fed to `wasmtime run` to execute files. Additionally error messages will be
52worse in this mode as less contextual information is provided.
53
54The final Wasmtime-specific optimization you can apply is to disable logging
55statements. Wasmtime and its dependencies make use of the [`log`
56crate](https://docs.rs/log) and [`tracing` crate](https://docs.rs/tracing) for
57debugging and diagnosing. For a minimal build this isn't needed though so this
58can all be disabled through Cargo features to shave off a small amount of code.
59Note that for custom embeddings you'd need to replicate the `disable-logging`
60feature which sets the `max_level_off` feature for the `log` and `tracing`
61crate.
62
63```shell
64$ cargo build --release --no-default-features --features disable-logging
65$ ls -l ./target/release/wasmtime
66-rwxr-xr-x@ 1 root  root    6.7M Oct 18 08:37 target/release/wasmtime
67```
68
69At this point the next line of tricks to apply to minimize binary size are
70[general tricks-of-the-trade for Rust
71programs](https://github.com/johnthagen/min-sized-rust) and are no longer
72specific to Wasmtime. For example the first thing that can be done is to
73optimize for size rather than speed via rustc's `s` optimization level.
74This uses Cargo's [environment-variable based configuration][cargo-env-config]
75via the `CARGO_PROFILE_RELEASE_OPT_LEVEL=s` environment variable to configure
76this.
77
78[cargo-env-config]: https://doc.rust-lang.org/cargo/reference/config.html#profile
79
80```shell
81$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
82$ cargo build --release --no-default-features --features disable-logging
83$ ls -l ./target/release/wasmtime
84-rwxr-xr-x@ 1 root  root    6.8M Oct 18 08:40 target/release/wasmtime
85```
86
87Note that the size has increased here slightly instead of going down. Optimizing
88for speed-vs-size can affect a number of heuristics in LLVM so it's best to test
89out locally what's best for your embedding. Further examples below continue to
90pass this flag since by the end it will produce a smaller binary than the
91default optimization level of "3" for release mode. You may wish to also try an
92optimization level of "2" and see which produces a smaller build for you.
93
94After optimizations levels the next compilation setting to configure is
95Rust's "panic=abort" mode where panics translate to process aborts rather than
96unwinding. This removes landing pads from code as well as unwind tables from the
97executable.
98
99```shell
100$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
101$ export CARGO_PROFILE_RELEASE_PANIC=abort
102$ cargo build --release --no-default-features --features disable-logging
103$ ls -l ./target/release/wasmtime
104-rwxr-xr-x@ 1 root  root    5.0M Oct 18 08:40 target/release/wasmtime
105```
106
107Next, if the compile time hit is acceptable, LTO can be enabled to provide
108deeper opportunities for compiler optimizations to remove dead code and
109deduplicate. Do note that this will take a significantly longer amount of time
110to compile than previously. Here LTO is configured with
111`CARGO_PROFILE_RELEASE_LTO=true`.
112
113```shell
114$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
115$ export CARGO_PROFILE_RELEASE_PANIC=abort
116$ export CARGO_PROFILE_RELEASE_LTO=true
117$ cargo build --release --no-default-features --features disable-logging
118$ ls -l ./target/release/wasmtime
119-rwxr-xr-x@ 1 root  root    3.3M Oct 18 08:42 target/release/wasmtime
120```
121
122Similar to LTO above rustc can be further instructed to place all crates into
123their own single object file instead of multiple by default. This again
124increases compile times. Here that's done with
125`CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1`.
126
127```shell
128$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
129$ export CARGO_PROFILE_RELEASE_PANIC=abort
130$ export CARGO_PROFILE_RELEASE_LTO=true
131$ export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1
132$ cargo build --release --no-default-features --features disable-logging
133$ ls -l ./target/release/wasmtime
134-rwxr-xr-x@ 1 root  root    3.3M Oct 18 08:43 target/release/wasmtime
135```
136
137Note that with LTO using a single codegen unit may only have marginal benefit.
138If not using LTO, however, a single codegen unit will likely provide benefit
139over the default 16 codegen units.
140
141One final flag before getting to nightly features is to strip debug information
142from the standard library. In `--release` mode Cargo by default doesn't generate
143debug information for local crates, but the Rust standard library may have debug
144information still included with it. This is configured via
145`CARGO_PROFILE_RELEASE_STRIP=debuginfo`
146
147```shell
148$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
149$ export CARGO_PROFILE_RELEASE_PANIC=abort
150$ export CARGO_PROFILE_RELEASE_LTO=true
151$ export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1
152$ export CARGO_PROFILE_RELEASE_STRIP=debuginfo
153$ cargo build --release --no-default-features --features disable-logging
154$ ls -l ./target/release/wasmtime
155-rwxr-xr-x@ 1 root  root    2.4M Oct 18 08:44 target/release/wasmtime
156```
157
158Next, if your use case allows it, the Nightly Rust toolchain provides a number
159of other options to minimize the size of binaries. Note the usage of `+nightly` here
160to the `cargo` command to use a Nightly toolchain (assuming your local toolchain
161is installed with rustup). Also note that due to the nature of nightly the exact
162flags here may not work in the future. Please open an issue with Wasmtime if
163these commands don't work and we'll update the documentation.
164
165The first nightly feature we can leverage is to remove filename and line number
166information in panics with `-Zlocation-detail=none`
167
168```shell
169$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
170$ export CARGO_PROFILE_RELEASE_PANIC=abort
171$ export CARGO_PROFILE_RELEASE_LTO=true
172$ export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1
173$ export CARGO_PROFILE_RELEASE_STRIP=debuginfo
174$ export RUSTFLAGS="-Zlocation-detail=none"
175$ cargo +nightly build --release --no-default-features --features disable-logging
176$ ls -l ./target/release/wasmtime
177-rwxr-xr-x@ 1 root  root    2.4M Oct 18 08:43 target/release/wasmtime
178```
179
180Further along the line of nightly features the next optimization will recompile
181the standard library without unwinding information, trimming out a bit more from
182the standard library. This uses the `-Zbuild-std` flag to Cargo. Note that this
183additionally requires `--target` as well which will need to be configured for
184your particular platform.
185
186```shell
187$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
188$ export CARGO_PROFILE_RELEASE_PANIC=abort
189$ export CARGO_PROFILE_RELEASE_LTO=true
190$ export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1
191$ export CARGO_PROFILE_RELEASE_STRIP=debuginfo
192$ export RUSTFLAGS="-Zlocation-detail=none"
193$ cargo +nightly build --release --no-default-features --features disable-logging \
194    -Z build-std=std,panic_abort --target aarch64-apple-darwin
195$ ls -l ./target/aarch64-apple-darwin/release/wasmtime
196-rwxr-xr-x@ 1 root  root    2.3M Oct 18 09:39 target/aarch64-apple-darwin/release/wasmtime
197```
198
199Next the Rust standard library has some optional features in addition to
200Wasmtime, such as printing of backtraces. This may not be required in minimal
201environments so the features of the standard library can be disabled with the
202`-Zbuild-std-features=` flag which configures the set of enabled features to be
203empty.
204
205```shell
206$ export CARGO_PROFILE_RELEASE_OPT_LEVEL=s
207$ export CARGO_PROFILE_RELEASE_PANIC=abort
208$ export CARGO_PROFILE_RELEASE_LTO=true
209$ export CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1
210$ export CARGO_PROFILE_RELEASE_STRIP=debuginfo
211$ export RUSTFLAGS="-Zlocation-detail=none"
212$ cargo +nightly build --release --no-default-features --features disable-logging \
213    -Z build-std=std,panic_abort --target aarch64-apple-darwin \
214    -Z build-std-features=
215$ ls -l ./target/aarch64-apple-darwin/release/wasmtime
216-rwxr-xr-x@ 1 root  root    2.1M Oct 18 09:39 target/aarch64-apple-darwin/release/wasmtime
217```
218
219## Minimizing further
220
221Above shows an example of taking the default `cargo build` result of 130M down
222to a 2.1M binary for the `wasmtime` executable. Similar steps can be done to
223reduce the size of the C API binary artifact as well which currently produces a
224~2.8M dynamic library. This is currently the smallest size with the source code
225as-is, but there are more size reductions which haven't been implemented yet.
226
227This is a listing of some example sources of binary size. Some sources of binary
228size may not apply to custom embeddings since, for example, your custom
229embedding might already not use WASI and might already not be included.
230
231* WASI in the Wasmtime CLI - currently the CLI includes all of WASI. This
232  includes two separate implementations of WASI - one for preview2 and one for
233  preview1. This accounts for 1M+ of space which is a significant chunk of the
234  remaining 2.1M.  While removing just preview2 or preview1 would be easy enough
235  with a Cargo feature, the resulting executable wouldn't be able to do
236  anything. Something like a [plugin feature for the
237  CLI](https://github.com/bytecodealliance/wasmtime/issues/7348), however, would
238  enable removing WASI while still being a usable executable.
239
240* Argument parsing in the Wasmtime CLI - as a command line executable `wasmtime`
241  contains parsing of command line arguments which currently uses the `clap`
242  crate. This contributes ~200k of binary size to the final executable which
243  would likely not be present in a custom embedding of Wasmtime. While this
244  can't be removed from Wasmtime it's something to consider when evaluating the
245  size of CI artifacts.
246
247* Cranelift in the C API - one of the features of Wasmtime is the ability to
248  have a runtime without Cranelift that only supports precompiled (AOT) wasm
249  modules. It's [not possible to build the C API without
250  Cranelift](https://github.com/bytecodealliance/wasmtime/issues/7349) though
251  because defining host functions requires Cranelift at this time to emit some
252  stubs.  This means that the C API is significantly larger than a custom Rust
253  embedding which doesn't suffer from the same restriction. This means that
254  while it's still possible to build an embedding of Wasmtime which doesn't have
255  Cranelift it's not easy to see what it might look like size-wise from
256  looking at the C API artifacts.
257
258* Formatting strings in Wasmtime - Wasmtime makes extensive use of formatting
259  strings for error messages and other purposes throughout the implementation.
260  Most of this is intended for debugging and understanding more when something
261  goes wrong, but much of this is not necessary for a truly minimal embedding.
262  In theory much of this could be conditionally compiled out of the Wasmtime
263  project to produce a smaller executable. Just how much of the final binary
264  size is accounted for by formatting string is unknown, but it's well known in
265  Rust that `std::fmt` is not the slimmest of modules.
266
267* Cranelift vs Winch - the "min" builds on CI try to exclude Cranelift from
268  their binary footprint (e.g. the CLI excludes it) but this comes at a cost of
269  the final executable not supporting compilation of wasm modules. If this is
270  required then no effort has yet been put into minimizing the code size of
271  Cranelift itself. One possible tradeoff that can be made though is to choose
272  between the Winch baseline compiler vs Cranelift. Winch should be much smaller
273  from a compiled footprint point of view while not sacrificing everything in
274  terms of performance. Note though that Winch is still under development.
275
276Above are some future avenues to take in terms of reducing the binary size of
277Wasmtime and various tradeoffs that can be made. The Wasmtime project is eager
278to hear embedder use cases/profiles if Wasmtime is not suitable for binary size
279reasons today. Please feel free to [open an
280issue](https://github.com/bytecodealliance/wasmtime/issues/new) and let us know
281and we'd be happy to discuss more how best to handle a particular use case.
282