1# Using Pulley 2 3On architectures such as x86\_64 or aarch64 Wasmtime will by default use the 4Cranelift compiler to translate WebAssembly to native machine code and execute 5it. Cranelift does not support all architectures, however, for example i686 6(32-bit Intel machines) is not supported at this time. To help execute 7WebAssembly on these architectures Wasmtime comes with an interpreter called 8Pulley. 9 10Pulley is a bytecode interpreter originally proposed [in an RFC][rfc] which is 11intended to primarily be portable. Pulley is a loose backronym for "Portable, 12Universal, Low-Level Execution strategY" but mostly just a theme on 13machines/tools (Cranelift, Winch, Pulley, ...). Pulley is a distinct target and 14execution environment for Wasmtime. 15 16## Enabling Pulley 17 18The Pulley interpreter is enabled via one of two means: 19 201. On architectures which have Cranelift support, Pulley must be enabled via the 21 `pulley` crate feature of the `wasmtime` crate. This feature is otherwise 22 off-by-default. 23 242. On architectures which do NOT have Cranelift support, Pulley is already 25 enabled by default. This means that Wasmtime can execute WebAssembly by 26 default on any platform, it'll just be faster on Cranelift-supported 27 platforms. 28 29For platforms in category (2) there is no opt-in necessary to execute Pulley as 30that's already the default target. Platforms in category (1), such as 31`x86_64-unknown-linux-gnu`, may still want to execute Pulley to run tests, 32evaluate the implementation, benchmark, etc. 33 34To force execution of Pulley on any platform the `pulley` crate feature of 35the `wasmtime` crate must be enabled in addition to configuring a target. 36Specifying a target is done with the `--target` CLI option to the `wasmtime` 37executable, the [`Config::target`] method in Rust, or the 38[`wasmtime_config_target_set`] C API. The target string for pulley must be one 39of: 40 41[`Config::target`]: https://docs.rs/wasmtime/latest/wasmtime/struct.Config.html#method.target 42[`wasmtime_config_target_set`]: https://docs.wasmtime.dev/c-api/config_8h.html#ae68a2737ba1680e75cddb6ede08d682a 43 44* `pulley32` - for 32-bit little-endian hosts 45* `pulley32be` - for 32-bit big-endian hosts 46* `pulley64` - for 64-bit little-endian hosts 47* `pulley64be` - for 64-bit big-endian hosts 48 49The Pulley target string must match the environment that the Pulley Bytecode 50will be executing in. Some examples of Pulley targets are: 51 52| Host target | Pulley target | 53|----------------------------|---------------| 54| `x86_64-unknown-linux-gnu` | `pulley64` | 55| `i686-unknown-linux-gnu` | `pulley32` | 56| `s390x-unknown-linux-gnu` | `pulley64be` | 57 58Wasmtime will return an error trying to load bytecode compiled for the wrong 59Pulley target. When Pulley is the default target for a particular host then the 60correct Pulley target will be selected automatically. Specifying the Pulley 61target may still be necessary when cross-compiling from one platform to another, 62however. 63 64## Using Pulley 65 66Using Pulley in Wasmtime requires no further configuration beyond specifying the 67target for Pulley. Once that is done all of the Wasmtime crate's Rust APIs or C 68API work as usual. For example when specifying `wasmtime run --target pulley64` 69on the CLI this will execute all WebAssembly in the interpreter rather than via 70Cranelift. 71 72Pulley at this time has the same feature parity for WebAssembly as Cranelift 73does. This means that all WebAssembly proposals and features supported by 74Wasmtime are supported by Pulley. 75 76If you notice anything awry, however, please feel free to file an issue. 77 78## Impact of using Pulley 79 80Pulley is an interpreter for its own bytecode format. While the design of Pulley 81is optimized for speed you should still expect a ~10x order-of-magnitude 82slowdown relative to native code or Cranelift. This means that Pulley is likely 83not suitable for compute-intensive tasks that must run in as little time as 84possible. 85 86The primary goal of Pulley is to enable using and embedding Wasmtime across a 87variety of platforms simultaneously. The same API/interface is used to interact 88with the runtime and loading WebAssembly module regardless of the host 89architecture. 90 91Pulley bytecode is produced by the Cranelift compiler today in a similar manner 92to native platforms. Pulley is not designed for quickly loading WebAssembly 93modules as Cranelift is an optimizing compiler. Compiling WebAssembly to Pulley 94bytecode should be expected to take about the same time as compiling to native 95platforms. 96 97## High-level Design of Pulley 98 99This section is not necessary for users of Pulley but for those interested this 100is a description of the high-level design of Pulley. The Pulley virtual machine 101consists of: 102 103* 32 "X" integer registers each of which are 64-bits large. (`XReg`) 104* 32 "F" float registers each of which are 64-bits large. (`FReg`) 105* 32 "V" vector registers each of which are 128-bits large. (`VReg`) 106* A dynamically allocated "stack" on the host's heap. 107* A frame pointer register. 108* A link register to store the return address for the current function. 109 110This state lives in [`MachineState`] which is in turned stored in a [`Vm`]. 111Pulley's source code lives in `pulley/` in the Wasmtime repository. 112 113Pulley's bytecode is defined in `pulley/src/lib.rs` with a combination of the 114`for_each_op!` and `for_each_extended_op!` macros. Opcode numbers and opcode 115layout are defined by the structure of these macros. The macros are used to 116"derive" encoding/decoding/traits/etc used throughout the `pulley_interpreter` 117crate. 118 119Pulley opcodes are a single discriminator byte followed by any immediates. 120Immediates are not aligned and require unaligned loads/stores to work with them. 121Pulley has more than 256 opcodes, however, which is where "extended" opcodes 122come into play. The final Pulley opcode is reserved to indicate that an extended 123opcode is being used. Extended opcodes follow this initial discriminator with a 12416-bit integer which further indicates which extended opcode is being used. This 125design is intended to allow common operations to be encoded more compactly while 126less common operations can still be packed in effectively without limit. 127 128Pulley opcode assignment happens through the order of the `for_each_op!` macro 129which means that it's not portable across multiple versions of Wasmtime. 130 131The interpreter is an implementation of the [`OpVisitor`] and 132[`ExtendedOpVisitor`] traits. This is located at `pulley/src/interp.rs`. Notably 133this means that there's a method-per-opcode and is how the interpreter is 134implemented. 135 136The interpreter loop itself is implemented in one of two ways: 137 1381. A "match loop" which is a Rust `loop { ... }` which internally uses the 139 [`Decode`] trait on each opcode. This is not literally modeled as but 140 compiles down to something that looks like `loop { match .. { ... } }`. This 141 interpreter loop is located at `pulley/src/interp/match_loop.rs`. 142 1432. A "tail loop" were each opcode handler is a Rust function. Control flow 144 between opcodes continues with tail-calls and exiting the interpreter is done 145 by returning from the function. Tail calls are not available in stable Rust 146 so this interpreter loop is not used by default. It can be enabled, though, 147 with `RUSTFLAGS=--cfg=pulley_assume_llvm_makes_tail_calls` to rely on LLVM's 148 tail-call-optimization pass to implement the loop. 149 150The "match loop" is the default interpreter loop as it's portable and works on 151stable Rust. The "tail loop" is thought to probably perform better than the 152"match loop" but it's not available on stable Rust (`become` in Rust is an 153unfinished nightly feature at this time) or portable (tail-call-optimization 154doesn't happen the same in LLVM on all architectures). 155 156### Inspecting Pulley Bytecode 157 158Like when compiling to native the `wasmtime objdump` command can be used to 159inspect compiled bytecode: 160 161```sh 162$ wasmtime compile --target pulley64 foo.wat 163$ wasmtime objdump foo.cwasm --addresses --bytes 1640x000000: wasm[0]::function[20]: 165 0: 9f 10 00 08 00 push_frame_save 16, x19 166 5: 40 13 00 xmov x19, x0 167 8: 03 13 13 3f cb 89 00 call2 x19, x19, 0x89cb3f // target = 0x89cb47 168 f: 03 13 13 8c ab 84 00 call2 x19, x19, 0x84ab8c // target = 0x84ab9b 169 16: 03 13 13 5b 12 00 00 call2 x19, x19, 0x125b // target = 0x1271 170 1d: 03 13 13 9f 12 00 00 call2 x19, x19, 0x129f // target = 0x12bc 171 24: 03 13 13 e0 45 00 00 call2 x19, x19, 0x45e0 // target = 0x4604 172... 173``` 174 175### Profiling Pulley 176 177Profiling the Pulley interpreter can be done with native profiler such as `perf` 178but this has a few downsides: 179 180* When profiling the "match loop" it's not clear what machine code corresponds 181 to which Pulley opcode. Most of the time all the samples are just in the one 182 big "run" function. 183 184* When profiling with the "tail loop" you can see hot opcodes much more clearly, 185 but it can be difficult to understand why a particular opcode was chosen. 186 187It can sometimes be more beneficial to see time spent per Pulley opcode itself 188in the context of the all Pulley opcodes. In a similar manner as you can look at 189instruction-level profiling in `perf` it can be useful to look at opcode-level 190profiling of Pulley. 191 192Pulley has limited support for opcode-level profiling. This is off-by-default as 193it has a performance hit for the interpreter. To collect a profile with the 194`wasmtime` CLI you'll have to build from source and enable the `profile-pulley` 195feature: 196 197```sh 198$ cargo run --features profile-pulley --release run --profile pulley --target pulley64 foo.wat 199``` 200 201This will compile an optimized `wasmtime` executable with the `profile-pulley` 202Cargo feature enabled. The `--profile pulley` flag can then be passed to the 203`wasmtime` CLI to enable the profiler at runtime. 204 205The command will emit a `pulley-$pid.data` file which contains raw data about 206Pulley opcodes and samples taken. To view this file you can use: 207 208```sh 209$ cargo run -p pulley-interpreter --example profiler-html --all-features ./pulley-$pid.data 210``` 211 212This will load the `pulley-*.data` file, parse it, collate the results, and 213display the hottest functions. The hottest function is emitted last and 214instructions are annotated with the `%` of samples taken that were executing at 215that instruction. 216 217Some more information can be found in [the PR that implemented Pulley profiling 218support][profile-pr] 219 220[`OpVisitor`]: https://docs.rs/pulley-interpreter/latest/pulley_interpreter/decode/trait.OpVisitor.html 221[`MachineState`]: https://docs.rs/pulley-interpreter/latest/pulley_interpreter/interp/struct.MachineState.html 222[`Vm`]: https://docs.rs/pulley-interpreter/latest/pulley_interpreter/interp/struct.Vm.html 223[rfc]: https://github.com/bytecodealliance/rfcs/blob/main/accepted/pulley.md 224[`ExtendedOpVisitor`]: https://docs.rs/pulley-interpreter/latest/pulley_interpreter/decode/trait.ExtendedOpVisitor.html 225[`Decode`]: https://docs.rs/pulley-interpreter/latest/pulley_interpreter/decode/trait.Decode.html 226[profile-pr]: https://github.com/bytecodealliance/wasmtime/pull/10034 227