Debugging: add builtin gdbstub component. (#12771)

* Debugging: add builtin gdbstub component.

This adds a debug component that makes use of the debug-main world
defined in #12756 and serves the gd

Debugging: add builtin gdbstub component. (#12771)

* Debugging: add builtin gdbstub component.

This adds a debug component that makes use of the debug-main world
defined in #12756 and serves the gdbstub protocol, with Wasm
extensions, compatible with LLDB.

This component is built and included inside the Wasmtime binary, and
is loaded using the lower-level `-D debugger=...` debug-main option;
the user doesn't need to specify the `.wasm` adapter
component. Instead, the user simply runs `wasmtime run -g <PORT>
program.wasm ...` and Wasmtime will load and prepare to run
`program.wasm` as the debuggee, waiting for a gdbstub connection on
the given TCP port before continuing.

The workflow is:

```
$ wasmtime run -g 1234 program.wasm
[ wasmtime starts and waits for connection ]

$ /opt/wasi-sdk/bin/lldb # use LLDB from wasi-sdk release 32 or later
(lldb) process connect --plugin wasm connect://localhost:1234
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0x40000000000001cc
-> 0x40000000000001cc: unreachable
0x40000000000001cd: end
0x40000000000001ce: local.get 0
0x40000000000001d0: call 13
(lldb) si
Process 1 stopped
* thread #1, stop reason = instruction step into
frame #0: 0x4000000000000184
-> 0x4000000000000184: block
0x4000000000000186: block
0x4000000000000188: global.get 1
0x400000000000018e: i32.const 3664
[ ... ]
```

This makes use of the `gdbstub` third-party crate, into which I've
upstreamed support for the Wasm extensions in daniel5151/gdbstub#188,
daniel5151/gdbstub#189, daniel5151/gdbstub#190, and
daniel5151/gdbstub#192. (I'll add vets as part of this PR.)

* cargo vets.

* Handle Trap events as well as breakpoints.

* Review feedback.

* Fix gdbstub artifact build to make it publishable (by disabling it when isolated crates are used).

* Review feedback.

* fix published-crates list

* For now, empty gdbstub data but no compile error when artifact crate is published.

* add some more Cargo metadata: version for artifact crate dep

show more ...

Debugging: add the debug-main world. (#12756)

* Debugging: add the debug-main world.

This PR "draws the rest of the owl" for the debug-main
world (bytecodealliance/rfcs#45). This includes a WIT wor

Debugging: add the debug-main world. (#12756)

* Debugging: add the debug-main world.

This PR "draws the rest of the owl" for the debug-main
world (bytecodealliance/rfcs#45). This includes a WIT world that hosts
debug components that have access to "host debug powers" via a
debugging API, and the ability to load such a debug-component and give
it control of the main program as a debuggee when using `wasmtime
run`.

The WIT is namespaced to `bytecodealliance:wasmtime` and is slightly
aspirational in places: for example, the host does not yet implement
injection of early return values or exception-throws. I intend to fill
out a series of TODO issues once this all lands to track followup
("post-MVP") work.

This PR does not include any debug components. I separately have a
gdbstub component, with which I tested and co-developed this host-side
implementation. My plan is to land it in a followup PR as a component
that will be embedded in/shipped with the Wasmtime CLI and available
under an easy-to-use CLI option. Once we have that gdbstub component,
we can also implement end-to-end integration tests that boot up LLDB
and run through an expected interaction. (Separately, those
integration tests will require a release of wasi-sdk to ship an LLDB
binary that we can use.) As such, there are no real tests in this PR:
interesting behaviors only really occur with a full end-to-end flow.

The integration with the CLI is a little awkward (we internally build
another `wasmtime run` command that invokes the debug component, and
tie it together with the debuggee via a special `invoke_debugger` API;
this seemed less bad than reworking all of the WASI setup to be more
reusable). Happy to take more ideas here.

* Review feedback.

* Review feedback.

* Review feedback: update vendor-wit.sh.

* Review feedback: -Ddebugger-arg= -> -Darg=.

* Review feedback.

* Review feedback.

* Review feedback: factor host.rs into several submodules.

* Review feedback: rename Debugger to Debuggee on host side.

* Review feedback: split inherit_stdin_stdout, and add corresponding options for the debug component.

* Review feedback.

* Review feedback.

* Add simple debug-component tests.

* Add wasm32-wasip2 target in a few places in CI

* Cargo vets for wstd dependency.

* Add wasm32-wasip2 in more places

* fix debug-component test dependence on componentization byte offsets

* Review feedback.

* Fix cancel-safety of EventFuture.

* Fix: Interrupted events should only occur after interrupt(), not on every epoch yield.

* Review feedback.

* Review feedback: strip down WASI imports in debugger world.

* fold debugger test component back into wasip1 + adapter test artifact compilation flow

show more ...

Refactor WASIp2 `wasi:http` implementation (#12748)

* Sequester WASIp2 in `wasmtime-wasi-http` to a module

This mirrors the `wasmtime-wasi` crate's organization where there's a
`p2` module and a `

Refactor WASIp2 `wasi:http` implementation (#12748)

* Sequester WASIp2 in `wasmtime-wasi-http` to a module

This mirrors the `wasmtime-wasi` crate's organization where there's a
`p2` module and a `p3` module at the top level.

* Refactor WASIp2 `wasi:http` implementation

This commit reorganizes and refactors the WASIp2 implementation of
`wasi:http` to look more like other `wasmtime-wasi`-style interfaces.
Specifically the old `WasiHttpImpl<T>` structure is removed in favor of
as `WasiHttpCtxView<'_>` type that is used to implement
bindgen-generated `Host` traits. This necessitated reorganizing the
methods of the previous `WasiHttpView` trait like so:

* The `WasiHttpView` trait is renamed to `WasiHttpHooks` to make space
for a new `WasiHttpView` which behaves like `WasiView`, for example.

* The `ctx` and `table` methods of `WasiHttpHooks` were removed since
they'll be fields in `WasiHttpCtxView`.

* Helper methods for WASIp2 were moved to methods on `WasiHttpCtxView`
instead of default methods on `WasiHttpHooks`.

With these changes in place the WASIp3 organization was also updated
slightly as well. Notably WASIp3 now contains a reference to the crate's
`WasiHttpCtx` structure (which has field limits for example). WASIp3's
previous `WasiHttpCtx` trait is now renamed to `WasiHttpHooks` as well.
This means that there are two `WasiHttpHooks` traits right now, one for
WASIp2 and one for WASIp3. In the future I would like to unify these two
but that will require some more work around the default `send_request`.

A final note here is that the `WasiHttpHooks` trait previously, and
continues to be, optional for embedders to implement. Default functions
are provided as `wasmtime_wasi_http::{p2, p3}::default_hooks`.
Additionally there's a `Default for &mut dyn WasiHttpHooks`
implementation, too.

With all that outlined: the motivation for this change is to bring the
WASIp2 and WASIp3 implementations of `wasi:http` closer together. This
is inspired by refactorings I was doing for #12674 to apply the same
header limitations for WASIp3 as is done for WASIp2. Prior to this
change there were a number of differences such as WASIp3 not having
`crate::WasiHttpCtx` around, WASIp2 having a different organization of
structures/borrows, etc. The goal is to bring the two implementations
closer in line with each other to make refactoring across them more
consistent and easier.

* Make `WasiHttp` in WASIp2 public

* Fix some conditional build

* Fix some doctests

* Fix configured build

* Fixup documentation

show more ...

Fix two security advisories. (#12652)

* Fix two security advisories.

This commit contains merged fixes for two security advisories in
Wasmtime:

* GHSA-852m-cvvp-9p4w
* GHSA-243v-98vx-264h

This in

Fix two security advisories. (#12652)

* Fix two security advisories.

This commit contains merged fixes for two security advisories in
Wasmtime:

* GHSA-852m-cvvp-9p4w
* GHSA-243v-98vx-264h

This introduces new knobs to Wasmtime to limit the scope of resources
that WASI implementations will allocate on behalf of guests. Unlike
backports to 41.0.x-and-prior these knobs all have default values which
are considered reasonable for hosts if they don't further tune them. The
following CLI knobs have been added:

* `-Smax-resources` - limits the total component-model resources a guest
can allocate in a table
* `-Shostcall-fuel` - a broad limit which enforces that at most this
amount of data will be copied from the guest to the host in any one
API call (e.g. `string` values can't be too big, `list<string>` can't
be quadratic, etc). This fuel is reset on each host function call.
* `-Smax-random-size` - the maximal size of the return value of the
`get-random-bytes` and `get-insecure-random-bytes` WASI functions.
* `-Smax-http-fields-size` - a limit on the size of `wasi:http` `fields`
values to avoid infinitely buffering data within the host.

The `http` crate has additionally been updated to avoid a panic when
adding too many headers to a `fields` object.

Co-authored-by: Mark Bundschuh <[email protected]>
Co-authored-by: Pat Hickey <[email protected]>
Co-authored-by: Joel Dice <[email protected]>

* CI fixes

* Run rustfmt
* Fix wasi-common build

* Fix tests on 32-bit

* Fix nightly test expectations

prtest:full

---------

Co-authored-by: Mark Bundschuh <[email protected]>
Co-authored-by: Pat Hickey <[email protected]>
Co-authored-by: Joel Dice <[email protected]>

show more ...

Migrate the Wasmtime CLI to `wasmtime::error` (#12295)

* Migrate wasmtime-cli to `wasmtime::error`

* migrate benches to `wasmtime::error` as well

* Remove new usage of anyhow that snuck in

Add `-Shttp,p3` support to `wasmtime run` (#11741)

Omitted from previous inclusions by accident due to how things
developed, but now it's there!

Closes #11736

Add a new `-Sp3` option to the CLI (#11423)

* Add a new `-Sp3` option to the CLI

This commit adds a new WASI-specific CLI option dubbed "p3" which
controls whether WASIp3 interfaces are added to th

Add a new `-Sp3` option to the CLI (#11423)

* Add a new `-Sp3` option to the CLI

This commit adds a new WASI-specific CLI option dubbed "p3" which
controls whether WASIp3 interfaces are added to the component linker and
whether WASIp3 exports are searched for. Integration here is only added
with `wasmtime run` and support for `wasmtime serve` will come once
wasi:http is migrated to this repository from the p3-prototyping
repository.

The `-Sp3` option is off-by-default at this time but this is structured
to enable it by default in the future. A test is added to ensure that
WASIp3 is usable and a basic "hello world" works.

Some internal refactoring was done to share more code between `run` and
`serve` with respect to adding WASI interfaces to a linker.

* Use a normal blocking test

* Fix warning

show more ...

Unify WASIp{2,3} context structures (#11370)

This removes `wasmtime_wasi::p{2,3}::{WasiCtx, WasiCtxBuilder,
WasiView}` in favor of only having `wasmtime_wasi::{WasiCtx,
WasiCtxBuilder, WasiView}` in

Unify WASIp{2,3} context structures (#11370)

This removes `wasmtime_wasi::p{2,3}::{WasiCtx, WasiCtxBuilder,
WasiView}` in favor of only having `wasmtime_wasi::{WasiCtx,
WasiCtxBuilder, WasiView}` instead. Conceptually these revisions of WASI
all provide the same functionality just with a different veneer that the
component model offers, so having only one way to configure host-side
behavior will make it easier to both organize implementations internally
(e.g. more sharing of code) as well as for embedders to configure (only
one context to create/manage).

show more ...

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtim

Update Wasmtime to the 2024 Rust Edition (#10806)

* Update Wasmtime to the 2024 Rust Edition

Now that our MSRV supports the 2024 edition it's possible to make this
switch. This commit moves Wasmtime to the 2024 Edition to keep
up-to-date with Rust idioms and access many of the edition features
exclusive to the 2024 edition.

prtest:full

* Reformat with the 2024 edition

show more ...

refactor: move wasip2 implementation to `wasmtime_wasi::p2` (#10073)

* refactor: move `wasmtime-wasi` p2-specific functionality to `p2`

Signed-off-by: Roman Volosatovs <[email protected]>

* d

refactor: move wasip2 implementation to `wasmtime_wasi::p2` (#10073)

* refactor: move `wasmtime-wasi` p2-specific functionality to `p2`

Signed-off-by: Roman Volosatovs <[email protected]>

* doc: move some of the p2 comments to top-level

Signed-off-by: Roman Volosatovs <[email protected]>

* chore: shorten `network` and `filesystem` module names

Signed-off-by: Roman Volosatovs <[email protected]>

---------

Signed-off-by: Roman Volosatovs <[email protected]>

show more ...

Add a `wasmtime objdump` subcommand (#10405)

This commit adds an `objdump` subcommand to the `wasmtime` CLI. Like all
other subcommands this can be disabled for a more minimal build of the
CLI as we

Add a `wasmtime objdump` subcommand (#10405)

This commit adds an `objdump` subcommand to the `wasmtime` CLI. Like all
other subcommands this can be disabled for a more minimal build of the
CLI as well. The purpose of this subcommand is to provide a
Wasmtime-specific spin on the venerable native `objdump` itself. Notably
this brings Wasmtime-specific knowledge for filtering functions, showing
Wasmtime metadata, etc.

This command is intended to look like `objdump` roughly but also has
configurable output with various flags and things that can be printed.
For now the main Wasmtime additions are showing the address map
section, stack map section, and trap section of a `*.cwasm` file.

This new subcommand replaces the infrastructure of the `disas` test
suite, and now that test suite uses `wasmtime objdump` to generate test
expectations. Additionally the subcommand replaces the Pulley `objdump`
example as a more full-featured objdump that also works natively with
Pulley.

The hope is that if we add more binary metadata in the future (such as
unwinding tables) that can be relatively easily added here for
exploration as well. Otherwise this is mostly just a developer
convenience for Wasmtime developers as well and hopefully doesn't cost
too much in maintenance burden.

Closes #10336

show more ...

Mention the filename when a module can't be opened (#10292)

This commit improves the error message of the `wasmtime` CLI when
running a file that can't be opened. This can happen for example when an

Mention the filename when a module can't be opened (#10292)

This commit improves the error message of the `wasmtime` CLI when
running a file that can't be opened. This can happen for example when an
invalid subcommand is passed such as `wasmtime foo` by accident.

show more ...

Add basic support for profiling Pulley (#10034)

* Add basic support for profiling Pulley

This commit adds basic support for profiling the Pulley interpreter.
This is partially achievable previously

Add basic support for profiling Pulley (#10034)

* Add basic support for profiling Pulley

This commit adds basic support for profiling the Pulley interpreter.
This is partially achievable previously through the use of native
profilers, but the downside of that approach is that you can find hot
instructions but it's not clear in what context the hot instructions are
being executed nor what functions are hot. The goal of this profiler is
to show pulley bytecode and time spent in bytecode itself to better
understand the shape of code around a hot instruction to identify new
macro opcodes for example.

The general structure of this new profiler is:

* There is a compile-time feature for Pulley which is off-by-default
where, when enabled, Pulley will record its current program counter
into an `AtomicUsize` before each instruction.

* When the CLI has `--profile pulley` Wasmtime will spawn a sampling
thread in the same process which will periodically read from this
`AtomicUsize` to record where the program is currently executing.

* The Pulley profiler additionally records all bytecode through the use
of the `ProfilingAgent` trait to ensure that the recording has access
to all bytecode as well.

* Samples are taken throughout the process and emitted to a
`pulley-$pid.data` file. This file is then interpreted and printed by
an "example" program `profiler-html.rs` in the `pulley/examples`
directory.

The end result is that hot functions of Pulley bytecode can be seen and
instructions are annotated with how frequently they were executed. This
enables finding hot loops and understanding more about the whole loop,
bytecodes that were selected, and such.

* Add missing source file

* Check the profile-pulley feature in CI

* Miscellaneous fixes for CI

* Fix type-checking of `become` on nightly Rust

* Fix more misc CI issues

* Fix dispatch in tail loop

* Update test expectations

* Review comments

* Fix a feature combo

show more ...

Remove `PartialEq` implementations from CLI flags (#9628)

These are historical artifacts from when the pre-Wasmtime-13 CLI was
supported but they're no longer needed nowadays.

Add `Module::deserialize_open_file` (#9571)

* Add `Module::deserialize_open_file`

Add an API to deserialize a `Module` from an already opened file. This
can be useful in situations where `wasmtime`

Add `Module::deserialize_open_file` (#9571)

* Add `Module::deserialize_open_file`

Add an API to deserialize a `Module` from an already opened file. This
can be useful in situations where `wasmtime` is running in a sandboxed
environment with limited access to the file system. Then another process
can handle opening the files and passing them to `wasmtime`.

* windows fix 1

prtest:full

* windows fix 2

* clippy

* windows test fix

* windows test fix 2

* windows prelude and doc comment

* windows_sys import in test

* typo

show more ...

Add function to downcast a stream error to a network-related error. Similar to the existing filesystem-error-code and http-error-code functions. (#9276)

Configure WIT feature gates at runtime & implement wasi-cli `exit-with-code` (#9381)

* Remove `features` configuration from component::bindgen! and always emit all unstable members. These features w

Configure WIT feature gates at runtime & implement wasi-cli `exit-with-code` (#9381)

* Remove `features` configuration from component::bindgen! and always emit all unstable members. These features will be gated at runtime.

* Implement wasi-cli's unstable `exit-with-code`

* Add codegen test for unstable features

* Add LinkOptions type and add a parameter to all add_to_linker functions in for worlds/interfaces that use any unstable feature.

* More descriptive test feature names.

* Generate feature gate `if` checks

* Expose `cli-exit-with-code` as CLI option

* Generate bespoke option types per interface and world.

* Add unit test

* Remove exit code restriction on Windows

* Add cli_exit_with_code test

* Use BTreeSet to generate the options in consistent order

* Change wasmtime-wasi's `add_to_linker_(a)sync` signature back to how it was and add new variants that take the option parameters.

* Lift Windows exit code restriction in tests

* Lift Windows exit code restriction

show more ...

Make the Wasmtime CLI use async (#9184)

* Make the Wasmtime CLI use async

This means that interrupting a running Wasm program will now work
correctly, even when the program is blocked on I/O or wai

Make the Wasmtime CLI use async (#9184)

* Make the Wasmtime CLI use async

This means that interrupting a running Wasm program will now work
correctly, even when the program is blocked on I/O or waiting on a timeout or
some such.

This also involved making `wasi-threads` async-compatible.

Co-Authored-By: Alex Crichton <[email protected]>

* rustfmt

* Make `run` command enable the `tokio` feature

* Add a test for CLI, timeouts, and sleeping forever

* Fix warning

---------

Co-authored-by: Alex Crichton <[email protected]>

show more ...

Refactor use of `CodeBuilder` on the CLI (#9125)

* Refactor use of `CodeBuilder` on the CLI

This commit refactors `wasmtime run` and `wasmtime compile` to
unconditionally use `CodeBuilder` internal

Refactor use of `CodeBuilder` on the CLI (#9125)

* Refactor use of `CodeBuilder` on the CLI

This commit refactors `wasmtime run` and `wasmtime compile` to
unconditionally use `CodeBuilder` internally. This will in theory help
out in the future if more debug-related options are added to
`CodeBuilder` for example. This refactoring required some changes to
`CodeBuilder` to be able to support a query about whether the internal
bytes were a component or a module. The text format is now converted to
binary immediately when supplied rather than during the compilation
phase. This in turn required some API changes to make the selection of
supporting the text format a compile-time choice of method rather than a
runtime value.

* Fix compile

* Fix no-cranelift build of CLI

show more ...

style: simplify string formatting (#9047)

* style: simplify string formatting

* fix: formatting in `benches/call.rs`

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime

Turn the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module (#8501)

* Expose `wasmtime-runtime` as `crate::runtime::vm` internally for the `wasmtime` crate

* Rewrite uses of `wasmtime_runtime` to `crate::runtime::vm`

* Remove dep on `wasmtime-runtime` from `wasmtime-cli`

* Move the `wasmtime-runtime` crate into the `wasmtime::runtime::vm` module

* Update labeler for merged crates

* Fix `publish verify`

prtest:full

show more ...

Support dirs/env vars in `wasmtime serve` (#8279)

* More flags like `--dir` and `--env` are moved into `RunCommon` to be
shared between `wasmtime serve` and `wasmtime run`, meaning that the
`ser

Support dirs/env vars in `wasmtime serve` (#8279)

* More flags like `--dir` and `--env` are moved into `RunCommon` to be
shared between `wasmtime serve` and `wasmtime run`, meaning that the
`serve` command can now configure environment variables.

* A small test has been added as well as infrastructure for running
tests with `wasmtime serve` itself. Previously there were no tests
that executed `wasmtime serve`.

* The `test_programs` crate had a small refactoring to avoid
double-generation of http bindings.

show more ...

Add a `CodeBuilder` type to the `wasmtime` crate (#8181)

* Add a `ModuleBuilder` type to the `wasmtime` crate

This commit is extracted from #8055 and adds a new `ModuleBuilder` type
which is intend

Add a `CodeBuilder` type to the `wasmtime` crate (#8181)

* Add a `ModuleBuilder` type to the `wasmtime` crate

This commit is extracted from #8055 and adds a new `ModuleBuilder` type
which is intended to be able to further configure compilation beyond
what the constructors of `Module` already provide. For example in #8055
knobs will be added to process `*.dwp` files for more debuginfo
processing.

Co-authored-by: yowl00 <[email protected]>

* Fix build

* Review feedback

* Rename ModuleBuilder to CodeBuilder

* Fix doc errors

---------

Co-authored-by: yowl00 <[email protected]>

show more ...

Enable compiling the Wasmtime CLI to Wasm (#7980)

* Enable compiling the Wasmtime CLI to Wasm

While not the most useful thing to do in the world it's kind of neat to
play around with. This builds o

Enable compiling the Wasmtime CLI to Wasm (#7980)

* Enable compiling the Wasmtime CLI to Wasm

While not the most useful thing to do in the world it's kind of neat to
play around with. This builds on the previous work to exclude the
runtime from the `wasmtime` crate so it's now possible to compile the
Wasmtime CLI's `compile` command, and only the `compile` command, to
wasm itself. This means you can run Wasmtime in Wasmtime!

* Fix warning on wasm

* Fix some feature combos

show more ...

Enable the component model by default (#7821)

* Enable the component model by default

This commit enables the component model by default in the embedding API
and the CLI. This means that an opt-in

Enable the component model by default (#7821)

* Enable the component model by default

This commit enables the component model by default in the embedding API
and the CLI. This means that an opt-in of `-W component-model` is no
longer required and additionally `.wasm_component_model(true)` is no
longer required. Note that this won't impact existing embeddings since
the component model feature doesn't do much less `wasmtime::component`
is used, and if that's being used this is probably good news for you.

* Fix non-component model build

show more ...