Debugging: add debugger support for `wasmtime serve`. (#12859)

This adopts a simple solution to #12776: it takes the "instance reuse"
paradigm to the extreme, instantiating exactly one instance and

Debugging: add debugger support for `wasmtime serve`. (#12859)

This adopts a simple solution to #12776: it takes the "instance reuse"
paradigm to the extreme, instantiating exactly one instance and
serializing all requests into that one instance. This allows the
debugger component to operate on one `Store`, setting breakpoint state
and presenting its execution to the attached debugger as a single
program execution and minimizing impedance mismatches.

This also adds an integration test that runs an existing wasi-http
test component under the debugger.

show more ...

Several fixes to debugging infrastructure: component vs. module PCs and gdbstub wasm module names. (#12901)

* Debugging: fix module-relative vs component-relative PCs and unique library names.

Two

Several fixes to debugging infrastructure: component vs. module PCs and gdbstub wasm module names. (#12901)

* Debugging: fix module-relative vs component-relative PCs and unique library names.

Two bugfixes for guest debugging with components:

1. Convert component-relative source locations to module-relative PCs
in the frame table. The guest-debug API presents a core-Wasm view
where components are deconstructed into individual modules, so all
PCs must be module-relative. This adds a `wasm_module_offset` field
to `ModuleTranslation` and `FuncEnvironment`, set during component
translation, and subtracts it in `debug_tags()`.

2. Give unique names to "library" entries in the gdbstub XML response.
LLDB's DynamicLoader deduplicates by name, so using "wasm" for all
modules caused only the first to be loaded.

* Debugging: add ModulePC and ComponentPC newtypes for Wasm PC offsets.

Introduce `ModulePC` (module-relative) and `ComponentPC`
(component-relative) newtype wrappers around u32 Wasm bytecode
offsets. These replace raw u32 values throughout the frame table,
breakpoint, and debug systems to prevent confusion between the two
offset spaces.

* Debugging: add regression test for component module-relative PCs.

show more ...

Use the same `WasiTlsView` types for both p2 & p3 (#12896)

feat(p3): implement wasi:tls (#12834)

* Split off p2-specific bits into submodule

* Vendor the 0.3.0-draft WIT files

* Host traits scaffolding

* Rename p2 test

* Work around bug in `tokio-native

feat(p3): implement wasi:tls (#12834)

* Split off p2-specific bits into submodule

* Vendor the 0.3.0-draft WIT files

* Host traits scaffolding

* Rename p2 test

* Work around bug in `tokio-native-tls`

* Create error type

* Implement p3

Co-authored-by: Roman Volosatovs <[email protected]>

* Fix test on Windows' SChannel

Same reason as described in https://github.com/bytecodealliance/wasmtime/pull/11064

* Satisfy clippy

* Fix typo

---------

Co-authored-by: Roman Volosatovs <[email protected]>

show more ...

Debugging: add integration test with LLDB and some minor tweaks. (#12856)

This PR adds:

- An integration-test that runs LLDB against the Wasmtime CLI to
verify basic debugging functionality, simi

Debugging: add integration test with LLDB and some minor tweaks. (#12856)

This PR adds:

- An integration-test that runs LLDB against the Wasmtime CLI to
verify basic debugging functionality, similar to the existing
native-debug tests.

- A CI job that runs the above in CI.

- Some minor tweaks to the gdbstub debugger design:
- Rather than the initial single-step to get to the first Wasm
instruction where module(s) will be instantiated into the store
and visible to the debugger, we pre-register modules with the
store eagerly. This avoids the slightly hacky flow and also is a
preparation step for `wasmtime serve` debugging, where we can't
single-step into execution eagerly (because execution doesn't
start at all until an HTTP request arrives).
- Add a separate message-printing path for "debugger info messages",
allowing us to print the "debugger is listening on <PORT>" message
without inheriting stderr for the whole debugger component
environment. This message is necessary for the above integration
test (it parses the message to determine when the debuggee is ready).

show more ...

Debugging: add builtin gdbstub component. (#12771)

* Debugging: add builtin gdbstub component.

This adds a debug component that makes use of the debug-main world
defined in #12756 and serves the gd

Debugging: add builtin gdbstub component. (#12771)

* Debugging: add builtin gdbstub component.

This adds a debug component that makes use of the debug-main world
defined in #12756 and serves the gdbstub protocol, with Wasm
extensions, compatible with LLDB.

This component is built and included inside the Wasmtime binary, and
is loaded using the lower-level `-D debugger=...` debug-main option;
the user doesn't need to specify the `.wasm` adapter
component. Instead, the user simply runs `wasmtime run -g <PORT>
program.wasm ...` and Wasmtime will load and prepare to run
`program.wasm` as the debuggee, waiting for a gdbstub connection on
the given TCP port before continuing.

The workflow is:

```
$ wasmtime run -g 1234 program.wasm
[ wasmtime starts and waits for connection ]

$ /opt/wasi-sdk/bin/lldb # use LLDB from wasi-sdk release 32 or later
(lldb) process connect --plugin wasm connect://localhost:1234
Process 1 stopped
* thread #1, stop reason = signal SIGTRAP
frame #0: 0x40000000000001cc
-> 0x40000000000001cc: unreachable
0x40000000000001cd: end
0x40000000000001ce: local.get 0
0x40000000000001d0: call 13
(lldb) si
Process 1 stopped
* thread #1, stop reason = instruction step into
frame #0: 0x4000000000000184
-> 0x4000000000000184: block
0x4000000000000186: block
0x4000000000000188: global.get 1
0x400000000000018e: i32.const 3664
[ ... ]
```

This makes use of the `gdbstub` third-party crate, into which I've
upstreamed support for the Wasm extensions in daniel5151/gdbstub#188,
daniel5151/gdbstub#189, daniel5151/gdbstub#190, and
daniel5151/gdbstub#192. (I'll add vets as part of this PR.)

* cargo vets.

* Handle Trap events as well as breakpoints.

* Review feedback.

* Fix gdbstub artifact build to make it publishable (by disabling it when isolated crates are used).

* Review feedback.

* fix published-crates list

* For now, empty gdbstub data but no compile error when artifact crate is published.

* add some more Cargo metadata: version for artifact crate dep

show more ...

fix: correct various typos (#12807)

Signed-off-by: Ho Kim <[email protected]>

Debugging: add the debug-main world. (#12756)

* Debugging: add the debug-main world.

This PR "draws the rest of the owl" for the debug-main
world (bytecodealliance/rfcs#45). This includes a WIT wor

Debugging: add the debug-main world. (#12756)

* Debugging: add the debug-main world.

This PR "draws the rest of the owl" for the debug-main
world (bytecodealliance/rfcs#45). This includes a WIT world that hosts
debug components that have access to "host debug powers" via a
debugging API, and the ability to load such a debug-component and give
it control of the main program as a debuggee when using `wasmtime
run`.

The WIT is namespaced to `bytecodealliance:wasmtime` and is slightly
aspirational in places: for example, the host does not yet implement
injection of early return values or exception-throws. I intend to fill
out a series of TODO issues once this all lands to track followup
("post-MVP") work.

This PR does not include any debug components. I separately have a
gdbstub component, with which I tested and co-developed this host-side
implementation. My plan is to land it in a followup PR as a component
that will be embedded in/shipped with the Wasmtime CLI and available
under an easy-to-use CLI option. Once we have that gdbstub component,
we can also implement end-to-end integration tests that boot up LLDB
and run through an expected interaction. (Separately, those
integration tests will require a release of wasi-sdk to ship an LLDB
binary that we can use.) As such, there are no real tests in this PR:
interesting behaviors only really occur with a full end-to-end flow.

The integration with the CLI is a little awkward (we internally build
another `wasmtime run` command that invokes the debug component, and
tie it together with the debuggee via a special `invoke_debugger` API;
this seemed less bad than reworking all of the WASI setup to be more
reusable). Happy to take more ideas here.

* Review feedback.

* Review feedback.

* Review feedback: update vendor-wit.sh.

* Review feedback: -Ddebugger-arg= -> -Darg=.

* Review feedback.

* Review feedback.

* Review feedback: factor host.rs into several submodules.

* Review feedback: rename Debugger to Debuggee on host side.

* Review feedback: split inherit_stdin_stdout, and add corresponding options for the debug component.

* Review feedback.

* Review feedback.

* Add simple debug-component tests.

* Add wasm32-wasip2 target in a few places in CI

* Cargo vets for wstd dependency.

* Add wasm32-wasip2 in more places

* fix debug-component test dependence on componentization byte offsets

* Review feedback.

* Fix cancel-safety of EventFuture.

* Fix: Interrupted events should only occur after interrupt(), not on every epoch yield.

* Review feedback.

* Review feedback: strip down WASI imports in debugger world.

* fold debugger test component back into wasip1 + adapter test artifact compilation flow

show more ...

Enable limiting wasip3 resource limits (#12761)

* Enable limiting wasip3 resource limits

This commit adds a new `Store::concurrent_resource_table` method which
enables getting a handle to the under

Enable limiting wasip3 resource limits (#12761)

* Enable limiting wasip3 resource limits

This commit adds a new `Store::concurrent_resource_table` method which
enables getting a handle to the underlying `ResourceTable` used by the
concurrent implementation of component-model-async. This can in turn be
used to set the max capacity on the table and limit the guest usage of
the table.

Closes #11552

* Adjust features

* Fix imports

show more ...

Refactor WASIp2 `wasi:http` implementation (#12748)

* Sequester WASIp2 in `wasmtime-wasi-http` to a module

This mirrors the `wasmtime-wasi` crate's organization where there's a
`p2` module and a `

Refactor WASIp2 `wasi:http` implementation (#12748)

* Sequester WASIp2 in `wasmtime-wasi-http` to a module

This mirrors the `wasmtime-wasi` crate's organization where there's a
`p2` module and a `p3` module at the top level.

* Refactor WASIp2 `wasi:http` implementation

This commit reorganizes and refactors the WASIp2 implementation of
`wasi:http` to look more like other `wasmtime-wasi`-style interfaces.
Specifically the old `WasiHttpImpl<T>` structure is removed in favor of
as `WasiHttpCtxView<'_>` type that is used to implement
bindgen-generated `Host` traits. This necessitated reorganizing the
methods of the previous `WasiHttpView` trait like so:

* The `WasiHttpView` trait is renamed to `WasiHttpHooks` to make space
for a new `WasiHttpView` which behaves like `WasiView`, for example.

* The `ctx` and `table` methods of `WasiHttpHooks` were removed since
they'll be fields in `WasiHttpCtxView`.

* Helper methods for WASIp2 were moved to methods on `WasiHttpCtxView`
instead of default methods on `WasiHttpHooks`.

With these changes in place the WASIp3 organization was also updated
slightly as well. Notably WASIp3 now contains a reference to the crate's
`WasiHttpCtx` structure (which has field limits for example). WASIp3's
previous `WasiHttpCtx` trait is now renamed to `WasiHttpHooks` as well.
This means that there are two `WasiHttpHooks` traits right now, one for
WASIp2 and one for WASIp3. In the future I would like to unify these two
but that will require some more work around the default `send_request`.

A final note here is that the `WasiHttpHooks` trait previously, and
continues to be, optional for embedders to implement. Default functions
are provided as `wasmtime_wasi_http::{p2, p3}::default_hooks`.
Additionally there's a `Default for &mut dyn WasiHttpHooks`
implementation, too.

With all that outlined: the motivation for this change is to bring the
WASIp2 and WASIp3 implementations of `wasi:http` closer together. This
is inspired by refactorings I was doing for #12674 to apply the same
header limitations for WASIp3 as is done for WASIp2. Prior to this
change there were a number of differences such as WASIp3 not having
`crate::WasiHttpCtx` around, WASIp2 having a different organization of
structures/borrows, etc. The goal is to bring the two implementations
closer in line with each other to make refactoring across them more
consistent and easier.

* Make `WasiHttp` in WASIp2 public

* Fix some conditional build

* Fix some doctests

* Fix configured build

* Fixup documentation

show more ...

Fix two security advisories. (#12652)

* Fix two security advisories.

This commit contains merged fixes for two security advisories in
Wasmtime:

* GHSA-852m-cvvp-9p4w
* GHSA-243v-98vx-264h

This in

Fix two security advisories. (#12652)

* Fix two security advisories.

This commit contains merged fixes for two security advisories in
Wasmtime:

* GHSA-852m-cvvp-9p4w
* GHSA-243v-98vx-264h

This introduces new knobs to Wasmtime to limit the scope of resources
that WASI implementations will allocate on behalf of guests. Unlike
backports to 41.0.x-and-prior these knobs all have default values which
are considered reasonable for hosts if they don't further tune them. The
following CLI knobs have been added:

* `-Smax-resources` - limits the total component-model resources a guest
can allocate in a table
* `-Shostcall-fuel` - a broad limit which enforces that at most this
amount of data will be copied from the guest to the host in any one
API call (e.g. `string` values can't be too big, `list<string>` can't
be quadratic, etc). This fuel is reset on each host function call.
* `-Smax-random-size` - the maximal size of the return value of the
`get-random-bytes` and `get-insecure-random-bytes` WASI functions.
* `-Smax-http-fields-size` - a limit on the size of `wasi:http` `fields`
values to avoid infinitely buffering data within the host.

The `http` crate has additionally been updated to avoid a panic when
adding too many headers to a `fields` object.

Co-authored-by: Mark Bundschuh <[email protected]>
Co-authored-by: Pat Hickey <[email protected]>
Co-authored-by: Joel Dice <[email protected]>

* CI fixes

* Run rustfmt
* Fix wasi-common build

* Fix tests on 32-bit

* Fix nightly test expectations

prtest:full

---------

Co-authored-by: Mark Bundschuh <[email protected]>
Co-authored-by: Pat Hickey <[email protected]>
Co-authored-by: Joel Dice <[email protected]>

show more ...

Remove subtask reparenting (#12570)

This commit updates the implementation of component-model-async
primitives to remove the manual subtask reparenting process. This is
required to fix #12544 at a s

Remove subtask reparenting (#12570)

This commit updates the implementation of component-model-async
primitives to remove the manual subtask reparenting process. This is
required to fix #12544 at a semantic level because a subtask isn't ever
actually reparented, even if its parent exits. The first part of this
change is to remove the `GuestTask::subtasks` field and all relevant
manipulations of it.

This field, however, powered the `TaskExit` abstraction returned from
`call_concurrent`. This commit then subsequently deletes `TaskExit` and
all related infrastructure as it's no longer directly applicable as-is.
The rest of this change is then updating tests/bindings/etc to account
for these two changes.

The main semantic changes related to tests are:

* `wasmtime run`, with and without `--invoke`, no longer waits for all
subtasks. This instead only waits for the main task returning before
exiting. Whether or not this is the correct behavior is under
discussion in WebAssembly/component-model#608

* `wasmtime serve` has been updated to keep the store alive at least
until the response body has been fully transmitted. This is also part
of WebAssembly/component-model#608.

* Some `component-async-tests`-related tests were updated to either
avoid blocking the store as it wasn't needed or yield enough times to
ensure that the test passes.

Closes #12544

prtest:full

show more ...