Several fixes to debugging infrastructure: component vs. module PCs and gdbstub wasm module names. (#12901)

* Debugging: fix module-relative vs component-relative PCs and unique library names.

Two

Several fixes to debugging infrastructure: component vs. module PCs and gdbstub wasm module names. (#12901)

* Debugging: fix module-relative vs component-relative PCs and unique library names.

Two bugfixes for guest debugging with components:

1. Convert component-relative source locations to module-relative PCs
in the frame table. The guest-debug API presents a core-Wasm view
where components are deconstructed into individual modules, so all
PCs must be module-relative. This adds a `wasm_module_offset` field
to `ModuleTranslation` and `FuncEnvironment`, set during component
translation, and subtracts it in `debug_tags()`.

2. Give unique names to "library" entries in the gdbstub XML response.
LLDB's DynamicLoader deduplicates by name, so using "wasm" for all
modules caused only the first to be loaded.

* Debugging: add ModulePC and ComponentPC newtypes for Wasm PC offsets.

Introduce `ModulePC` (module-relative) and `ComponentPC`
(component-relative) newtype wrappers around u32 Wasm bytecode
offsets. These replace raw u32 values throughout the frame table,
breakpoint, and debug systems to prevent confusion between the two
offset spaces.

* Debugging: add regression test for component module-relative PCs.

show more ...

fix: correct various typos (#12807)

Signed-off-by: Ho Kim <[email protected]>

Debugging: add the debug-main world. (#12756)

* Debugging: add the debug-main world.

This PR "draws the rest of the owl" for the debug-main
world (bytecodealliance/rfcs#45). This includes a WIT wor

Debugging: add the debug-main world. (#12756)

* Debugging: add the debug-main world.

This PR "draws the rest of the owl" for the debug-main
world (bytecodealliance/rfcs#45). This includes a WIT world that hosts
debug components that have access to "host debug powers" via a
debugging API, and the ability to load such a debug-component and give
it control of the main program as a debuggee when using `wasmtime
run`.

The WIT is namespaced to `bytecodealliance:wasmtime` and is slightly
aspirational in places: for example, the host does not yet implement
injection of early return values or exception-throws. I intend to fill
out a series of TODO issues once this all lands to track followup
("post-MVP") work.

This PR does not include any debug components. I separately have a
gdbstub component, with which I tested and co-developed this host-side
implementation. My plan is to land it in a followup PR as a component
that will be embedded in/shipped with the Wasmtime CLI and available
under an easy-to-use CLI option. Once we have that gdbstub component,
we can also implement end-to-end integration tests that boot up LLDB
and run through an expected interaction. (Separately, those
integration tests will require a release of wasi-sdk to ship an LLDB
binary that we can use.) As such, there are no real tests in this PR:
interesting behaviors only really occur with a full end-to-end flow.

The integration with the CLI is a little awkward (we internally build
another `wasmtime run` command that invokes the debug component, and
tie it together with the debuggee via a special `invoke_debugger` API;
this seemed less bad than reworking all of the WASI setup to be more
reusable). Happy to take more ideas here.

* Review feedback.

* Review feedback.

* Review feedback: update vendor-wit.sh.

* Review feedback: -Ddebugger-arg= -> -Darg=.

* Review feedback.

* Review feedback.

* Review feedback: factor host.rs into several submodules.

* Review feedback: rename Debugger to Debuggee on host side.

* Review feedback: split inherit_stdin_stdout, and add corresponding options for the debug component.

* Review feedback.

* Review feedback.

* Add simple debug-component tests.

* Add wasm32-wasip2 target in a few places in CI

* Cargo vets for wstd dependency.

* Add wasm32-wasip2 in more places

* fix debug-component test dependence on componentization byte offsets

* Review feedback.

* Fix cancel-safety of EventFuture.

* Fix: Interrupted events should only occur after interrupt(), not on every epoch yield.

* Review feedback.

* Review feedback: strip down WASI imports in debugger world.

* fold debugger test component back into wasip1 + adapter test artifact compilation flow

show more ...

Debugger: a few updates to the debugger crate. (#12684)

This PR upstreams a few changes from my WIP debug-component branch to
the async-wrapper in the debugger crate:

- The inner debuggee body take

Debugger: a few updates to the debugger crate. (#12684)

This PR upstreams a few changes from my WIP debug-component branch to
the async-wrapper in the debugger crate:

- The inner debuggee body takes a future that need not be `'static`;
this is important later in the integration.
- Two race conditions fixed: (i) wait for initial debuggee body
startup before sending a `with_store` query; and (ii) allow
`with_store` queries after receiving a `Finished` message, which may
happen when processing final debugger commands after the program
completes.
- As a result of some shuffling around future lifetimes and other
type-system head-banging, as well as the termination race-condition
fixed, the `Store<T>` is not transferred to the debuggee body and
back via the `JoinHandle`; rather it is passed back in a message
upon completion.

show more ...

Debugging: refactor stack frame cursor into frame handle abstraction. (#12566)

* Debugging: refactor stack frame cursor into frame handle abstraction.

This addresses some of the issues described #1

Debugging: refactor stack frame cursor into frame handle abstraction. (#12566)

* Debugging: refactor stack frame cursor into frame handle abstraction.

This addresses some of the issues described #12486: we need the ability
to keep a handle to a stack frame as long as execution is frozen, and
keep multiple of these handles around, alongside the `Store`, without
any handle directly holding a borrow of the store.

The frame handles work by means of an "execution version" scheme: the
idea is that whenever any execution resumes in a given store, all
handles to existing frames could be invalidated, but if no such
execution occurs, all handles should still be valid. A tuple of
(globally unique for process lifetime) store ID, and execution version
within that store, should be sufficient to uniquely identify any
frozen-stack period during execution. This accomplishes cheap handle
invalidation without the need to track existing handles.

This PR also implements a cache of parsed frame-table data. Previously
this was lazily parsed by the cursor as it walked up a stack, but with
multiple handles hanging around, and with handles meant to be cheap to
hold and clone, and with handles being invalidated eagerly, it makes
much more sense to persist this parsed metadata at the `Store` level.
(It cannot persist at the `Engine` level because PCs are local per
store.)

* Re-bless disas tests (offsets in VMStoreContext changed).

* Handle invalidation tests.

* Review comments, and make API return `Result`s rather than panic'ing on stale handles.

* Review feedback.

* Doc-comment link fix.

* Review feedback.

* cfg-gate Activation method to `debug` feature only.

* Fix unused-import warning in no-debug cfg.

* Fix doc link (again, after rename from latest feedback).

show more ...

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async

Remove need for explicit `Config::async_support` knob (#12371)

* Refactor component model host function definitions

Push the `async`-ness down one layer.

* Remove need for explicit `Config::async_support` knob

This commit is an attempt to step towards reconciling "old async" and
"new async" in Wasmtime. The old async style is the original async
support in Wasmtime with `call_async`, `func_wrap_async`, etc, where the
main property is that the store is "locked" during an async operation.
Put another way, a store can only execute at most one async operation at
a time. This is in contrast to "new async" support in Wasmtime with the
component-model-async (WASIp3) support, where stores can have more than
one async operation in flight at once.

This commit does not fully reconcile these differences, but it does
remove one hurdle along the way: `Config::async_support`. Since the
beginning of Wasmtime this configuration knob has existed to explicitly
demarcate a config/engine/store as "this thing requires `async` stuff
internally." This has started to make less and less sense over time
where the line between sync and async has become more murky with WASIp3
where the two worlds comingle. The goal of this commit is to deprecate
`Config::async_support` and make the function not actually do anything.

In isolation this can't simply be done, however, because there are many
load-bearing aspects of Wasmtime that rely on this `async_support` knob.
For example once epochs + yielding are enabled it's required that all
Wasm is executed on a fiber lest it hit an epoch and not know how to
yield. That means that this commit is not a simple removal of
`async_support` but instead a refactoring/rearchitecting of how async is
used internally within Wasmtime. The high-level ideas within Wasmtime
now are:

* A `Store` has a "requires async" boolean stored within it.
* All configuration options which end up requiring async, such as
yielding with epochs, turn this boolean on.
* Creation of host functions which use async
(e.g. `func_wrap_{async,concurrent}`) will also turn this option on.
* Synchronous API entrypoints into Wasmtime ensure that this boolean is
disabled.
* Asynchronous APIs are usable at any time.

This means that the concept of an async store vs a sync store is now
gone. All stores are equally capable of executing sync/async, and the
change now is that dynamically some stores will require that async is
used with certain configuration. Additionally all panicking conditions
around `async_support` have been converted to errors instead. All
relevant APIs already returned an error and things are murky enough now
that it's not necessarily trivial to get this right at the embedder
level. In the interest of avoiding panics all detected async mismatches
are now first-class `wasmtime::Error` values.

The end result of this commit is that `Config::async_support` is a
deprecated `#[doc(hidden)]` function that does nothing. While many
internal changes happened as well as having new tests for all this sort
of behavior this is not expected to have a great impact on external
consumers. In general a deletion of `async_support(true)` is in theory
all that's required. This is intended to make it easier to think about
async/sync/etc in the future with WASIp3 and eventually reconcile
`func_wrap_async` and `func_wrap_concurrent` for example. That's left
for future refactorings however.

prtest:full

* Review comments

* Fix CI failures

show more ...

Migrate debugger to `wasmtime::error` (#12261)

Debugger: add top-level crate and async wrapper allowing for a "debugger environment". (#12183)

* Debugger: add top-level crate and async wrapper allowing for a "debug environment".

The debug suppo

Debugger: add top-level crate and async wrapper allowing for a "debugger environment". (#12183)

* Debugger: add top-level crate and async wrapper allowing for a "debug environment".

The debug support in Wasmtime so far is structured around async
callbacks that occur at certain kinds of events, like breakpoints. This
is a suitable foundation but makes for an awkward implementation of a
top-level debugger implementation, which likely has an event loop
dealing with user commands (via a UI or a protocol connection) and
expects to perform actions such as "run until next breakpoint".

This PR introduces a new crate that wraps a `Store` in a `Debugger`.
This wrapper embodies an inner async body that can perform whatever
actions it likes on the `Store` that is passed back in. This inner body
is spawned as an async task. The debugger wrapper registers its own
`DebugHandler` callback that communicates with the outside world via
bidirectional command/response queues.

On the "outside", the `Debugger` presents an interface suitable for
inserting into a debug protocol server or UI: an async method that runs
until next event and returns that event, and a method that permits
querying or modifying the store whenever the `run` method is not
executing. The latter operates by sending a closure over the queue,
because the `Store` must continue to be owned by the async task that is
(still) running and suspended in async callbacks.

Right now, this is exercised only via a few unit tests, but the intent
is to next build up the "top half" of the debugger using this
abstraction, e.g. by running a gdbstub protocol server (likely as a Wasm
component in a "debug-main WIT world" -- RFC needed for this).

Also, when we eventually move debugging over to native use of
`run_concurrent`, this paradigm should remain mostly unchanged at this
level of API: there can still be an object that has an async method that
runs and yields the next event, and there can still be a method that
takes a closure that can operate (within its scope only) on the `Store`.

A few warts that I could use feedback on:

- Cancelation safety is weird. Fibers panic when dropped before
execution of their body completes, and this seems to mean that we
can't allow a `Debugger` to drop early (or at least, the `tokio::test`
unit test that owns the runtime that runs the async task to finish
before the debugged body completes!). If there is a better way to
handle cancelation safety here, I'm all ears.

- It's not clear to me if the boxed-closure-and-`Any` approach to
providing access to the `Store` is the best we can do, but I suspect
it is.

* Cancel safety!

* Add new crate to publish.rs script.

* Review feedback.

* Review feedback: state diagram.

* Update after merge from main: new DebugEvent for epoch yields.

* Make Debugger drop-safe by making debug event callbacks compatible with fiber teardown.

* CI fix on `debugger` crate manifest.

- Do not link to non-existent README in new crate.
- Remove a few other attributes that our internal crates don't have (now
the set of attributes at the top level is the same as for the new
error crate).

show more ...