WASIP3 update to latest rc (#12781)

Unconditionally set SO_REUSEADDR (#12597)

Run s390x CI on a native machine (#11711)

The hardware for this is provided by IBM and it's definitely faster than
QEMU. We've still got slow jobs elsewhere but removing this as a blocker
is still n

Run s390x CI on a native machine (#11711)

The hardware for this is provided by IBM and it's definitely faster than
QEMU. We've still got slow jobs elsewhere but removing this as a blocker
is still nice for CI times!

prtest:linux-s390x

show more ...

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendez

Revamp component model stream/future host API (again) (#11515)

* Revamp component model stream/future host API (again)

This changes the host APIs for dealing with futures and streams from a
"rendezvous"-style API to a callback-oriented one.

Previously you would create e.g. a `StreamReader`/`StreamWriter` pair and call
their `read` and `write` methods, respectively, and those methods would return
`Future`s that resolved when the operation was matched with a corresponding
`write` or `read` operation on the other end.

With the new API, you instead provide a `StreamProducer` trait implementation
whe creating the stream, whose `produce` method will be called as soon as a read
happens, giving the implementation a chance to respond immediately without
making the reader wait for a rendezvous. Likewise, you can match the read end
of a stream to a `StreamConsumer` to respond immediately to writes. This model
should reduce scheduling overhead and make it easier to e.g. pipe items to/from
`AsyncWrite`/`AsyncRead` or `Sink`/`Stream` implementations without needing to
explicitly spawn background tasks. In addition, the new API provides direct
access to guest read and write buffers for `stream<u8>` operations, enabling
zero-copy operations.

Other changes:

- I've removed the `HostTaskOutput`; we were using it to run extra code with
access to the store after a host task completes, but we can do that more
elegantly inside the future using `tls::get`. This also allowed me to
simplify `Instance::poll_until` a bit.

- I've removed the `watch_{reader,writer}` functionality; it's not needed now
given that the runtime will automatically dispose of the producer or consumer
when the other end of the stream or future is closed -- no need for embedder
code to manage that.

- In order to make `UntypedWriteBuffer` `Send`, I had to wrap its raw pointer
`buf` field in a `SendSyncPtr`.

- I've removed `{Future,Stream}Writer` entirely and moved
`Instance::{future,stream}` to `{Future,Stream}Reader::new`, respectively.

- I've added a bounds check to the beginnings of `Instance::guest_read` and
`Instance::guest_write` so that we need not do it later in
`Guest{Source,Destination}::remaining`, meaning those functions can be
infallible.

Note that I haven't updated `wasmtime-wasi` yet to match; that will happen in
one or more follow-up commits.

Signed-off-by: Joel Dice <[email protected]>

* Add `Accessor::getter`, rename `with_data` to `with_getter`

* fixup bindgen invocation

Signed-off-by: Roman Volosatovs <[email protected]>

* add support for zero-length writes/reads to/from host

I've added a test to cover this; it also tests direct buffer access for
`stream<u8>`, which I realized I forgot to cover earlier. And of course there
was a bug :facepalm:.

Signed-off-by: Joel Dice <[email protected]>

* add `{Destination,Source}::remaining` methods

This can help `Stream{Producer,Consumer}` implementations determine how many
items to write or read, respectively.

Signed-off-by: Joel Dice <[email protected]>

* wasi: migrate sockets to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* tests: read the socket stream until EOF

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: account for cancellation

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: mostly ensure byte buffer cancellation-safety

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-filesystem: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3-sockets: mostly ensure byte buffer cancellation-safety

* p3-cli: switch to new API

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: limit maximum buffer size

Signed-off-by: Roman Volosatovs <[email protected]>

* p3-sockets: remove reuseaddr test loop workaround

Signed-off-by: Roman Volosatovs <[email protected]>

* p3: drive I/O in `when_ready`

Signed-off-by: Roman Volosatovs <[email protected]>

* fixup! p3: drive I/O in `when_ready`

* Refine `Stream{Producer,Consumer}` APIs

Per conversations last week with Roman, Alex, and Lann, I've updated these
traits to present a lower-level API based on `poll_{consume,produce}` functions
and have documented the implementation requirements for various scenarios which
have come up in `wasmtime-wasi`, particularly around graceful cancellation. See
the doc comments for those functions for details.

Signed-off-by: Joel Dice <[email protected]>

* being integration of new API

Signed-off-by: Roman Volosatovs <[email protected]>

* update wasi/src/p3/filesystem to use new stream API

This is totally untested so far; I'll run the tests once we have everything else
compiling.

Signed-off-by: Joel Dice <[email protected]>

* update wasi/src/p3/cli to use new stream API

This is totally untested and doesn't even compile yet due to a lifetime issue I
don't have time to address yet. I'll follow up later with a fix.

Signed-off-by: Joel Dice <[email protected]>

* fix: remove `'a` bound on `&self`

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:sockets` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:cli` adaptation

Note, that this removes the read optimization - let's get the
implementation complete first and optimize later

Signed-off-by: Roman Volosatovs <[email protected]>

* remove redundant loop in sockets

Signed-off-by: Roman Volosatovs <[email protected]>

* wasi: buffer on 0-length reads

Signed-off-by: Roman Volosatovs <[email protected]>

* finish `wasi:filesystem` adaptation

Signed-off-by: Roman Volosatovs <[email protected]>

* remove `MAX_BUFFER_CAPACITY`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor `Cursor` usage

Signed-off-by: Roman Volosatovs <[email protected]>

* impl Default for VecBuffer

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: use consistent import styling

Signed-off-by: Roman Volosatovs <[email protected]>

* feature-gate fs Arc accessors

Signed-off-by: Roman Volosatovs <[email protected]>

* Update test expectations

---------

Signed-off-by: Joel Dice <[email protected]>
Signed-off-by: Roman Volosatovs <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>
Co-authored-by: Roman Volosatovs <[email protected]>

show more ...

Update WASIp3 wasi:sockets WITs (#11422)

Brings in a few cosmetic changes to definitions to keep things up-to-date.

Closes bytecodealliance/wasip3-prototyping#242

feat(wasip3): implement `wasi:sockets` (#11291)

* chore(wasip3): update `wasi:sockets` WIT

pull in `wasi:sockets` from
https://github.com/WebAssembly/wasi-sockets/pull/126

Signed-off-by: Roman Vol

feat(wasip3): implement `wasi:sockets` (#11291)

* chore(wasip3): update `wasi:sockets` WIT

pull in `wasi:sockets` from
https://github.com/WebAssembly/wasi-sockets/pull/126

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: rearrange Ctx/View

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: name modules after packages

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: move `Network` to `p2`

Signed-off-by: Roman Volosatovs <[email protected]>

* feat(wasip3): implement `wasi:sockets`

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: reuse socket utilities

Signed-off-by: Roman Volosatovs <[email protected]>

* test(wasip3): allow name resolution to fail

Signed-off-by: Roman Volosatovs <[email protected]>

* fix: expect macos-only var to be unused on other OSes

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: whole buffer is written for UDP

Signed-off-by: Roman Volosatovs <[email protected]>

* undo non-blocking write

Signed-off-by: Roman Volosatovs <[email protected]>

* extract buf capacity const, refactor tcp receive

Signed-off-by: Roman Volosatovs <[email protected]>

* introduce container struct for TCP non-inherited opts

Signed-off-by: Roman Volosatovs <[email protected]>

* handle `listen` receiver close

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: clean up

Signed-off-by: Roman Volosatovs <[email protected]>

* adapt to API changes

prtest:full

Signed-off-by: Roman Volosatovs <[email protected]>

* refactor: check for listen receiver close each iteration

Signed-off-by: Roman Volosatovs <[email protected]>

* Try to make test less flaky

* Cut down on `#[cfg]` required for wasi:sockets

* Fix test for real this time

---------

Signed-off-by: Roman Volosatovs <[email protected]>
Co-authored-by: Alex Crichton <[email protected]>

show more ...