USB4 Transport: Link Speed, Copies and Measured Throughput

A cable can report a 40 Gb/s physical link while an application transfers useful data at less than 1 GB/s. Both observations can be correct. They describe different layers, and the gap is where transport engineering begins.

Halo USB4 is a Rust userspace fabric built on Linux USB4STREAM. Its connected qualification uses copied host-memory buffers between an Intel Core Ultra 7 356H and an AMD Ryzen AI 9 HX 370 / Radeon 890M. The project explores an interconnect for heterogeneous inference, but its hardware acceptance must follow the machines actually tested.

The October 27 campaign recorded a 20 Gb/s link. The October 31 follow-up observed 40 Gb/s in each direction and reran the bounded campaigns with verified artifacts. It did not establish what caused the changed link negotiation. As of this article’s evidence check, overnight qualification remained incomplete.

USB4NET and USB4STREAM are different paths

The Linux USB4 documentation describes both networking and streaming across a cable. USB4NET presents a network path; USB4STREAM exposes a stream interface configured through ConfigFS and accessed through a character device.

Halo’s payload transport uses the streaming path. It is not an IP benchmark renamed after a cable, and it does not silently fall back to TCP when a stream fails. A benchmark therefore needs to identify its actual backend rather than reporting only “USB4.”

USB4STREAM availability also depends on the kernel and its configuration. The recorded hosts ran Fedora 44 kernels 7.2.10 and 7.2.11. Copying these results to a different kernel, controller or operating system would skip the compatibility work that qualification is meant to establish.

The interface is only one part of the system. Endpoint provisioning, ownership, ring settings, moderation and process lifetimes also influence whether a real application progresses.

A DMA engine does not imply zero-copy

The upstream stream driver uses a copied iterator path between userspace buffers and kernel DMA pages. The controller performing DMA on its own buffers does not eliminate those copies.

This matters when interpreting CPU usage and bandwidth. The useful-byte counter belongs to the application; controller traffic includes its own protocol work. Byte verification and checksums can also consume resources during a qualification benchmark.

Direct GPU DMA would require a supported memory-sharing and synchronization path, appropriate lifetimes and evidence that GPU-produced data reaches the receiving consumer correctly. Those capabilities remain outside the accepted Halo profile. Loading an RCCL plugin does not supply that evidence.

Keep physical rate and goodput separate

At 40 Gb/s, division by eight gives a nominal 5 GB/s per direction before protocol overhead. That is arithmetic on the physical rate, not a measured application result.

The verified October 31 bounded campaign reported:

Both campaigns passed all 15 selected stages, including 72 ABI cases and 16 GiB of verified streaming. The numbers above are application sender goodput, including the tested copies and verification. They are decimal GB/s, not GiB/s.

The sanitized data keeps dates, settings, results and limitations together. A reader should be able to identify which physical profile produced each value without needing access to private operational logs.

The earlier October 27 campaign used the same documented ring sizes and moderation choices but a 20 Gb/s link. It recorded readiness goodput of 1.180 and 0.846 GB/s, and io_uring goodput of 0.367 and 0.418 GB/s, in the same direction order.

Those dated observations do not establish that increasing physical rate slowed or accelerated the application. The tests were separate campaigns, not a controlled link-speed comparison. Earlier ceiling investigations also used a different benchmark profile. Treating their best number as the result of the latest qualification would produce a misleading comparison.

io_uring was a profile to qualify

Asynchronous I/O does not guarantee better end-to-end throughput. Its result depends on the driver, completion behavior, queueing, workload and the cost of moving data around the I/O operation.

Repeated 32 MiB physical transfers failed under the tested io_uring profile with 8192 ns moderation. With both descriptors closed, changing moderation to zero allowed the qualified campaign to pass. A controlled rechallenge retained the failing comparison.

The supported native io_uring profile therefore uses zero moderation, a ring size of 256 and SQPOLL disabled. Managed startup rejects the incompatible moderation setting. Readiness polling remains the deployment default at 8192 ns.

This is a compatibility requirement for these kernels and endpoints, not a claim that interrupt moderation is generally incorrect. Zero moderation increases interrupt activity. It also did not make io_uring faster than readiness in the recorded application benchmark.

The important engineering outcome is an explicit supported profile with a failing configuration preserved, rather than a general preference for a fashionable I/O API.

Progress is part of correctness

Large transfers can pass while an isolated control operation stalls. In the qualification work, later health traffic allowed an outstanding operation to progress; simply draining completions differently did not resolve the observed behavior.

The final workaround caps control probes at a 1 ms cadence during traffic, then returns to idle cadence one second after non-health traffic ends. Health frames do not extend the active window or the failure deadline.

That change was tested with independent request progress, large messages and duplex repetitions. It is an observed progress fix. It does not establish an exact controller or kernel root cause.

A transport test should therefore include independent requests and idle-to-active transitions, not only a long stream that continually supplies new work.

Cancellation and peer death need separate tests

If an application cancels, chunks already authorized by receive credits can remain in flight. Rejecting every late chunk as an impossible event previously allowed one abandoned client to terminate the shared fabric.

The qualified behavior drains late data for retired connections after integrity and hop-credit checks. Live connections still require a matching grant. This preserves the distinction between expected cancellation residue and a protocol violation.

Peer failure has a different boundary. October 31 pending-send SIGKILL tests observed errors after approximately 14.77 seconds for readiness and 14.70 seconds for io_uring. The runner’s observation allowance was 20 seconds from injection.

Those are measured observations around a configured liveness deadline, not hard real-time guarantees. The harness must measure from the fault itself; starting its timer before management commands reach the peer can race legitimate detection.

Interrupted jobs fail. Fresh epochs restore a known session. Requests are not silently replayed, and application payloads are not redirected through another transport.

Verify the executable, not the directory name

The first October 31 campaign accidentally selected an old daemon from a directory named final. It passed many cases but failed independent progress. Comparing executable hashes showed that it was not the documented qualified artifact.

The rebuilt daemon and plugin matched the qualified identities. Fresh bounded campaigns passed. The stale run remains a failed result attributable to its own executable.

This is a useful lesson beyond USB4: a source commit, a successful build and a directory name do not prove what is running. Qualification should bind executable hashes, endpoint profile, peer identity and test results in one record.

What remains open

The two-host tests do not establish Strix Halo GPU coherency, HIP/RCCL collectives, vLLM serving, direct GPU DMA, physical multi-hop routing or parallel-cable bonding. The AMD cable-test machine is an HX 370, not Strix Halo.

Bounded fault and throughput campaigns also do not replace an overnight soak. The October 31 report was still incomplete when checked for this article, so long-duration reliability remains open.

For a reproduction, record exact artifacts and kernels, verify the negotiated lanes, confirm moderation with endpoints closed, then run integrity, independent progress, streaming and fault tests separately. Retain failed runs and their identities. That produces a defensible claim about a specific transport path—and a clearer foundation for heterogeneous inference.