Building a Heterogeneous Inference Cluster over USB4

Putting two computers next to each other does not combine their GPU memory into one allocation. A distributed inference runtime must assign work, move intermediate values, synchronize state and decide what happens when a participant disappears.

My Halo USB4 work investigates that boundary using a Rust userspace transport and a patched llama.cpp Vulkan RPC adapter. The recorded two-node acceptance run placed Qwen3-8B-Q4_K_M across an Intel Core Ultra 7 356H and an AMD Ryzen AI 9 HX 370 with Radeon 890M.

Both GPUs did work, and model execution generated verified Halo traffic after loading. The October 31 follow-up repeated mixed-vendor inference successfully. This is evidence of a particular heterogeneous execution path. It does not establish that splitting this model is faster than running it on one machine.

Decide why a cluster is needed

There are at least three different reasons to distribute inference: fitting a model that exceeds one device’s usable memory, increasing aggregate request throughput, or reducing the latency of one request.

Those objectives can lead to different designs. Splitting layers can address capacity while making one token slower. Independent replicas can increase throughput without sharing a model’s execution. Partitioning work inside layers introduces more frequent communication and synchronization.

The first useful comparison is therefore a single-node baseline with the same model, quantization, context and output settings. If it already meets the requirement, a second machine adds a protocol and a failure boundary that need a concrete benefit.

The Strix Halo experiments illustrate the same principle within one machine: a larger memory pool helps capacity, while runtime settings determine whether that capacity delivers useful latency.

Split execution, not physical memory

In the October 27 inference run, layers 0–18 were assigned to the Intel side and layers 19–36 to the AMD RPC worker. Those indices belong to that model and adapter; they should not be copied to a different architecture.

Each participant owns its local allocations and execution. The transport moves data needed by the remote computation. It does not make one GPU’s pointers valid on the other computer.

The upstream llama.cpp RPC documentation is useful background for remote execution. Halo’s experiment used a pinned, patched integration; it should not be represented as the behavior or support policy of every upstream RPC build.

Nor should the diagram be read as an exact packet trace. A runtime can issue multiple remote operations around a boundary. Counting only the final hidden-state tensor may understate real transport traffic.

Account for the whole critical path

A simple sequential model helps identify what needs measurement:

$$T_{token} \approx T_{local} + T_{remote} + T_{transfer} + T_{coordination}.$$

This is an explanatory approximation, not a fitted model of the measured runtime. Overlap, batching and multiple outstanding operations can change the critical path. It still highlights why adding a GPU does not automatically reduce latency.

For a payload of size $S$, transfer time has a bandwidth-dependent term and a per-operation term:

$$T_{transfer} \approx L + \frac{S}{B_{effective}}.$$

Here $B_{effective}$ must come from a relevant application measurement. Substituting the cable’s advertised rate ignores copies, framing, checksums, queueing and the endpoint profile.

Small operations can be dominated by latency and coordination even when a streaming benchmark looks healthy. Large prefill operations can shift the balance toward sustained throughput. The same cluster may behave differently for a short chat turn and a long document.

Balance stages with observed timings

An even layer count is a reasonable first experiment, but heterogeneous devices do not necessarily process identical layers at the same speed. Memory availability, backend kernels, cache sizes and thermal behavior can all change the balance.

Measure local and remote stage time during prefill and decoding, then adjust the partition. Keep enough headroom for cache growth and scratch allocations. A split that fits during initialization may fail later under a larger context.

The transport also consumes CPU time and memory bandwidth. On an integrated GPU, host work and device work can contend for shared resources. An apparent network optimization that increases CPU activity may move cost into model execution.

This suggests a disciplined sequence: establish a stable split, record end-to-end latency and per-stage timings, change one boundary, and repeat the same workload. Do not infer an optimal partition from peak compute specifications.

Prove that the intended path ran

The small inference answer used in qualification was “2 + 2 equals 4.” That confirms a narrow execution smoke test, not model quality. Its value comes from the surrounding evidence.

The recorded checks included GPU work counters on both devices, intended layer placement, Halo payload counters after model loading, matching artifacts and cleanup. Observing traffic only during loading would not establish that generation itself used the cable.

Similarly, a running worker process is not proof of GPU execution. A plugin loading is not proof that the application selected it. Acceptance has to connect the request, backend, transport and resulting output.

The sanitized qualification summary separates those outcomes. It records passing mixed-vendor inference and bounded physical transport campaigns, while retaining incomplete long-duration acceptance.

Make failures explicit

A remote stage turns one request into work distributed across independently failing processes. If a worker dies halfway through a request, blindly repeating the last operation can disagree with the surviving cache and sequence state.

Halo’s qualified behavior fails the interrupted job and requires an explicitly initialized fresh epoch for recovery. It does not silently replay requests or reroute USB4 payloads over TCP.

That policy makes the failure visible to the caller. A serving layer can then decide whether to start a new request from known input. The transport itself does not have enough application context to reconstruct a correct conversation or promise exactly-once generation.

Application cancellation also needs containment. Previously granted chunks can still arrive after a client stops waiting. The recorded fix drains validated late data for retired connections without allowing it to terminate unrelated clients. Active connections still need valid receive grants.

This matters for a shared inference service: one abandoned request should not take down the fabric used by everyone else.

Separate transport qualification from GPU collectives

The project also has an RCCL network plugin and a native oneCCL integration. Those interfaces cover different execution contracts.

The qualified oneCCL adapter supports two ranks and synchronous host buffers. The recorded Vulkan inference path likewise uses the host-memory fabric. Neither result proves GPU RCCL collectives, PyTorch distributed execution, vLLM serving or direct GPU DMA.

The distinction is especially important around Strix Halo. The AMD node in these cable tests is an HX 370 with Radeon 890M, not a Ryzen AI Max+ 395 with Radeon 8060S. A successful Intel/HX 370 run cannot establish acceptance on a different APU.

Physical multi-hop routing and parallel-cable bonding also remain separate from simulator coverage. More nodes change topology, contention and failure behavior; a two-host test cannot settle those questions.

Reproduce a claim in layers

A useful reproduction record starts with exact daemon, plugin and adapter identities, model hashes, software versions and endpoint settings. It then checks software parity, physical host-buffer integrity, the real application path and finally the intended workload.

Keep model loading separate from timed generation. Collect first-content and completion latency, tokens generated, errors, device activity and transport counters. Compare against one-node execution before claiming a performance benefit.

For long-duration use, include cancellation, peer failure, recovery and a completed soak. As checked on October 31, the new overnight campaign was still incomplete; its partial progress is not a reliability pass.

The current result is a demonstrated two-node heterogeneous Vulkan inference path over copied host-memory USB4 transport. The next article examines the transport itself: USB4 link speed, copies and measured throughput.