LLM Inference on Strix Halo

A large shared memory pool makes Strix Halo interesting for local language models. It also makes it easy to confuse three different questions: can the model load, can it generate useful answers, and can it sustain acceptable latency under the intended workload?

My Gemma 4 31B experiments explored those questions on a ROG Flow Z13 with a Ryzen AI Max+ 395, Radeon 8060S, and 64 GiB of memory. The recorded configuration used Fedora 44, llama.cpp v0.6.0 at d812350, Mesa 26.2.3 and ROCm 10.1.0. These are historical experiment pins, not a promise that the same commands or results apply to another driver release.

The most useful outcome was a set of tradeoffs. Speculation helped one interactive workload substantially, but its best setting changed under concurrency. Prompt reuse mattered more to initial latency than the headline decode rate. Several plausible optimizations failed to qualify.

Start with a memory budget

AMD’s UMA explanation describes a physical memory pool shared by CPU and GPU. Application allocations still depend on the operating system, firmware configuration and runtime. A machine advertised with 64 GiB does not provide 64 GiB exclusively to a model.

For planning, I separate the budget into weights, KV cache, temporary buffers, an optional draft model, and room for the operating system:

$$M_{total} = M_{weights} + M_{KV} + M_{scratch} + M_{draft} + M_{system}.$$

The recorded Q4_K_M target file occupies 18.32 GB, and the Q8 assistant adds about 0.51 GB. File sizes are decimal bytes; the machine’s capacity is expressed in GiB. Neither number describes total resident memory.

Context length and concurrent slots increase the cache requirement. A configuration that starts with one request can fail when the server allocates several independent contexts. The experiment log includes a four-slot HIP configuration that failed a 4,800 MiB KV allocation. That is a failed configuration on this stack, not a universal limit of the APU.

This is why I would first reproduce one small, known configuration. Increasing context, changing cache precision and enabling more slots simultaneously makes an allocation failure difficult to interpret.

Measure the request, not only decoding

The measurement protocol separates time to first content, generation speed and request completion time. Interactive cases used English prose, Python and Korean prompts, with three timed requests per prompt after a discarded warmup.

The initial batch recorded:

Median decode uses each request’s generated-token timing. Wall throughput includes the time required to finish the batch. Pooling these measurements into one “tokens per second” number loses the distinction between how generation feels and how much work the server completes.

These small samples describe the recorded workloads. They are not an advertised device benchmark or a statistically qualified speedup across arbitrary prompts. Even the p95 values are descriptive with so few observations.

Why speculation has a sweet spot

A draft or assistant proposes tokens, and the target verifies them. The opportunity is to amortize target work over several accepted tokens. The cost includes running the assistant, verifying proposals and handling rejected continuations.

In the depth sweep, MTP-4 reached 24.34 median tok/s, while MTP-8 reached 10.34. The recorded acceptance fractions were approximately 0.46 and 0.28 respectively. A deeper proposal therefore did not imply more useful work per unit time.

The objective is not to maximize acceptance in isolation either. MTP-1 accepted a larger fraction but produced only 18.11 median tok/s in that sweep. Shorter proposals can be cheaper and still miss opportunities to amortize verification.

Kernel dispatch also matters. Verification changes the shape of the target’s operations. A dispatch path that works well for a single decoded token may behave differently for a short block. The repository retains experiments around that boundary rather than treating speculation as a runtime switch with a fixed multiplier.

Those observations suggest a practical search: sweep a few depths using the same prompts and output lengths, retain a plain baseline, and examine completion time alongside decode rate. Stop when the extra work no longer pays for itself.

Context reuse changes the experience

One recorded 4K retrieval workload used the same reference configuration with and without cache reuse. Median time to first content was about 0.85 seconds with reuse and 18.58 seconds cold. Median decode remained near 23.9 tok/s.

Those numbers describe different starting conditions. A fast warm follow-up does not establish fast processing of a new document. Mixing cold and reused prompts into one average hides the distinction a user will notice immediately.

The longer-context campaign also varied configured context capacity and cache precision. Capacity and actual input tokens are separate settings: a 16K context allocation does not mean every prompt contained 16K tokens. Any comparison needs both.

For a local document assistant, I would report cold first-answer latency, a reused follow-up, and the maximum admitted context independently. That exposes whether a change improves prompt processing, generation or only the amount of work that fits.

Concurrency changes the best setting

The concurrency campaign shows why a single-user result should not choose a shared server’s configuration:

These are selected rows from a separate campaign, not direct replacements for the initial table. The full matrix remains in the recorded data.

More aggregate output can coexist with slower individual responses. The setting appropriate for one person typing into a chat window may be unsuitable for several users or a batch queue.

Keep failed experiments visible

Adaptive MTP and alternative assistant precision did not establish a qualifying gain. A ROCmFPX speculative path failed validation. The repository records no execution path for the 31B target and its assistant on the NPU. None of those outcomes should be silently promoted because a related component compiled.

Vulkan and HIP results also belong to their exact runtime and workload. This campaign is insufficient to rank the APIs generally. Kernel versions, memory pressure, quantization and dispatch choices can change the result.

Most significantly, final validation remains incomplete: the thermal plateau check failed, absolute perplexity scores were unusually high, and an early JSON smoke test accepted fenced output before later checks parsed the whole response. Performance observations remain useful, but they do not establish sustained reliability or full correctness.

Reproducing the useful parts

Start with the pinned repository, its model hashes, and the setup guide. Record the hardware, power mode, software versions, target and assistant files, cache precision, context capacity and slots.

Run serially with baseline measurements before and after each group. Keep failures in the dataset and separate warm from cold prompts. Test the actual response format, retrieval behavior, natural termination and sustained load before calling a faster configuration usable.

The article data contains the selected values used here. To explore what changes when memory spans machines, continue with heterogeneous inference over USB4.