Serving Qwen3.8-Flash-Next on Atlas: what it took, what it measured, what broke

1 · Bottom line

Performance. Atlas decodes this model at 16–17 tok/s regardless of context length, with prefill around 430 tok/s (a 13.4K-token prompt took 31 s). That is a third of prod's speed on the same task and below both llama.cpp Flash-Next configs measured on 2026-08-28 (20–25 tok/s). The recipe's own claim (16.5 tok/s, 747 tok/s prefill) is consistent with what we saw, so this is the engine's current ceiling, not a misconfiguration.

Output quality. The model produced a full-featured game design ("IRONGATE", 47.6 KB) in round 1 but shipped it dead: four of six <script> blocks were unterminated. Round 2 thrashed, rewriting the file three times inside one response until it hit the 26K-token cap. Two targeted fix rounds later the game loads with zero console errors, towers place and a wave starts; whether it plays through to wave 20 was the check in progress when the run was stopped.

Engine. The dominant cost was not the model but Atlas: every long-form request path (rewinds, speculative verify, prompts over 8K tokens, generations past 32K positions) hit a separate failure. Section 5 lists them with the verbatim error lines and the workaround that got past each.

2 · Setup and parameters

What was compared

Benchmark protocol (unchanged from the 2026-08-28 shootout)

• Round 1: the fixed tower-defense spec (20×12 grid, 3 towers with 2 upgrade levels and sell, 4 enemy types with a boss every 5th wave, economy, wave scaling, win at wave 20). Round 2: the model's own file returned for review / fix / balance / polish.
• Sampling: temperature 0.7, top_p 0.8, 26,624 max tokens per round, thinking disabled per request (chat_template_kwargs.enable_thinking=false, which Atlas honours).
• Harness extracts the single ```html block; an unterminated block is kept as a truncated file. A self-fix round feeds the file back with the browser's console error (config A got one such round in August; Atlas got two, on request "no budget limit").

The serve line that finally worked

ATLAS_QSA_MAX_TOKENS=65536 ATLAS_PLE_CACHE_SLOTS=524288 ATLAS_PLE_MAX_TOKENS=20500 \
spark serve --model-from-path ~/models/radixark-fn-nvfp4 --model-name qwen \
  --kernel-target qwen3.8-flash-next --bind 127.0.0.1 --port 8001 \
  --max-seq-len 49152 --max-num-seqs 1 --max-batch-size 1 \
  --gpu-memory-utilization 0.95 --fast-load-prefetch-shards \
  --max-prefill-tokens 20480 --content-loop-watchdog false --request-timeout 3600

Every value above that is not a default exists because a default failed (section 3). No --speculative: the recipe's documented C=1 win (--num-drafts 1, +18%) breaks once the sparse-attention indexer activates at position 2051.

Prefill: the single-chunk 13,397-token fix-1 prompt showed TTFT 31.3 s, about 430 tok/s. That is in line with the llama.cpp Flash-Next configs (466–475 tok/s on 21K-token prompts) and well below prod (720 tok/s).

Speculative decoding could not be measured: with --speculative --num-drafts 1 the MTP-heavy regime (98% of steps) ran at 23.5 tok/s until it hit defect 2 at position 2,051, and a run where the auto gate parked in serial mode gave 18.8 tok/s. The +18% the recipe documents is real but currently unusable past 2K tokens.

6 · The fix loop (rounds 1–2 of "no budget limit")

A persistent server stayed up between rounds. Each round fed the model its own file plus what the browser actually reported; the prompt was even-padded via /tokenize while the chunk-parity hypothesis was being tested. Output budget was raised to 28K–40K tokens.

The model's fix-1 preamble diagnosed its own round-2 file accurately: "unclosed IIFE in the main script … no game loop, no rendering, no event handlers exist despite all UI elements being present". Given a precise symptom it repairs well; left to review its own 60 KB file it rewrites from scratch and runs out of budget. That matches the August finding for config A.

Playtesting note for anyone repeating this: a hidden Browser pane reports document.hidden = true and Chrome suspends requestAnimationFrame, so the game loop appears frozen. Test copies with a virtual-clock shim (test2.html, test3.html under /bench/atlas/) were used to drive the loop deterministically; they should be deleted before anything is published.

7 · sparkrun recipe check

@atlas/qwen3.8-flash-next-nvfp4 (registry Avarok-Cybersecurity/atlas-recipes) cannot reproduce this run: it pins the Inferact checkpoint (BF16 PLE table; its text says the FP8 layout is refused, which is no longer true on main), image avarok/atlas-gb10:5be3fc1 from before PR #16, max_model_len 2048, utilization 0.85, and none of the PLE/QSA environment pins or the request timeout. Its full serve line is spark serve Inferact/… --gpu-memory-utilization 0.85 --max-seq-len 2048 --kv-cache-dtype bf16 --max-batch-size 1. The recipe format supports the needed knobs (siblings pin ssm_cache_slots, dflash, etc.), so an update is mechanical once a configuration is agreed.

8 · Fix campaign and the second run (2026-09-04)

All seven plan items were implemented on assist/flashnext-longform-fixes (fork branch, seven commits on 4df6e609, pushed; no upstream PR yet). Executor: Flash-Next Q4 on llama.cpp for the edits with exact anchors; three items were finished by hand where the executor stalled in search loops. Every commit passed fmt, clippy (deny-all) and the full spark-server test targets.

D-4 root cause. compute-sanitizer was ~100× too slow to reach the failing chunk, so the reproduction used CUDA_LAUNCH_BLOCKING=1, which attributed the fault to dense_gemm_bf16_pipelined launched as grid=[80,9]: the up_pre = low × up_wt^T GEMM inside hc_pre_gemm. Its scratch was packed as raw byte products and inj_pre is lay × 8 bytes, so an odd slab (1,117 tokens, or the default 8,193) placed up_wt 8 bytes off a 16-byte boundary. Even slabs (1,074 / 1,862 / 4,092) never faulted, which is why it looked random. The cuBLASLt "status 14" seen earlier was the sticky error from that kernel surfacing at the next API call, not a GEMM operand problem.

Second benchmark run (ATLAS2, fixed engine, default chunking, no env pins)

The engine side is now clean: one attempt, no workarounds, no 500s. The output is not: "BULWARK — Grid Defense Protocol" is a syntactically complete 38 KB file that defines the whole game and never boots — no newRun() call, no requestAnimationFrame loop, no event listeners. It loads with zero console errors and does nothing. Decode speed is unchanged by the fixes (16–17 tok/s single-stream; ~430 tok/s prefill): they remove failure modes, not cost.

9 · What is left

1. Pin the fix-5 boundary accounting (restored QSA cursor one token behind the resume position) and re-run with the watchdog on.
2. Decide on an upstream PR for assist/flashnext-longform-fixes (seven commits, CI-style checks green locally; the benchmark gate would need a re-measure).
3. Investigate why Flash-Next's outputs through Atlas are worse than through llama.cpp at identical sampling (template/sampling presets, min_p/top_n_sigma pins, EOS handling).
4. Recipe PR: post-fix image, RadixArk checkpoint, 32–64K context, longer request deadline.

Artifacts on spark: ~/tdbench/ATLAS/ (v1.trunc, v2, fix1, fix2, raw responses, rounds.jsonl), ~/driver_td2*.sh, ~/tdbench_r2.py, ~/tdfix3.py, ~/agentbench/atlas-td2-*.log. Prod (qwen38-dflash) was restored after every attempt and is running.

10 · 2026-09-04 afternoon: sampling, seven fix rounds, and the second engine batch

Sampling was the quality lever. Atlas fills unspecified request fields from the model card's [sampling.non_thinking] preset, including presence_penalty = 1.5. Over a 20K-token file that suppresses every token already emitted: ATLAS and ATLAS2 shipped dead/inert games. Pinning the request to llama.cpp's effective values (presence 0, top_k 40, min_p 0.05) on the same fixed engine gave ATLAS3: a complete, wired 51 KB game ("Ironline") with one typo.

Pattern: the model repairs exactly what the brief names, and both times it edited something unasked it introduced a regression. Config A on llama.cpp needed one round in August; with one sample per configuration that is suggestive, not proof.

Engine batch 2 (PR #23, commits 953820a0 … 7db64cca)

• Preset penalties opt-out — [behavior].use_sampling_preset_penalties_for_core (default true; Flash-Next sets false). Tool requests keep the tools preset.
• BF16 lm_head by default for qwen4_exp; --attn-proj-dtype bf16 keeps Q/K/V/O at checkpoint precision (ModelOpt's ignore list does the same).
• Rollback fixed for real. The content-loop handler runs before the step pushes its token, the fuzzy detector after; a fixed pop count was wrong at one of them ("pos N+1 but N ingested" vs "pos N but N+1"). rewind_buffers now pops every fed token from the boundary on, and the phantom token sampled from the discarded context is dropped instead of appended. On the GB10: two fuzzy and three content-loop rollbacks with re-steer, zero indexer desyncs.

Throughput (item 7)

The investigation (config A as investigator, 129 tool calls, report in ~/claude/reports/): CUDA graphs are vetoed model-wide because a QSA layer (host top-k) and a PLE layer (host hash) both veto; every token is ~1,500–2,000 eager launches with 12 D2H syncs and 36 PLE stream barriers. The two zero-code levers, ATLAS_D2H_SPIN_SYNC=1 and ATLAS_PLE_FAULT_THREADS=128, measured neutral (15.4 tok/s vs 15.5). The real levers are code: device-side QSA top-k to lift the veto, an event instead of the PLE barrier, one launch per hyper-connection site.

ATLAS4: BF16 head and attention

Same request sampling as ATLAS3, watchdog off: R1 20,217 tokens at 15.2 tok/s, R2 22,432 at 14.0, a 57.7 KB game ("Bastion Defense") with zero syntax errors and one logic bug (killed and leaked enemies are never removed from the enemy list, so a wave never clears). Placement, road, combat all work. Against ATLAS3's typo plus four logic bugs that is better, at a ~10% speed cost; one sample each. With the watchdog on, the same build rolled back cleanly three times but the loop detector false-positived on a coordinate array and cut a coherent file at 8.4K tokens: keep it off for code.

Pages: bench round 2 · ATLAS3 final game · ATLAS4 game · PR #23.