← Andrei Vogulkin · all writing
Field notes · 2026-09-03 → 2026-09-06 · Atlas × Qwen3.8-Flash-Next · NVIDIA DGX Spark (GB10)
Closing the gap to llama.cpp on a desk-side GB10 — six Atlas iterations, measured
Four days of serving one model on two engines and refusing to guess. Per iteration: the analysis and the comparison, the changes proposed and the decisions taken, the tower-defense benchmark, and the defects the model shipped. From 15 to 21.6 tokens per second against llama.cpp's 24–25, with every engine change proven bit-exact before its speed was written down.
The setup, in one paragraph
The model is Qwen3.8-Flash-Next: 48 layers, 36 of them Gated-DeltaNet, 12 full attention with a sparse indexer, a 512-expert MoE on every layer, a 51 GB n-gram embedding table read by row, and four-stream hyper-connections around every block. Prod serves it with llama.cpp (Unsloth UD-Q4_K_XL GGUF) at 24–25 tok/s on a DGX Spark. The candidate engine is Atlas (Rust + CUDA) with the RadixArk NVFP4 checkpoint. The benchmark is the tower-defense build-off from August: two ~20K-token rounds, identical prompts, and then fix rounds driven by console errors or a runtime description, exactly as the llama.cpp configurations were scored. The rules that held throughout: one investigator/reviewer, the models on the GB10 execute; every repo change goes through one PR; an engine change is measured only after a greedy 160-token generation on the previous binary and the new build hash the same.
| Iteration | Engine head | What changed | R1 / R2 tok/s | Game |
| ATLAS | main @ 4df6e609 | nothing — find the failures | 17.3 (11 attempts) | needed a fix round to boot |
| ATLAS2 | cb6b1a59 | five engine defects fixed, defaults restored | 17.3 → 16.7 | inert |
| ATLAS3 | cb6b1a59 | request sampling equalised with llama.cpp | 17.0 → 16.0 | playable after 8 fix rounds |
| ATLAS4 | 7db64cca | BF16 head + attention; rollback rework | 15.2 → 14.0 | 1 logic bug |
| ATLAS5 | 0b9ada13 | gap-study items A1–A3 | 14.7 → 13.9 | playable after 1 fix round |
| ATLAS6 | e7b6af2f | profile-driven: NVFP4 GDN, x4 hyper-connection kernel | 21.6 → 20.3 | playable after 3 fix rounds |
| llama.cpp (prod) | PR #27742 build | — | 24–25 | playable after 1 fix round (August) |
The games: versions and references
Same table as the August build-off, extended with the six Atlas iterations. Every run is one complete HTML5 tower-defense game built from identical prompts; the August rows are the reference the Atlas rows are measured against. "Play" opens the final published version; the version table underneath lists every round that produced a file, what the model was given for it, and where that intermediate version can be opened.
| Config | Model | Engine & serving | Memory | tok/s R1→R2 | Build time | Game size | Outcome | |
P prod (August) | Huihui-Qwen3.8-27B-abliterated 27B dense · Q4_K_M GGUF | llama.cpp DFlash build (PR #27342) DFlash2 speculative decoding · Sharp template | 17 GB | 46.6 → 49.0 | ~9 min | 35.8 KB | ✅ works functional, plainest design | Play |
E2 August | Qwen3.8-Flash-Next Unsloth UD-Q3_K_XL (84 GB) | llama.cpp PR #27742 · PLE → CPU/mmap · 64K ctx · f16 KV | ~89 GB | 24.6 → 20.0 | ~32 min | 56.1 KB | ✅ works most polished game | Play |
A August · prod today | Qwen3.8-Flash-Next Unsloth UD-Q4_K_XL (111 GB) | llama.cpp PR #27742 · PLE → CPU/mmap · 128K ctx · q8_0 KV | ~111 GB | 23.4 → 19.4 fix: 19.4 | ~31 + 16 min | 50.2 KB | ⚠️→✅ works 1 fatal typo, self-fixed from the console error | Play |
ATLAS main | Qwen3.8-Flash-Next RadixArk NVFP4 (126 GB on disk) | Atlas 4df6e609 · single-chunk prefill (20K) · 48K ctx · watchdog off · 11 serve attempts | ~100 GB | 17.3 → 16.3 fix: 16.6, 16.4 | ~17 + 27 min + 2 fix rounds | 47.8 KB | ⚠️ dead → loads 4 unclosed script blocks; loads after 2 fix rounds, play-through unverified | Play |
ATLAS2 fixed engine | Qwen3.8-Flash-Next (same) | Atlas branch 97aa9fa4 (7 fixes) · default 8K chunked prefill · 64K ctx · no env pins | ~100 GB | 17.3 → 16.7 | ~9 + 14 min | 38.2 KB | ⚠️ inert "BULWARK": complete file that never boots (no init call, no frame loop, no listeners) | Open |
ATLAS3 + llama.cpp sampling | Qwen3.8-Flash-Next (same) | Atlas branch cb6b1a59 · request pins presence 0 · top_k 40 · min_p 0.05 | ~100 GB | 17.0 → 16.0 fix: 15.5 | ~19 + 20 min + 8 fix rounds | 51.2 KB | ✅ playable after 8 fix rounds "Ironline": one typo, then four logic bugs, one regression per round | Play |
ATLAS4 rollback rework · BF16 head + attention | Qwen3.8-Flash-Next (same NVFP4 experts) | Atlas branch 7db64cca · BF16 lm_head + --attn-proj-dtype bf16 · watchdog off | ~102 GB | 15.2 → 14.0 | ~22 + 27 min | 57.7 KB | ⚠️ one logic bug "Bastion Defense": everything works except dead enemies are never removed, so no wave clears | Open |
ATLAS5 gap-study items A1–A3 | Qwen3.8-Flash-Next (same NVFP4 experts) | Atlas branch 0b9ada13 · + device QSA top-k · 4M-slot PLE cache · event-scoped pin release | ~102 GB | 14.7 → 13.9 | ~26 + 29 min + 1 fix round | 62.2 KB | ✅ playable after 1 fix round "IRONGATE": zero errors on load; stats panel never refreshed | Play |
ATLAS6 NVFP4 GDN + x4 hyper-connection kernel | Qwen3.8-Flash-Next (same NVFP4 experts) | Atlas branch e7b6af2f · ATLAS_QWEN4EXP_BF16_GDN=0 · BF16 head · NVFP4 attention · hc_pre_finish_x4 | ~108 GB | 21.6 → 20.3 | ~16 + 17 min + 3 fix rounds | 52.9 KB | ✅ playable after 3 fix rounds "IRONWORKS · TD": wrong helper name, unbound level variable; then four waves cleared | Play |
Serve command per game
Every Atlas round used the same line; the head, the env pins and the extra flags differ per iteration and are listed against each game. Prod is stopped first because the 128 GB of unified memory is one pool.
The serve line (Atlas rounds)
ATLAS_PLE_MAX_TOKENS=8212 <env pins> ./target/release/spark serve \
--model-from-path ~/models/radixark-fn-nvfp4 --model-name qwen --kernel-target qwen3.8-flash-next \
--bind 127.0.0.1 --port 8001 --max-seq-len <SEQ> --max-num-seqs 1 --max-batch-size 1 \
--gpu-memory-utilization 0.95 --fast-load-prefetch-shards --max-prefill-tokens 8192 \
--request-timeout 3600 <extra flags>
| Config | Head | Env pins | --max-seq-len | Extra flags | Request sampling pins | Result |
ATLAS Play | 4df6e609 (main) | ATLAS_QSA_MAX_TOKENS=65536 ATLAS_PLE_CACHE_SLOTS=524288 ATLAS_PLE_MAX_TOKENS=20500 | 49152 | --max-prefill-tokens 20480 --content-loop-watchdog false | none (Atlas model preset applies: presence 1.5, top_k 20) | 17.3 / 16.3; the only line that survived 11 attempts (single-chunk prefill sized to the whole prompt) |
ATLAS2 Play | 97aa9fa4 | — (PLE cap derived from --max-prefill-tokens: 8212) | 65536 | --content-loop-watchdog false | none | 17.3 / 16.7; default 8K chunked prefill, zero server errors |
ATLAS3 Play | cb6b1a59 | — | 65536 | --content-loop-watchdog false | presence_penalty 0 · top_k 40 · min_p 0.05 | 17.0 / 16.0; sampling equalised with llama.cpp |
ATLAS4 Play | 7db64cca | — | 65536 | --attn-proj-dtype bf16 (BF16 lm_head is the new default) | same pins | watchdog ON first: R1 cut to 8.4K tokens by three false-positive rollbacks on a coordinate array; published run with --content-loop-watchdog false: 15.2 / 14.0 |
ATLAS5 Play | 0b9ada13 | ATLAS_QSA_DEVICE_TOPK=1 ATLAS_PLE_CACHE_SLOTS=4194304 | 65536 | --attn-proj-dtype bf16 --content-loop-watchdog false | same pins | 14.7 / 13.9; device-side QSA selection, 4M-slot PLE row cache, event-scoped pin release |
ATLAS6 Play | e7b6af2f (fix rounds at 22321152) | ATLAS_QWEN4EXP_BF16_GDN=0 ATLAS_QSA_DEVICE_TOPK=1 ATLAS_PLE_CACHE_SLOTS=4194304 | 49152 | --lm-head-dtype bf16 --attn-proj-dtype nvfp4 --content-loop-watchdog false | same pins | 21.6 / 20.3; NVFP4 GDN projections (+6 GB, hence 48K ctx), x4 hyper-connection kernel |
Common to all rounds: temperature 0.7, top_p 0.8, 26,624 tokens per round, thinking disabled. The request pins from ATLAS3 on stop Atlas's model preset (presence penalty 1.5, top_k 20) from applying, which is what llama.cpp had been getting all along.
Reference: the prod llama.cpp line on the same box
LLAMA_ATTN_ROT_DISABLE=1 llama-server --host 0.0.0.0 --port 8000 --alias qwen -ngl 999 --jinja -fa on \
-m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --override-tensor 'per_layer_token_embd=CPU' \
-c 131072 -ctk q8_0 -ctv q8_0 --parallel 1 # llama.cpp PR #27742 build, container llama-flashnext
Other tested variations (single prompt, no game built)
3,882-token prompt, 1,200-token greedy generations, watchdog off, 48K context, 8K prefill chunks. Every kernel change had to reproduce the previous binary's greedy output hash before its speed was counted.
| Variation | Env | Flags | tok/s | Verdict |
| precision: BF16 head + BF16 attention | ATLAS_QSA_DEVICE_TOPK=1 ATLAS_PLE_CACHE_SLOTS=4194304 | --attn-proj-dtype bf16 | 13.8 | each BF16 choice ≈ −5–6% |
| precision: BF16 head + NVFP4 attention | same | --lm-head-dtype bf16 --attn-proj-dtype nvfp4 | 15.0 | the ATLAS6 base |
| precision: NVFP4 head + BF16 attention | same | --lm-head-dtype nvfp4 --attn-proj-dtype bf16 | 14.7 | |
| precision: NVFP4 head + NVFP4 attention | same | --lm-head-dtype nvfp4 --attn-proj-dtype nvfp4 | 15.6 | fastest, argmax-flip risk in the head |
| GDN projections NVFP4 | + ATLAS_QWEN4EXP_BF16_GDN=0 | --lm-head-dtype bf16 --attn-proj-dtype nvfp4 | 18.3 | adopted (ATLAS6) |
| + hc_pre_finish_x4 (e7b6af2f) | same | same | 20.3 | adopted; ATLAS_HC_FIN_X4=0 restores the old kernel |
| + FP8 lm_head | same | --lm-head-dtype fp8 --attn-proj-dtype nvfp4 | 21.2 | not adopted (precision) |
| + NVFP4 lm_head | same | --lm-head-dtype nvfp4 --attn-proj-dtype nvfp4 | 21.7 | not adopted (precision) |
| GDN fused norm | + ATLAS_GDN_FUSED_NORM=1 | same as ATLAS6 | 20.1 (neutral) | rejected: greedy output differs |
| MoE grouped decode arm (A2) | + ATLAS_MOE_DECODE_ARM=grouped | --attn-proj-dtype bf16 | 5.3 | 3× slower than the scalar arm; scalar stays |
| device top-k on/off (A1) | ATLAS_QSA_DEVICE_TOPK=1 vs unset | --attn-proj-dtype bf16 | +3–5% with sparse selection active | adopted |
| 4M PLE slots + event pins (A3) | ATLAS_PLE_CACHE_SLOTS=4194304 | --attn-proj-dtype bf16 | 14.5 / 15.1 (neutral) | kept (harmless) |
| D2H spin sync / PLE fault threads | ATLAS_D2H_SPIN_SYNC=1 · ATLAS_PLE_FAULT_THREADS=128 | --attn-proj-dtype bf16 | neutral | not adopted |
| decode CUDA graphs on/off (llama.cpp, for reference) | — | llama-server graphs on/off | 24.8/25.1 vs 24.4/24.1 | ≈3% |
Every version, round by round
Fix rounds come in three kinds: a console error (the browser's message, nothing else), a playtest brief (the reviewer's description of what is wrong and, usually, where), and a literal brief (the exact edit to make, with "do not change anything else"). The pattern across the table: the model repairs precisely what a literal brief names and regresses on whatever it touches unasked.
| Config | Round | Input given | Tokens | Wall | tok/s | Result | Version |
| ATLAS | R1 build | build prompt | 18,027 | 17 m 19 s | 17.3 | unterminated; 4 of 6 script blocks unclosed | — |
| R2 fix/polish | review prompt + own file | 26,624 | 27 m 18 s | 16.3 | hit the 26K cap; best of three rewrites, broken | — |
| fix 1 | console error: Unexpected end of input | 16,687 | 16 m 48 s | 16.6 | loads | — |
| fix 2 | playtest brief (click mapping, towers never fire, wave never ends) | 17,475 | 17 m 46 s | 16.4 | loads; play-through unverified | Play |
| ATLAS2 | R1 build | build prompt | 9,612 | 9 m 16 s | 17.3 | closed file | — |
| R2 fix/polish | review prompt + own file | 13,866 | 13 m 51 s | 16.7 | closed; never boots | Open |
| ATLAS3 | R1 build | build prompt | 18,954 | 18 m 34 s | 17.0 | closed file | — |
| R2 fix/polish | review prompt + own file | 18,953 | 19 m 44 s | 16.0 | closed; one fatal typo | — |
| fix 1 (self-fix) | console error: c is not defined at drawOverlayGrid | 18,980 | 20 m 22 s | 15.5 | one-token diff; intro-state bug remains | — |
| fix 2 | playtest brief: intro state, placement, road visibility | 19,014 | 19 m 51 s | 16.0 | state fix correct; a stray } broke the script | v |
| fix 3 | console error: Unexpected token '}' | 30,334 | 35 m 55 s | 14.1 | one statement changed; 11K tokens of extra output | v |
| fix 4 | playtest brief: overlay never shown, path in cell units | 19,073 | 20 m 24 s | 15.6 | fixed the two lines named, ignored the other path consumers | v |
| fix 5 | literal brief: path in pixels at one place, HUD writer | 19,318 | 20 m 32 s | 15.7 | playable; kill reward 0/0 → NaN gold | v |
| fix 6 | playtest brief: NaN gold | 19,259 | 20 m 11 s | 15.9 | NaN fixed; an unasked edit reintroduced the original typo | v |
| fix 7 | console error: c is not defined (again) | 19,046 | 20 m 10 s | 15.7 | single-token diff; fully playable (3 waves, 20/20 lives) | v |
| fix 8a | playtest brief: towers placeable on the road | 16,480 | 18 m 17 s | 15.0 | no code block — explanation, then EOS mid-file | — |
| fix 8 (final) | tighter brief, rerun (two named changes, nothing else) | 19,012 | 22 m 31 s | 14.1 | both fixes correct; an unasked edit broke the overlay grid again — reviewer reverted that one token | v Final |
| ATLAS4 | R1 build (watchdog on) | build prompt | 8,401 | 9 m 13 s | 15.2 | cut by 3 false-positive rollbacks on a coordinate array | — |
| R1 build (watchdog off) | build prompt | 20,217 | 22 m 08 s | 15.2 | closed file | — |
| R2 fix/polish | review prompt + own file | 22,432 | 26 m 39 s | 14.0 | closed; 0 syntax errors; dead enemies never removed | Open |
| ATLAS5 | R1 build | build prompt | 23,227 | 26 m 24 s | 14.7 | closed file | — |
| R2 fix/polish | review prompt + own file | 23,711 | 28 m 30 s | 13.9 | closed; 0 errors; playable except the stats panel | — |
| fix 1 (final) | literal brief: call updateStats() from the frame loop | 23,581 | 29 m 28 s | 13.3 | exactly the two calls requested; panel verified live | v Final |
| ATLAS6 | R1 build | build prompt | 20,621 | 15 m 57 s | 21.6 | closed file | — |
| R2 fix/polish | review prompt + own file | 20,697 | 16 m 59 s | 20.3 | closed; calls undefined buildStatic() | — |
| fix 1 | console error: buildStatic is not defined | 20,719 | 17 m 35 s | 19.6 | file echoed unchanged | — |
| fix 2 | literal brief: rename the one call to bakeStatic() | 20,399 | 16 m 39 s | 20.4 | exactly the one-token rename; update loop then throws on lv | v |
| fix 3 (final) | literal brief: bind lv in the gun/cannon branch | 20,516 | 17 m 12 s | 19.9 | exactly the two bindings; playable (4 waves cleared) | v Final |
The tower-defense benchmark: prompts, briefs and harness
The two build-off prompts are byte-identical to the August ones. The fix templates and every brief a fix round was given are below, verbatim, so each row of the version table can be reproduced.
Round 1 — build prompt (identical for all configs)
Build a complete, polished single-file HTML5 tower-defense game (index.html: all CSS and JS inline, canvas-based, no external assets or libraries). Requirements:
- 20x12 grid map with a fixed winding enemy path (visually distinct road) and buildable cells
- 3 tower types: Gun (fast, single-target), Cannon (slow, splash AoE), Frost (slows enemies). Each has cost, range, and 2 upgrade levels (better damage/range/fire-rate). Towers can be sold for 70% of invested cost.
- Enemies: normal, fast, tank; a boss every 5th wave. Health bars above enemies. HP/speed/count scale each wave.
- Economy: start with 100 gold and 20 lives; kill bounties; wave-completion bonus; a leaked enemy costs 1 life (boss costs 5).
- UI: top bar (gold, lives, wave, score); tower shop panel with prices; click a shop tower then click the map to place (show range preview and valid/invalid cell highlight); click a placed tower to select it and show upgrade/sell panel; Start Wave button plus auto-start toggle; game speed x1/x2 button.
- Targeting: enemies closest to the exit first. Render projectiles. Show splash radius on cannon impact and a frost visual on slowed enemies.
- Win after beating wave 20; lose at 0 lives. Proper end screens with a Restart button.
- Polish: hover states, particle effects on kills, hit flashes, clean modern dark theme.
Output the COMPLETE index.html in one ```html code block. No placeholders, no TODOs — fully working code only.
Round 2 — review / fix / balance / polish prompt
Below is your current index.html for the tower-defense game. 1) Review it for bugs: logic errors, undefined variables, broken event handlers, targeting/pathing mistakes, economy bugs, rendering issues — fix them all. 2) Ensure every item of the original spec is implemented: 3 tower types with 2 upgrade levels + sell; 4 enemy types incl. boss every 5th wave; wave scaling; gold/lives/wave/score UI; place/select/upgrade/sell interactions with range preview; start-wave + auto-start + speed toggle; closest-to-exit targeting; projectiles + splash + frost visuals; win at wave 20 / lose at 0 lives with restart; particles and polish. 3) Balance: tune costs, bounties and wave scaling so waves 1-20 are beatable but challenging. 4) Polish: particles, hit flashes, nicer end screens. If the file below is cut off mid-code, complete it properly. Output the COMPLETE final index.html in one ```html code block.
Current file:
```html
{FILE}
```
Fix round — console-error template
Your single-file HTML5 tower-defense game below fails to load. The browser console shows this error:
{ERR}
Find and fix the error. Also scan the whole file for any other syntax mistakes (unbalanced brackets/parentheses, broken statements) that would prevent it from running. Do not redesign anything — make the minimal corrections needed so the game runs. Output the COMPLETE fixed index.html in one ```html code block.
Current file:
```html
{FILE}
```
Fix round — playtest / literal-brief template
Your single-file HTML5 tower-defense game below now loads without console errors, but a playtest found these bugs:
{ERR}
Fix every one of them. Do not redesign anything and do not rewrite from scratch — make the minimal, targeted corrections needed so the game plays correctly end to end (place towers where the player clicks, towers acquire and shoot enemies in range, waves end and the next wave can start, HUD/toasts behave). Output the COMPLETE fixed index.html in one ```html code block.
Current file:
```html
{FILE}
```
How the rounds were run (sampling, protocol, serve line)
Sampling (all rounds, all configs): temperature 0.7 · top_p 0.8 · max_tokens 26,624 per round · thinking
disabled (chat_template_kwargs: enable_thinking=false). From ATLAS3 on, the request also pins
presence_penalty 0 · top_k 40 · min_p 0.05 (TDBENCH_EXTRA) so Atlas's model preset (presence 1.5) does not
apply — that is what llama.cpp had been getting all along.
Protocol: Round 1 = the build prompt; Round 2 = the review prompt with the model's own file pasted in. The
harness (tdbench_s.py) extracts the single ```html block from each reply; the last complete file is the
published game. A fix round (tdfix3.py) sends the current file back with either the browser console error
(FIX template) or a playtest brief written by the reviewer (RUNTIME template, texts below); the harness
pads the prompt with whitespace until its tokenised length is even (a workaround kept from the first run).
Every game was loaded in a browser and played through by the reviewer before its result was written down.
Serving (Atlas rounds): prod llama-flashnext stopped first (one 128 GB pool); the exact serve line, env pins
and flags per iteration are in "Serve command per game" above. Prod restored after each run. Publishing: spark → orange ~/sites/bench → Caddy → wg umbilical → edge (TLS).
The briefs, per fix round
ATLAS · fix 1 — console error
Uncaught SyntaxError: Unexpected end of input
ATLAS · fix 2 — playtest brief
RUNTIME:
1. Click-to-cell mapping is wrong on the Y axis: clicking a buildable cell places the tower on a different, lower row (roughly double the clicked canvas Y). The canvas is displayed at CSS size (width:100%) while its internal resolution differs, and the click handler does not scale mouse coordinates by canvas.width/rect.width and canvas.height/rect.height consistently for both axes. The range preview / hover highlight must use the same corrected mapping.
2. Because of (1) (and possibly independently) towers never fire: with two Gun towers placed beside the road, wave 1's six enemies walked the whole path untouched — score stayed 0, gold never increased, no projectiles were drawn. Verify targeting (closest-to-exit in range), fire-rate cooldown, projectile spawn/update/hit and bounty payout actually run inside the main game loop.
3. Wave completion never triggers: after all wave-1 enemies leaked (lives 20 -> 14) the HUD stayed "WAVE IN PROGRESS", the Start Wave button never re-enabled, and the "LANE CLEAR — deploy your towers" toast was re-added on every frame (dozens of stacked toasts). End the wave once all enemies of the wave have spawned and none remain (killed or leaked), pay the wave bonus, advance the wave counter, re-enable Start Wave / honor auto-start, and show status toasts only once per state change.
ATLAS3 · fix 1 — console error
Uncaught ReferenceError: c is not defined at drawOverlayGrid
ATLAS3 · fix 2 — playtest brief
RUNTIME:
1. The game can never be started. Clicking "Deploy Forces" (or anywhere on the canvas) runs handleClick, whose `state==='intro'` branch only toggles the intro overlay's CSS classes ('show'/'hide') and never sets `state='idle'`, so updateControls() keeps the Start Wave button disabled forever and the Space key does nothing. The intro overlay also flickers: it is briefly re-shown then hidden on every click. Fix: on Deploy Forces (and only there), hide the intro overlay for good, set state='idle', call updateControls(), and stop the canvas click handler from treating clicks as intro dismissal.
2. Because state stays 'intro', towers cannot be placed: handleClick returns early (`if(state!=='idle'&&state!=='wave')return;`) before the placement code. After fix 1, verify that picking a tower in the Armory (pendingType) and clicking a buildable cell calls placeTower and deducts gold.
3. The enemy road is nearly invisible: buildTerrain() strokes it with #3a4b58 / #2a3a46 / #33475a on a #101c25 background, so the player sees no path. Make the road clearly visible (higher-contrast fill and edge, e.g. a lighter road with a bright dashed centre line) and make the spawn/goal portals clearly visible.ATLAS3 · fix 3 — console error
index.html:530:135 Uncaught SyntaxError: Unexpected token '}'
ATLAS3 · fix 4 — playtest brief
RUNTIME:
1. The intro overlay is never shown, so the player cannot press "Deploy Forces" and the game stays in state 'intro' with Start Wave disabled. The `.overlay` CSS rule defaults to opacity:0 and pointer-events:none, and nothing ever adds the 'show' class to the #intro element at startup. Fix: add `intro.classList.add('show')` during init (after updateControls()), and make the Deploy Forces handler remove 'show' (it already sets state='idle' and calls updateControls()).
2. The enemy road is drawn as a tiny squiggle in the top-left corner instead of across the map: PATH from buildPath() holds grid-cell coordinates (0..19, 0..10.5), but strokePath(c) passes PATH[i].x / PATH[i].y straight to moveTo/lineTo as pixels. The portals are correct because drawPortal is called with SPAWN.x*CELL. Fix: in strokePath multiply by CELL (c.moveTo(PATH[0].x*CELL, PATH[0].y*CELL) and c.lineTo(PATH[i].x*CELL, PATH[i].y*CELL)). Then check every other consumer of PATH — computePathCells, pathDist, pointAt, PATH_LEN, enemy movement and the shadow dots in buildTerrain — and make sure they all agree on one unit (pixels) so that enemies walk along the visible road and pathCells marks the road cells as unbuildable.ATLAS3 · fix 5 — literal brief
RUNTIME:
1. Enemies still move in grid-cell units while everything else is in pixels: spawnEnemy starts them at PATH[0] (cell coords), updateEnemies advances e.dist by spd px/s (55–92) and pointAt(e.dist) interpolates along PATH in cell units, so PATH_LEN is only 25.5 and every hostile "breaches the line" in well under a second, never entering any tower's range (ranges are 125–172 px). Fix it at ONE place: define the path in pixels, i.e. replace `const PATH=buildPath();` with `const PATH=buildPath().map(p=>({x:p.x*CELL,y:p.y*CELL}));`. Then make these three consumers consistent with pixel coordinates: (a) strokePath(c): use PATH[i].x and PATH[i].y directly again (remove the *CELL you added there); (b) computePathCells(): the number of steps should be Math.ceil(Math.hypot(b.x-a.x,b.y-a.y)/CELL*4) and the cell key must be Math.floor(x/CELL)+'_'+Math.floor(y/CELL); (c) pointAt(s) and directionAt(s): the fallback return past the end must be {x:GOAL.x*CELL,y:GOAL.y*CELL} (GOAL/SPAWN stay in cells; drawPortal already multiplies by CELL). pathDist(px,py) and isBuildable() already pass pixels, so they become correct automatically. Do not change the enemy speeds or tower ranges.
2. The HUD counters never update: popStat(el) only replays the 'bump' animation and nothing ever writes the numbers, so the header keeps showing 100 gold / 20 lives / wave 0 / score 0 while the game state changes. Add a function updateHud() that writes gold, lives, the wave counter (current wave / total waves) and score into the existing HUD elements (goldVal, livesVal and the wave/score elements), call it at the start of popStat(el), once at init, and at the end of loop() every frame.ATLAS3 · fix 6 — playtest brief
RUNTIME:
1. Gold becomes NaN after the first kill, so the HUD shows "NaN" and no further towers can be bought. In updateEnemies(), the kill reward is computed as `Math.round(e.bounty*(1+e.shield/e.shieldMax*SHIELD_PEN))`; for unshielded enemies (normal, fast) shield and shieldMax are both 0, so 0/0 = NaN and `gold+=reward` poisons gold. Fix: compute the shield fraction safely, e.g. `const sf=e.shieldMax>0?e.shield/e.shieldMax:0; const reward=Math.round(e.bounty*(1+sf*SHIELD_PEN));`. Check the other places that divide by shieldMax (the enemy shield bar drawing, damageEnemy) and guard them the same way.
ATLAS3 · fix 7 — console error
index.html:497 Uncaught ReferenceError: c is not defined
at drawOverlayGrid (index.html:497)
at render
at loopATLAS3 · fix 8 — literal brief
RUNTIME:
1. Towers can be placed on the road. Cause: pathDist(px,py) returns NaN for every point, so `pathDist(c.x,c.y)<CELL*.5` in isBuildable() is never true. pathDist is NaN because PATH contains consecutive duplicate waypoints (buildPath() calls `push(3,4);corner(3,4);` and corner() pushes (3,4) again — same at (9,5) and (15,6)), which makes zero-length segments; pointSeg() divides by the squared segment length, 0/0 = NaN, and Math.min propagates it. Make exactly two changes and nothing else: (a) in pointSeg(), if the squared segment length is 0, return the plain distance from the point to the segment's start point; (b) in isBuildable(i,j), return false when pathCells.has(i+'_'+j). Do NOT modify buildPath() or the path shape, and do not rewrite any other function. Output the complete file.
ATLAS5 · fix 1 — literal brief
RUNTIME:
1. The top stats panel (gold, lives, wave, score — elements #sGold, #sLives, #sWave, #sScore) never changes while playing: updateStats() is only reached through updateAll(), which runs at init and on shop/inspector actions, so kills, leaks, wave changes and gold income are never reflected. Fix: in frame(now), right after the `for(let i=0;i<steps;i++)update(dt);` loop (inside the same `if(!G.paused&&G.phase!=='start')` block), call `updateStats();` and `updateStartButton();`. Do not change update(), draw(), or anything else.
ATLAS6 · fix 1 — console error
index.html:962 Uncaught ReferenceError: buildStatic is not defined
at resetGame (index.html)
at index.html:962ATLAS6 · fix 2 — literal brief
RUNTIME:
1. The game never starts: at init, resetGame() calls buildStatic(), which does not exist, so the script throws "ReferenceError: buildStatic is not defined" and requestAnimationFrame(loop) is never reached (the board stays black; only the HUD and shop clicks work). The pre-render function that builds roadBase/roadTop/decor/staticBake IS defined — it is named bakeStatic(). Make exactly one change: in resetGame(), replace the call `buildStatic();` with `bakeStatic();`. Do not rename bakeStatic, do not add a new function, do not change anything else. Output the complete file.
ATLAS6 · fix 3 — literal brief
RUNTIME:
1. The game freezes the moment a turret acquires a target: update(dt) throws "ReferenceError: lv is not defined" and the requestAnimationFrame loop dies. In update(), inside `for(const t of gridMap.flat())`, only the frost branch defines `const lv=t.lv`; the other branch (gun/cannon) uses `lv.dmg`, `lv.cd`, `lv.range` etc. without defining `lv` (it calls `lv2(t)` for the range check but never binds `lv`). Make exactly one change: at the start of that `else` branch (before `let best=null,bd=-1;`) add `const lv=lv2(t);`. Also check that the frost branch's `const lv=t.lv` is the intended level object (if `t.lv` is the level INDEX, use `const lv=lv2(t);` there too). Do not change anything else. Output the complete file.
Iteration 1 — ATLAS: serve it as-is and write down every failure
Analysis + comparison
Stock Atlas at main@4df6e609 took eleven serve attempts to complete the two rounds once. Five distinct engine defects: the loop-watchdog rollback desynchronised the sparse-attention indexer (HTTP 500 mid-stream); speculative verify with the indexer active ended responses as a clean stop at 1,679 tokens (silent truncation); prompts over 8,192 tokens were refused twice at the n-gram layer (a cap off by one, then a fixed 65,536-slot row cache); continuation prefill chunks faulted with CUDA_ERROR_MISALIGNED_ADDRESS and left a zombie holding 100 GB; and the indexer's capacity defaulted to 32,768 tokens regardless of context. Against the August llama.cpp run of the same model (config A: one typo, one fix round, playable), the difference was not the model.
Proposed changes + decisions
Two investigations of the same five defects, one by the prod model and one by config A on llama.cpp, each with read-only tools over the tree; config A's was the better-anchored one. A seven-item plan came out, ordered smallest-safe-first. Decision: execute all seven with config A as the executor from briefs carrying exact file:line anchors, accumulate on one branch, then run the benchmark again on default settings. (Wide briefs made the executor loop in code search; anchored briefs of at most three subtasks worked, and two items were finished by the reviewer.)
TD benchmark
17.3 tok/s single stream, ~300 tok/s prefill, with five environment pins and single-chunk prefill as workarounds.
Defects
The game ("IRONGATE") needed a second fix round to boot; the file itself was a symptom of the sampling problem found two iterations later.
Iteration 2 — ATLAS2: the fixed engine on defaults
Analysis + comparison
All seven items landed as separate commits: derived n-gram cache sizing (e623e010), indexer capacity from --max-seq-len (561be215), honest finish_reason="error" instead of a fake stop (16af4d33), the scheduler breaking on a latched CUDA fault instead of 500-ing for hours (35028544), rollback restoring the indexer and n-gram aux state (f4e81291), MTP latched off before the indexer's inert bound (86bdf303), and the prefill scratch laid out on 256-byte boundaries — the misaligned-address fault was an odd-width slab packed by raw byte products (97aa9fa4). The fault attribution came from CUDA_LAUNCH_BLOCKING=1; compute-sanitizer was a hundred times too slow to reach the failing chunk.
Proposed changes + decisions
Run on default chunking with no environment pins and no speculative decoding, and treat any server error as a failed run.
TD benchmark
17.3 → 16.7 tok/s, zero server errors. The fixes remove failure modes, not cost.
Defects
The game ("BULWARK") was inert: no boot loop, no event listeners, a 38 KB file that never ran. Same engine, same model, twice now. The conclusion had to be that the engine was not the quality problem.
Iteration 3 — ATLAS3: the sampling that was never equal
Analysis + comparison
Reading sampling_setup.rs: with the model's use_sampling_presets_for_core flag on, Atlas fills every request field the client leaves unset from the card's [sampling.non_thinking] preset — top_k 20, min_p 0, and presence_penalty 1.5. The August llama.cpp runs had used llama.cpp's own defaults: top_k 40, min_p 0.05, no penalty. A flat −1.5 on every already-emitted token is poison for a 20K-token file: closing braces, repeated identifiers and boilerplate all get suppressed. That is what "dead" and "inert" looked like.
Proposed changes + decisions
Pin the request to llama.cpp's values and re-run on the identical engine. Later, make it a model property: [behavior].use_sampling_preset_penalties_for_core, default true so nothing changes for other models, set false for Flash-Next (953820a0; tool-calling requests keep the tools preset).
TD benchmark
17.0 → 16.0 tok/s; a complete, wired 51 KB game ("Ironline") with a single typo, and a self-repair that changed exactly that token — the same behaviour config A showed on llama.cpp in August.
Defects
Eight fix rounds, each about twenty minutes, each repairing exactly what the brief named and nothing else: the typo; the intro state that never left "intro"; a path stored in cell units but consumed in pixels; a HUD that was never written; a 0/0 kill reward; a road that was buildable because a duplicated waypoint made the path distance NaN. Every unrequested edit the model made introduced a regression (a stray brace, a variable name "corrected" in a sibling function three separate times), and one round emitted end-of-sequence in the middle of the file. The lesson that carried forward: literal briefs, one change each, and name the lines the model must not touch.
| Round | Tokens | tok/s | Outcome |
| R1 / R2 | 18,954 / 18,953 | 17.0 / 16.0 | 51.2 KB, one fatal typo |
| fix1 · console error | 18,980 | 15.5 | byte-identical except the one token |
| fix2 · runtime brief | 19,014 | 16.0 | state fix correct, stray } added |
| fix3 · console error | 30,334 | 14.1 | one statement changed, 11K tokens of extra output |
| fix4 · runtime brief | 19,073 | 15.6 | fixed the two lines named, ignored the other consumers |
| fix5 · literal brief | 19,318 | 15.7 | playable; kill reward 0/0 |
| fix6 · runtime brief | 19,259 | 15.9 | NaN fixed; unasked edit re-broke the typo |
| fix7 · console error | 19,046 | 15.7 | single-token diff; fully playable |
| fix8 · literal brief | 19,012 | 14.1 | road fix correct; unasked edit reverted by hand |
Iteration 4 — ATLAS4: precision, and a rollback that was wrong on both sides
Analysis + comparison
The checkpoint ships attention projections and the lm_head in BF16; Atlas requantised both to NVFP4 at load, while llama.cpp keeps them at Q8_0. The repo's own 27B recipe pins --lm-head-dtype bf16 "for exact argmax". The rollback item from iteration 2 still resumed one token behind the snapshot with the watchdog on (pos 6050 but 6049 ingested).
Proposed changes + decisions
- BF16 lm_head by default for this model (
61f1993b); a new --attn-proj-dtype bf16 flag that hands the layer no NVFP4 Q/K/V so the resident BF16 sources feed the dense arms (c120f479) — ModelOpt's own ignore list keeps every self-attention projection at BF16.
- Rollback, twice. The first fix popped one extra token from the sequence buffer and produced the mirror error (
pos 8535 but 8536). The real mechanism: the two watchdog sites sit on different sides of the step's token push, so no fixed pop count is right — rewind_buffers now takes the caller's unfed tail; and the content handler had been appending the token it sampled from the discarded context, a phantom that reached the client and shifted every later rollback by one (6c5b61c3, 7db64cca).
TD benchmark
15.2 → 14.0 tok/s. With the watchdog on, the same build rolled back and re-steered three times with zero indexer desyncs — and then the loop detector false-positived on a coordinate array and cut a coherent file at 8.4K tokens. It stays off for code. The speed drop is the precision trade, later measured per half at long context: BF16/BF16 13.8, BF16 head with NVFP4 attention 15.0, NVFP4 head with BF16 attention 14.7, both NVFP4 15.6 tok/s.
Defects
A 57.7 KB game with zero syntax errors and one logic bug: killed and leaked enemies were never removed from the enemy list, so a wave never registered as cleared. Placement, road and combat all worked.
Iteration 5 — ATLAS5: the gap study, and three items that did not move the needle
Analysis + comparison
Two investigations by the prod model: first an Atlas-only decode anatomy, then Atlas versus llama.cpp side by side with both trees cited by line. The picture it drew: llama.cpp keeps every data-dependent decision on the device (MoE ids through the mul_mat_id fast path, the indexer top-k as a device sort), has zero host synchronisations per token, keeps the n-gram table in host memory the unified GB10 serves directly, and carries a chunked Gated-DeltaNet kernel for prefill; Atlas had 48 stream synchronisations per token and vetoed CUDA graphs model-wide. The report's one shaky claim — that llama.cpp's graphs were compile-gated — was wrong (the prod build has them on), so the runtime question was measured: GGML_CUDA_DISABLE_GRAPHS=1 costs llama.cpp about 3%. Capture went to the bottom of the plan.
Before A3, a memory check: a resident 51 GB table cannot fit for either engine (Atlas sits at 115.6 of 121.7 GB after load; llama.cpp has only ~10 GB of its 25.6 GB table in page cache). What matters is the per-sequence working set, at most 16 rows of 160 bytes per token — a 4M-slot cache is 640 MB.
Proposed changes + decisions
Execute A3, A2, A1 with the executor from anchored briefs, verify each contribution separately, then benchmark. A3 (event-scoped n-gram pin release, pinned slot staging; 69cb6625) needed two executor passes. A2 (ATLAS_MOE_DECODE_ARM=grouped, routing every layer's single-token MoE through the tensor-core arm; ad7203f5) was done in one. A1 (device-side indexer selection) produced no change after 33 executor steps and was written by the reviewer: a qsa_select_topk kernel with torch's tie-break, behind ATLAS_QSA_DEVICE_TOPK=1, with a parity mode that runs both arms and fails on the first mismatch (0b9ada13).
TD benchmark
| Item | Measured |
| A3 — n-gram barrier + staging | neutral (14.5 / 15.1 tok/s): the table sits on one model layer, one barrier per token, not 36 |
| A2 — tensor-core MoE arm at M=1 | 3× slower (5.3 vs 15.0 tok/s); scalar arm stays |
| A1 — device-side selection | parity clean over 1,400 tokens; unchanged below the 2,051-token inert bound; +3–5% once selection is active (13.1/13.5 → 13.8/13.9) |
| ATLAS5, all three on | 14.7 → 13.9 tok/s on 23K-token rounds |
Host synchronisation, table placement and graph capture together were worth a few percent. The gap had to be inside the decode kernels.
Defects
A 62 KB game ("IRONGATE") that played out of the box — briefing, road rules, three tower types, waves, kills, win and lose — except that its stats panel was refreshed only on shop clicks. One two-line fix round.
Iteration 6 — ATLAS6: measure, then act on the two largest stages
Analysis + comparison
One nsys window over 40 seconds of pure decode (642 tokens, BF16 head, NVFP4 attention, device top-k, 3,882-token prompt, ~69 ms per token). The profile:
| Stage | Launches / token | ms / token | Reading |
dense_gemv_bf16 | 73 | 24.0 (42%) | one is the BF16 lm_head; 72 are the 36 GDN layers' in/out projections, which the loader keeps in BF16: 4.15 GB/token, bandwidth-bound at that precision |
hyper-connections (hc_pre_finish and siblings) | 388 | 17.5 (27%) | ~1.25 GB of weights whose floor is ~5.4 ms; stage 3 alone 121 µs × 97 |
| MoE (gate/up, silu/down, top-k, blend) | ~100 | 8.3 (14%) | 1.46 GB at ~176 GB/s ≈ 77% of achievable: already near its floor |
| everything else | ~450 | ~6 | recurrence, attention, selection, norms |
Analytically Atlas moved more bytes per token than llama.cpp (8.5 GB vs 7.6 GB) because those projections never got quantised, and spent a further ~12 ms in the hyper-connection kernels. The prod model's own reading of the same profile misattributed the MoE term (it recomputed with the wrong intermediate size) and proposed a GEMV rewrite worth "26 ms"; the numbers overruled it.
Proposed changes + decisions
- A flag, not code:
ATLAS_QWEN4EXP_BF16_GDN=0 requantises the GDN projections to NVFP4 through a loader path that already existed. Its comment said GDN bandwidth "is simply not what bounds decode here" — written when the whole engine ran at 2.2 tok/s and something else did. 15.0 → 18.3 tok/s, for ~6 GB of memory (context 64K → 48K at 0.95 utilisation).
- A kernel:
hc_pre_finish gave every thread all four hyper-connection streams of one output dim, capping the kernel at 2,560 threads walking four dependent rank-320 chains. The maintainers had already rejected a warp-shuffle rewrite because it reassociates the FP32 sum (they had measured it moving logits). hc_pre_finish_x4 gives each of the four independent accumulators its own warp, lanes over 32 dims — 4× the threads, identical summation order under the module's --fmad=false: greedy output hash identical to the previous binary, 18.3 → 20.3 tok/s (e7b6af2f). An unroll-only attempt first (3fff25af) was neutral, which is what told me the kernel was thread-starved rather than load-starved.
- Measured and rejected:
ATLAS_GDN_FUSED_NORM=1 — no speed change and the greedy output differs; spin-sync and n-gram fault-thread depth — neutral; FP8 / NVFP4 lm_head — +1 / +1.5 tok/s at the documented argmax-flip risk, not adopted. The selection kernel's scores staged in shared memory (22321152) — bit-exact, neutral.
TD benchmark
21.6 → 20.3 tok/s on 20.6K-token rounds: +47% over ATLAS5, past ATLAS3's 17.0 with the quality precision kept on the head. llama.cpp on the same box: 24–25.
Defects
The 52.9 KB game ("IRONWORKS · TD") shipped with two: the reset code called a helper by the wrong name (buildStatic for bakeStatic), so the render loop never started, and the turret loop used a level variable it never bound. A console-error round echoed the file unchanged; two literal-brief rounds made exactly the one-token rename and the two bindings. Then four waves cleared with correct gold, score and lives. Two defects in one sample versus one for ATLAS5: the quality gate on the NVFP4 GDN requantisation is neither passed nor failed.
The decision log, entry by entry
The same six iterations as a log: decision, commits and the files each touched, impact, what was rejected and what is left. This is the record kept on the benchmark page; the PR carries the same for its reviewers.
ATLAS — 2026-09-03 · main @ 4df6e609
Serve the RadixArk NVFP4 checkpoint on stock Atlas2026-09-03R1 17.3 tok/s · 11 serve attempts
Decision
Run the same two-round TD benchmark llama.cpp had run in August, on Atlas main, and record every failure rather than tune around it.
What happened
Five distinct engine defects over eleven serve attempts: watchdog rollback desyncing the QSA indexer (HTTP 500), MTP verify with QSA active ending the response as a clean stop (silent truncation), prompts over 8,192 tokens refused twice at the PLE layer (cap off by one, then a 65,536-slot row cache), continuation prefill chunks faulting with CUDA_ERROR_MISALIGNED_ADDRESS and a zombie process holding 100 GB, and ATLAS_QSA_MAX_TOKENS defaulting to 32,768 regardless of context.
Workarounds used
ATLAS_PLE_MAX_TOKENS, ATLAS_QSA_MAX_TOKENS, ATLAS_PLE_CACHE_SLOTS env pins, single-chunk prefill (--max-prefill-tokens 20480), --content-loop-watchdog false, no speculative decoding.
Output
Game "IRONGATE" needed a second fix round to boot. Speed 17.3 tok/s single stream, ~300 tok/s prefill.
Files
no engine changes; scripts
~/driver_td2*.sh,
~/tdbench_r2.py,
~/tdfix3.py on the bench machine; report
bench report.
ATLAS2 — 2026-09-04 · fixed engine, default settings
Fix the five defects on a branch, then run with no env pins2026-09-04R1 17.3 → R2 16.7 tok/s
Decision
Two independent investigations (prod model and config A) produced a seven-item plan; execute all seven with the config-A executor on one branch, then re-run the benchmark on defaults.
Commits
e623e010 PLE row cache derived from max_batch_tokens × ngram_heads, env stays an override; cap clamped up with a warning — crates/spark-model/src/weight_loader/qwen4_exp/ple.rs, qwen4_exp.rs
561be215 QSA indexer capacity from --max-seq-len (ModelConfig.max_seq_len) — crates/atlas-core/src/config.rs, config/factory.rs, spark-server/src/main_modules/serve_load.rs, spark-model/src/layers/qsa.rs, attach.rs
16af4d33 engine errors reported as finish_reason="error" (500 for blocking clients) instead of stop; cuBLASLt failure logged with M/N/K — scheduler/types.rs, lifecycle.rs, ir/response.rs, anthropic/translate.rs, qwen3_ssm/trait_prefill_proj.rs
35028544 scheduler breaks on a latched CUDA fault instead of holding 100 GB and 500-ing — scheduler/mod.rs
f4e81291 watchdog rollback saves/restores the QSA/PLE aux state with the SSM snapshot, declines when missing — traits/model.rs, model/types.rs, impl_a1.rs, model/trait_impl/mod.rs, scheduler/rollback.rs
86bdf303 MTP latched off before a sequence crosses the QSA inert bound — scheduler/mtp_gate/qsa_latch.rs, scheduler/mod.rs, multi_seq/mod.rs
97aa9fa4 hyper-connection prefill scratch laid out on 256-byte boundaries (odd slabs faulted the GEMM) — spark-runtime/src/buffers/sizes.rs, buffers.rs, ops/hyper_connection_lowrank.rs
d32605b2 merge of upstream main; cb6b1a59 test fix
Impact
Zero server errors across the run on default chunking with no env pins. Speed unchanged (the fixes remove failure modes, not cost). The game "BULWARK" was inert (no boot loop, no listeners): the quality problem was not the engine.
ATLAS3 — 2026-09-04 · sampling equalised
Pin the request to llama.cpp's effective sampling2026-09-04R1 17.0 → R2 16.0 tok/s
Decision
Found in spark-server/src/api/chat/sampling_setup.rs: Atlas fills unset request fields from the model card's [sampling.non_thinking] preset, including presence_penalty 1.5. Over a 20K-token file that suppresses every token already emitted. Re-run with presence 0, top_k 40, min_p 0.05 on the same engine.
Impact
A complete, wired 51 KB game ("Ironline") with one typo; the identical engine had produced dead/inert files twice. Then eight fix rounds: the model repairs exactly what a brief names, and every unrequested edit it made introduced a regression (fix2, fix6, fix8 each re-broke one token; one round emitted EOS mid-file; one echoed the file). Playable after fix7; a road-placement bug (path distance NaN from duplicate waypoints) fixed in fix8 with one reviewer token reverted. Per-round data on the bench page.
Commit
953820a0 [behavior].use_sampling_preset_penalties_for_core (default true, Flash-Next opts out) — crates/atlas-kernels/build_parse_behavior.rs, build.rs, build_codegen.rs, src/lib.rs, spark-server/src/api/chat/sampling_setup.rs (+ tests), kernels/gb10/qwen3.8-flash-next/MODEL.toml
ATLAS4 — 2026-09-04 · precision and rollback
BF16 head and attention for quality; make the watchdog rollback correct2026-09-04R1 15.2 → R2 14.0 tok/s (watchdog off)
Decision
Attribute the remaining quality gap to precision: the checkpoint ships attention and lm_head in BF16 and Atlas requantised both to NVFP4 at load. Keep the head BF16 by default and add a flag for BF16 attention projections. Finish the rollback fix (the earlier item left a one-token lag).
Commits
61f1993b BF16 lm_head default for qwen4_exp — crates/atlas-core/src/config/methods.rs
c120f479 --attn-proj-dtype bf16 — atlas-core/src/config.rs, spark-server/src/cli/{serve_args,validate,flag_values}.rs, main_modules/serve_load.rs, spark-model/src/weight_loader/qwen35/load_layers/attention_arms.rs
6c5b61c3 + 7db64cca rollback rewinds by fed tokens (the two watchdog sites sit on different sides of the step's push) and drops the token sampled from the discarded context — scheduler/rollback.rs, rollback_tests.rs, decode_logits_content.rs, decode_logits_step.rs
Impact
Rollback verified on the GPU: five rollback + re-steer events, zero indexer desyncs. Two side findings: the content-loop watchdog false-positives on code (it cut a coherent file at a coordinate array), so it stays off for code; and each BF16 choice costs 5–6% decode (both ≈ 12%). Game: 57.7 KB, zero syntax errors, one logic bug (killed enemies never removed). The speed drop from ATLAS3 is the precision trade, measured later per half: BF16/BF16 13.8 · BF16/NVFP4 15.0 · NVFP4/BF16 14.7 · NVFP4/NVFP4 15.6 tok/s at 3.9K context.
ATLAS5 — 2026-09-04 · gap study executed
Compare the two engines from source, execute the three cheapest items2026-09-04R1 14.7 → R2 13.9 tok/s
Decision
Two investigations by the prod model (Atlas-only decode anatomy, then Atlas vs llama.cpp side by side,
gap study) proposed six items. Measured first: CUDA graphs are worth ≈3% to llama.cpp (
GGML_CUDA_DISABLE_GRAPHS=1: 24.8/25.1 → 24.4/24.1), so capture was dropped to last. Execute A3, A2, A1 with the executor; verify each contribution.
Commits
69cb6625 A3: event-scoped PLE pin release + pinned slot staging (the PLE is one layer, so one barrier per token) — spark-runtime/src/gpu.rs, cuda_backend/gpu_impl.rs, spark-model/src/layers/ple/{aux_state,layer,verify}.rs — neutral
ad7203f5 A2: ATLAS_MOE_DECODE_ARM=grouped selector — spark-model/src/layers/moe/forward.rs, MODEL.toml — grouped tensor-core arm at M=1 is 3× slower (5.3 vs 15.0 tok/s); scalar stays
0b9ada13 A1: device-side QSA block selection qsa_select_topk behind ATLAS_QSA_DEVICE_TOPK=1, parity mode ATLAS_QSA_TOPK_VERIFY=1 — kernels/gb10/qwen3.8-flash-next/nvfp4/qsa_indexer.cu, spark-model/src/layers/ops/qsa.rs, layers/qsa.rs, new layers/qsa_decode_select.rs — parity clean; +3–5% only once the sparse selection is active (past 2,051 tokens)
Impact
Decode unchanged on the full benchmark: host synchronisation, PLE placement and graph capture together are worth a few percent. Game "IRONGATE" (62 KB) played out of the box except a stats panel refreshed only on shop clicks; one two-line fix round. Conclusion recorded: the gap is in the decode kernels.
ATLAS6 — 2026-09-06 · measured, not reasoned
Profile one decode window with nsys, act on the two largest stages2026-09-06R1 21.6 → R2 20.3 tok/s
Decision
Stop estimating. 40 s of pure decode under nsys (642 tokens, ~69 ms each): dense_gemv_bf16 73 launches = 24 ms (42%) — 72 of them the 36 GDN layers' in/out projections, which the loader keeps in BF16 (4.15 GB/token, bandwidth-bound); hyper-connection stage 3 hc_pre_finish 97 × 121 µs = 11.7 ms (27% with its siblings); MoE ≈ 8.3 ms at ~77% of achievable bandwidth (already near its floor, so the proposed GEMV rewrite was dropped). The prod model's own reading of the same profile misattributed the MoE term and was overruled by the numbers. Profile: reports/atlas-flashnext-decode-profile-2026-09-06.md.
Changes
- Flag
ATLAS_QWEN4EXP_BF16_GDN=0: requantise the GDN projections to NVFP4 through the loader path that already existed (its comment, written at a 2.2 tok/s era, said GDN bandwidth "is not what bounds decode") — 15.0 → 18.3 tok/s; costs ~6 GB (context 64K → 48K at 0.95 utilisation)
e7b6af2f hc_pre_finish_x4: one warp per hyper-connection stream, lanes over 32 output dims (4× the threads), identical FP32 accumulation order under --fmad=false — greedy output hash identical to the previous binary; 18.3 → 20.3 tok/s — kernels/gb10/qwen3.8-flash-next/nvfp4/hyper_connection.cu, spark-model/src/layers/ops/hyper_connection_lowrank.rs, hyper_connection_lowrank_gemm.rs (ATLAS_HC_FIN_X4=0 reverts)
3fff25af hc_pre_finish rank loop unrolled ×8 — neutral (the kernel was thread-starved, not load-starved; kept as documentation)
22321152 qsa_select_topk scores staged in shared memory — neutral on tok/s, bit-exact
Measured, rejected
ATLAS_GDN_FUSED_NORM=1 (recurrence + gated norm fused): no speed change and the greedy output differs. ATLAS_D2H_SPIN_SYNC, ATLAS_PLE_FAULT_THREADS=128: neutral. FP8 / NVFP4 lm_head: +1 / +1.5 tok/s on top, at the documented argmax-flip risk; not adopted.
Impact
Full benchmark 21.6 / 20.3 tok/s (+47% over ATLAS5, ATLAS3 was 17.0 on NVFP4 head and attention). Game "IRONWORKS · TD" shipped with two defects (a helper called by the wrong name, an unbound level variable in the turret loop); a console-error round echoed the file, two literal-brief rounds fixed exactly what was named; then four waves cleared cleanly. Quality gate for the NVFP4 GDN requantisation: two defects in one sample versus one for ATLAS5 — not conclusive.
Left on the table
BF16 lm_head ≈ 6 ms/token (precision decision); hc_pre_down and the other hyper-connection launches ≈ 5 ms; conv1d launched separately from the GDN recurrence ≈ 1 ms (the existing fused kernels are not bit-exact); ~1,000 launches/token of gaps. llama.cpp: 24–25 tok/s.
Totals: main as found to ATLAS6
Same model, hardware and prompts throughout. The first row is the engine as it was on main before any change; the last is the head of the pull request.
| R1 tok/s | R2 tok/s | vs llama.cpp (24.8) |
| ATLAS, main as found | 17.3 | 16.3 | 70% |
| ATLAS4/5, BF16 head + BF16 attention | 15.2 | 14.0 | 61% |
| ATLAS6, final | 21.6 | 20.3 | 87% |
Net against main: +25% on both rounds. Against the BF16-precision configuration the last two changes were measured from: +44%. Per-token cost on the 3.9K-token single-prompt A/B went from 69 ms to about 49 ms.
Where the speed came from
| Change (single-prompt A/B, bit-exact unless noted) | tok/s | Gain |
| Base: BF16 head, NVFP4 attention, BF16 GDN | 15.0 | |
| GDN projections requantised to NVFP4 | 18.3 | +22% |
hc_pre_finish_x4 kernel | 20.3 | +11% |
| Device-side QSA top-k (A1) | — | +3 to 5% while sparse selection is active |
| FP8 or NVFP4 head | 21.2 / 21.7 | +4 to 7%, not adopted |
Tried and rejected: the grouped MoE decode arm (3x slower), the fused GDN norm (not bit-exact), the 4M-slot PLE cache with event-scoped pins (neutral, kept as harmless), D2H spin sync and PLE fault threads (neutral).
Reliability and correctness
| Parameter | Main | Final |
| Serve attempts to get one complete run | 11 | 1 |
| Server errors during a benchmark | HTTP 500s; engine errors laundered into a clean stop | 0; engine errors surface as errors |
| Prompts over 8,192 tokens | refused | accepted with the default 8K chunked prefill |
| Loop-watchdog rollback | desynced the sparse-attention indexer, HTTP 500 | works, 0 desyncs on the GPU probe; left off for code because it false-positives on coordinate arrays |
| Sampling vs llama.cpp | presence penalty 1.5 applied silently from the model preset | equalised; preset penalties opt-in |
| Tower-defense game | dead on load | playable |
Cost. Resident memory rose from about 100 to 108 GB with NVFP4 GDN, which cut the KV budget from 21 to 14 GB and the practical context from 64K to 48K on this box.
Other models. The ten regression gates the repository runs on other models (Qwen3.6-35B-A3B-FP8, Qwen3.6 and 3.8 27B NVFP4) all pass at the PR head, with decode-floor at 22.7 tok/s. They did not at first: the BFCL function-calling subset on the 27B missed its bar by 0.22 normalized points, and main's own binary and harness scored the identical 86.23 / 83.10 on the same box and draw. Bisecting from the last passing tree to main found the cause in one file: an upstream commit had changed the greedy tie-break rule in the argmax kernel from lowest index wins, the rule llama.cpp and vLLM use, to highest index wins, and BF16 logits at temperature 0 tie often. Reverting that file alone restored every one of the nine flipped samples, on which the old rule answered five correctly and the new rule two. The PR now restores first-index-wins in the kernel and the two host paths, and the full gate draw came back at 86.53 / 83.38, the earlier record to the digit. The concurrency sweep needed a second run because one prompt looped under greedy decoding and the watchdog cut it below the sweep's delivery floor; on the re-run every floor was met.
Still open. The remaining 13% to llama.cpp is the BF16 head (about 6 ms per token), the other hyper-connection launches (about 5 ms), the separate conv1d launch (about 1 ms) and roughly 1,000 launches per token of gaps.
A long-horizon check: four hours of agentic work on the PR head
Speed on a fixed prompt is one thing; an engine also has to hold up when a model works with tools for hours, carrying 20 to 50 thousand tokens of context into every turn. So the last test on the PR head was agentic and phased: build an idle RPG in Rust compiled to WebAssembly with a Yew interface, from a formalised specification, in seven phases that follow a test-driven plan: formalise the mechanic math; write the public interfaces and a failing test suite; implement the battle core; implement shop, events and save; build the Yew components; integrate the game mechanics; test and harden. One Atlas session at a 98K context served all seven phases. The harness was rebuilt after a first attempt at the same task: each phase now opens with an orientation packet instead of a read loop, reads are budgeted, edits outside the phase's scope need a logged reason, and an independent verify tool reports tests, compile errors, clippy, the wasm check and module wiring after every phase. Every prompt, step trace, tool transcript and crate snapshot is published in the report.
What the engine showed. Native tool calling held for four hours and 118 requests, every one finishing on a stop token, with no engine error at the 98K context. Decode stayed between 18.5 and 20.8 tokens per second median across the phases and between 19.6 and 22 on long generations, including a single 12,155-token file. Time to first token was the cost: 55 seconds median at 20 thousand tokens of context, 122 to 138 seconds at 35 to 50 thousand, with a worst case of 256, which is roughly 330 to 380 tokens per second of effective prefill in 8K chunks. Over the run the model consumed about 3.06 million input tokens to produce 70 thousand. One new engine finding: the chat-level loop detector judged three similar file reads a loop and masked the tool-call token for a turn; agentic serving should disable it with the loop-suppression environment flag.
What the model showed. The three failure modes of the first attempt were gone: phase one made a single read instead of nine to fourteen, phase two declared every module and the verify tool collected 44 tests, and no phase drifted back to the balance test. What replaced them was compile-error gravity. Phase three spent its budget on one 12 thousand token battle module, regenerated it after judging the first version corrupted, and left six compile errors. Phases four and five, whose own scopes were the shop and the interface, spent thirty-two steps on those errors through logged scope overrides, and one lifetime fix cascaded into twelve new errors and was reverted. Phase six, with a fresh context and the same tools, fixed the crate in eleven steps, chose a fix that does not cascade and explained why, and left it compiling natively and for wasm with 17 of 44 tests passing. Phase seven found one real bug and two contradictory constants and ended with 19 passing. No interface was written and no game shipped. The honesty was good in all seven phases: every summary states what was not done and what is still red.
What the harness learned. The orientation packet, the wiring check and the scope fence each did what they were built for, and the fence turned an invisible drift into a logged decision without being able to prevent it. The next gate is obvious in hindsight: a phase must not advance while the crate does not compile. Single writes should be capped at about four thousand tokens, because a ten-minute generation that the model then discards is the most expensive thing that happened in the run. And a fresh context beat persistence, so the harness should reset rather than push on when a phase stalls on the same errors. One harness bug surfaced live, an empty file path that escaped the tool node, and was fixed between phases.
Eighteen more hours, and the game. The crate from that run was then continued with a fourth harness for ten hours, extended by eight at my request: a compile gate that follows any round leaving compile errors with a fresh-context repair round, writes capped at four thousand tokens with an append tool, a git-backed revert, and reads limited to the region about to change. Over 21 measured rounds the model consumed 14.9 million input tokens for 117 thousand output tokens, at a decode rate that never left the 17.6 to 20.6 tokens per second band, while time to first token ranged from 39 seconds in the small-context repair rounds to 218 seconds median and 301 seconds worst in the phases that carried 45 to 56 thousand tokens. The model finished the core to 41 of 44 tests, wrote a self-driving tick engine, and produced the first browser build of the whole effort, but never connected the interface to the engine: the rounds that could have done it went to the three failing balance tests instead, three times in a row despite a written deferral, until the scope fence put the core out of reach. I then wired the interface by hand in about forty-five minutes, labelled as mine in the report, and found two loop bugs on the way: kills banked gold but never awarded experience, and fights restarted on the beat they ended so travel events never rolled. The game is playable at the bench site, and the full account, with every prompt, transcript and per-round crate, is in the second report. The engine conclusion is a single number: on this stack an agent should keep its per-turn context under about twenty-five thousand tokens, because above that it spends its budget waiting for its own history to be re-read.
Reflection: what moved the number and what did not
- Measuring beat reasoning, twice. Two careful source-level investigations proposed six items; the three cheapest were executed and moved decode by a few percent. One nsys window found the 42% term in an afternoon. The order should have been the other way round, and the cost of finding that out was a day.
- Comments are dated measurements. The loader note that dismissed NVFP4 GDN projections was true when written and false by the time it mattered. So was the assumption, in the agent report, that the n-gram layer ran on 36 layers rather than one.
- Bit-exactness as the gate kept the wins honest. Every kernel change was hashed against the previous binary before its speed counted; the one fused-kernel flag that failed the hash was dropped even though it looked free. The x4 kernel exists because that constraint forced a design that parallelises independent accumulators instead of splitting one.
- What the models could and could not execute. The prod model wrote good investigations and the plumbing items from anchored briefs (A3, A2), and produced nothing for the kernel items (A1, the hyper-connection kernel), which the reviewer wrote. In the game rounds it repaired precisely what a brief named and regressed on everything it touched unasked — a property worth designing briefs around rather than fighting.
- Precision is a product decision, not a benchmark one. Each BF16 choice on this model costs 5–6% decode; the head is where argmax flips live and is worth keeping; the attention projections were the pricier half for little evidence of gain; NVFP4 GDN bought 22% and has one worse and one better game to its name. Two more full runs would settle it; one cannot.
- What is left between 21.6 and 25. The BF16 lm_head (~6 ms/token),
hc_pre_down and the remaining hyper-connection launches (~5 ms), the conv1d step still launched separately from the recurrence (~1 ms, the existing fused kernels are not bit-exact), and ~1,000 launches per token of gaps. None of it is a flag; all of it is the kind of kernel work iteration 6 did.
Code: Atlas-Inf/atlas#23 (draft, 25 commits). Benchmark data and every game: the shootout page. The August baselines this compares against: the tower-defense build-off and the agentic-flows benchmark.