← Andrei Vogulkin · all writing

Field notes · 2026-09-03 → 2026-09-06 · Atlas × Qwen3.8-Flash-Next · NVIDIA DGX Spark (GB10)

Closing the gap to llama.cpp on a desk-side GB10 — six Atlas iterations, measured

Four days of serving one model on two engines and refusing to guess. Per iteration: the analysis and the comparison, the changes proposed and the decisions taken, the tower-defense benchmark, and the defects the model shipped. From 15 to 21.6 tokens per second against llama.cpp's 24–25, with every engine change proven bit-exact before its speed was written down.

The setup, in one paragraph

The model is Qwen3.8-Flash-Next: 48 layers, 36 of them Gated-DeltaNet, 12 full attention with a sparse indexer, a 512-expert MoE on every layer, a 51 GB n-gram embedding table read by row, and four-stream hyper-connections around every block. Prod serves it with llama.cpp (Unsloth UD-Q4_K_XL GGUF) at 24–25 tok/s on a DGX Spark. The candidate engine is Atlas (Rust + CUDA) with the RadixArk NVFP4 checkpoint. The benchmark is the tower-defense build-off from August: two ~20K-token rounds, identical prompts, and then fix rounds driven by console errors or a runtime description, exactly as the llama.cpp configurations were scored. The rules that held throughout: one investigator/reviewer, the models on the GB10 execute; every repo change goes through one PR; an engine change is measured only after a greedy 160-token generation on the previous binary and the new build hash the same.

IterationEngine headWhat changedR1 / R2 tok/sGame
ATLASmain @ 4df6e609nothing — find the failures17.3 (11 attempts)needed a fix round to boot
ATLAS2cb6b1a59five engine defects fixed, defaults restored17.3 → 16.7inert
ATLAS3cb6b1a59request sampling equalised with llama.cpp17.0 → 16.0playable after 8 fix rounds
ATLAS47db64ccaBF16 head + attention; rollback rework15.2 → 14.01 logic bug
ATLAS50b9ada13gap-study items A1–A314.7 → 13.9playable after 1 fix round
ATLAS6e7b6af2fprofile-driven: NVFP4 GDN, x4 hyper-connection kernel21.6 → 20.3playable after 3 fix rounds
llama.cpp (prod)PR #27742 build—24–25playable after 1 fix round (August)
Sub-articles (the working documents, verbatim): bench report — eleven attempts and five defects · two independent defect investigations · fix plan · Atlas decode anatomy · Atlas vs llama.cpp gap study · handover briefs A1–A3 · the nsys decode profile and the optimisation search. Live pages: benchmark tables and every game. On this page: game versions and references · serve command per game · totals · agentic run · the benchmark prompts and briefs.

The games: versions and references

Same table as the August build-off, extended with the six Atlas iterations. Every run is one complete HTML5 tower-defense game built from identical prompts; the August rows are the reference the Atlas rows are measured against. "Play" opens the final published version; the version table underneath lists every round that produced a file, what the model was given for it, and where that intermediate version can be opened.

ConfigModelEngine & servingMemorytok/s R1→R2Build timeGame sizeOutcome
P
prod (August)
Huihui-Qwen3.8-27B-abliterated
27B dense · Q4_K_M GGUF
llama.cpp DFlash build (PR #27342)
DFlash2 speculative decoding · Sharp template
17 GB46.6 → 49.0~9 min35.8 KB✅ works functional, plainest designPlay
E2
August
Qwen3.8-Flash-Next
Unsloth UD-Q3_K_XL (84 GB)
llama.cpp PR #27742 · PLE → CPU/mmap · 64K ctx · f16 KV~89 GB24.6 → 20.0~32 min56.1 KB✅ works most polished gamePlay
A
August · prod today
Qwen3.8-Flash-Next
Unsloth UD-Q4_K_XL (111 GB)
llama.cpp PR #27742 · PLE → CPU/mmap · 128K ctx · q8_0 KV~111 GB23.4 → 19.4
fix: 19.4
~31 + 16 min50.2 KB⚠️→✅ works 1 fatal typo, self-fixed from the console errorPlay
ATLAS
main
Qwen3.8-Flash-Next
RadixArk NVFP4 (126 GB on disk)
Atlas 4df6e609 · single-chunk prefill (20K) · 48K ctx · watchdog off · 11 serve attempts~100 GB17.3 → 16.3
fix: 16.6, 16.4
~17 + 27 min
+ 2 fix rounds
47.8 KB⚠️ dead → loads 4 unclosed script blocks; loads after 2 fix rounds, play-through unverifiedPlay
ATLAS2
fixed engine
Qwen3.8-Flash-Next (same)Atlas branch 97aa9fa4 (7 fixes) · default 8K chunked prefill · 64K ctx · no env pins~100 GB17.3 → 16.7~9 + 14 min38.2 KB⚠️ inert "BULWARK": complete file that never boots (no init call, no frame loop, no listeners)Open
ATLAS3
+ llama.cpp sampling
Qwen3.8-Flash-Next (same)Atlas branch cb6b1a59 · request pins presence 0 · top_k 40 · min_p 0.05~100 GB17.0 → 16.0
fix: 15.5
~19 + 20 min
+ 8 fix rounds
51.2 KB✅ playable after 8 fix rounds "Ironline": one typo, then four logic bugs, one regression per roundPlay
ATLAS4
rollback rework · BF16 head + attention
Qwen3.8-Flash-Next (same NVFP4 experts)Atlas branch 7db64cca · BF16 lm_head + --attn-proj-dtype bf16 · watchdog off~102 GB15.2 → 14.0~22 + 27 min57.7 KB⚠️ one logic bug "Bastion Defense": everything works except dead enemies are never removed, so no wave clearsOpen
ATLAS5
gap-study items A1–A3
Qwen3.8-Flash-Next (same NVFP4 experts)Atlas branch 0b9ada13 · + device QSA top-k · 4M-slot PLE cache · event-scoped pin release~102 GB14.7 → 13.9~26 + 29 min
+ 1 fix round
62.2 KB✅ playable after 1 fix round "IRONGATE": zero errors on load; stats panel never refreshedPlay
ATLAS6
NVFP4 GDN + x4 hyper-connection kernel
Qwen3.8-Flash-Next (same NVFP4 experts)Atlas branch e7b6af2f · ATLAS_QWEN4EXP_BF16_GDN=0 · BF16 head · NVFP4 attention · hc_pre_finish_x4~108 GB21.6 → 20.3~16 + 17 min
+ 3 fix rounds
52.9 KB✅ playable after 3 fix rounds "IRONWORKS · TD": wrong helper name, unbound level variable; then four waves clearedPlay

Serve command per game

Every Atlas round used the same line; the head, the env pins and the extra flags differ per iteration and are listed against each game. Prod is stopped first because the 128 GB of unified memory is one pool.

The serve line (Atlas rounds)
ATLAS_PLE_MAX_TOKENS=8212 <env pins> ./target/release/spark serve \
  --model-from-path ~/models/radixark-fn-nvfp4 --model-name qwen --kernel-target qwen3.8-flash-next \
  --bind 127.0.0.1 --port 8001 --max-seq-len <SEQ> --max-num-seqs 1 --max-batch-size 1 \
  --gpu-memory-utilization 0.95 --fast-load-prefetch-shards --max-prefill-tokens 8192 \
  --request-timeout 3600 <extra flags>
ConfigHeadEnv pins--max-seq-lenExtra flagsRequest sampling pinsResult
ATLAS
Play
4df6e609 (main)ATLAS_QSA_MAX_TOKENS=65536 ATLAS_PLE_CACHE_SLOTS=524288 ATLAS_PLE_MAX_TOKENS=2050049152--max-prefill-tokens 20480 --content-loop-watchdog falsenone (Atlas model preset applies: presence 1.5, top_k 20)17.3 / 16.3; the only line that survived 11 attempts (single-chunk prefill sized to the whole prompt)
ATLAS2
Play
97aa9fa4— (PLE cap derived from --max-prefill-tokens: 8212)65536--content-loop-watchdog falsenone17.3 / 16.7; default 8K chunked prefill, zero server errors
ATLAS3
Play
cb6b1a59—65536--content-loop-watchdog falsepresence_penalty 0 · top_k 40 · min_p 0.0517.0 / 16.0; sampling equalised with llama.cpp
ATLAS4
Play
7db64cca—65536--attn-proj-dtype bf16 (BF16 lm_head is the new default)same pinswatchdog ON first: R1 cut to 8.4K tokens by three false-positive rollbacks on a coordinate array; published run with --content-loop-watchdog false: 15.2 / 14.0
ATLAS5
Play
0b9ada13ATLAS_QSA_DEVICE_TOPK=1 ATLAS_PLE_CACHE_SLOTS=419430465536--attn-proj-dtype bf16 --content-loop-watchdog falsesame pins14.7 / 13.9; device-side QSA selection, 4M-slot PLE row cache, event-scoped pin release
ATLAS6
Play
e7b6af2f (fix rounds at 22321152)ATLAS_QWEN4EXP_BF16_GDN=0 ATLAS_QSA_DEVICE_TOPK=1 ATLAS_PLE_CACHE_SLOTS=419430449152--lm-head-dtype bf16 --attn-proj-dtype nvfp4 --content-loop-watchdog falsesame pins21.6 / 20.3; NVFP4 GDN projections (+6 GB, hence 48K ctx), x4 hyper-connection kernel

Common to all rounds: temperature 0.7, top_p 0.8, 26,624 tokens per round, thinking disabled. The request pins from ATLAS3 on stop Atlas's model preset (presence penalty 1.5, top_k 20) from applying, which is what llama.cpp had been getting all along.

Reference: the prod llama.cpp line on the same box
LLAMA_ATTN_ROT_DISABLE=1 llama-server --host 0.0.0.0 --port 8000 --alias qwen -ngl 999 --jinja -fa on \
  -m Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf --override-tensor 'per_layer_token_embd=CPU' \
  -c 131072 -ctk q8_0 -ctv q8_0 --parallel 1        # llama.cpp PR #27742 build, container llama-flashnext

Other tested variations (single prompt, no game built)

3,882-token prompt, 1,200-token greedy generations, watchdog off, 48K context, 8K prefill chunks. Every kernel change had to reproduce the previous binary's greedy output hash before its speed was counted.

VariationEnvFlagstok/sVerdict
precision: BF16 head + BF16 attentionATLAS_QSA_DEVICE_TOPK=1 ATLAS_PLE_CACHE_SLOTS=4194304--attn-proj-dtype bf1613.8each BF16 choice ≈ −5–6%
precision: BF16 head + NVFP4 attentionsame--lm-head-dtype bf16 --attn-proj-dtype nvfp415.0the ATLAS6 base
precision: NVFP4 head + BF16 attentionsame--lm-head-dtype nvfp4 --attn-proj-dtype bf1614.7
precision: NVFP4 head + NVFP4 attentionsame--lm-head-dtype nvfp4 --attn-proj-dtype nvfp415.6fastest, argmax-flip risk in the head
GDN projections NVFP4+ ATLAS_QWEN4EXP_BF16_GDN=0--lm-head-dtype bf16 --attn-proj-dtype nvfp418.3adopted (ATLAS6)
+ hc_pre_finish_x4 (e7b6af2f)samesame20.3adopted; ATLAS_HC_FIN_X4=0 restores the old kernel
+ FP8 lm_headsame--lm-head-dtype fp8 --attn-proj-dtype nvfp421.2not adopted (precision)
+ NVFP4 lm_headsame--lm-head-dtype nvfp4 --attn-proj-dtype nvfp421.7not adopted (precision)
GDN fused norm+ ATLAS_GDN_FUSED_NORM=1same as ATLAS620.1 (neutral)rejected: greedy output differs
MoE grouped decode arm (A2)+ ATLAS_MOE_DECODE_ARM=grouped--attn-proj-dtype bf165.33× slower than the scalar arm; scalar stays
device top-k on/off (A1)ATLAS_QSA_DEVICE_TOPK=1 vs unset--attn-proj-dtype bf16+3–5% with sparse selection activeadopted
4M PLE slots + event pins (A3)ATLAS_PLE_CACHE_SLOTS=4194304--attn-proj-dtype bf1614.5 / 15.1 (neutral)kept (harmless)
D2H spin sync / PLE fault threadsATLAS_D2H_SPIN_SYNC=1 · ATLAS_PLE_FAULT_THREADS=128--attn-proj-dtype bf16neutralnot adopted
decode CUDA graphs on/off (llama.cpp, for reference)—llama-server graphs on/off24.8/25.1 vs 24.4/24.1≈3%

Every version, round by round

Fix rounds come in three kinds: a console error (the browser's message, nothing else), a playtest brief (the reviewer's description of what is wrong and, usually, where), and a literal brief (the exact edit to make, with "do not change anything else"). The pattern across the table: the model repairs precisely what a literal brief names and regresses on whatever it touches unasked.

ConfigRoundInput givenTokensWalltok/sResultVersion
ATLASR1 buildbuild prompt18,02717 m 19 s17.3unterminated; 4 of 6 script blocks unclosed—
R2 fix/polishreview prompt + own file26,62427 m 18 s16.3hit the 26K cap; best of three rewrites, broken—
fix 1console error: Unexpected end of input16,68716 m 48 s16.6loads—
fix 2playtest brief (click mapping, towers never fire, wave never ends)17,47517 m 46 s16.4loads; play-through unverifiedPlay
ATLAS2R1 buildbuild prompt9,6129 m 16 s17.3closed file—
R2 fix/polishreview prompt + own file13,86613 m 51 s16.7closed; never bootsOpen
ATLAS3R1 buildbuild prompt18,95418 m 34 s17.0closed file—
R2 fix/polishreview prompt + own file18,95319 m 44 s16.0closed; one fatal typo—
fix 1 (self-fix)console error: c is not defined at drawOverlayGrid18,98020 m 22 s15.5one-token diff; intro-state bug remains—
fix 2playtest brief: intro state, placement, road visibility19,01419 m 51 s16.0state fix correct; a stray } broke the scriptv
fix 3console error: Unexpected token '}'30,33435 m 55 s14.1one statement changed; 11K tokens of extra outputv
fix 4playtest brief: overlay never shown, path in cell units19,07320 m 24 s15.6fixed the two lines named, ignored the other path consumersv
fix 5literal brief: path in pixels at one place, HUD writer19,31820 m 32 s15.7playable; kill reward 0/0 → NaN goldv
fix 6playtest brief: NaN gold19,25920 m 11 s15.9NaN fixed; an unasked edit reintroduced the original typov
fix 7console error: c is not defined (again)19,04620 m 10 s15.7single-token diff; fully playable (3 waves, 20/20 lives)v
fix 8aplaytest brief: towers placeable on the road16,48018 m 17 s15.0no code block — explanation, then EOS mid-file—
fix 8 (final)tighter brief, rerun (two named changes, nothing else)19,01222 m 31 s14.1both fixes correct; an unasked edit broke the overlay grid again — reviewer reverted that one tokenv Final
ATLAS4R1 build (watchdog on)build prompt8,4019 m 13 s15.2cut by 3 false-positive rollbacks on a coordinate array—
R1 build (watchdog off)build prompt20,21722 m 08 s15.2closed file—
R2 fix/polishreview prompt + own file22,43226 m 39 s14.0closed; 0 syntax errors; dead enemies never removedOpen
ATLAS5R1 buildbuild prompt23,22726 m 24 s14.7closed file—
R2 fix/polishreview prompt + own file23,71128 m 30 s13.9closed; 0 errors; playable except the stats panel—
fix 1 (final)literal brief: call updateStats() from the frame loop23,58129 m 28 s13.3exactly the two calls requested; panel verified livev Final
ATLAS6R1 buildbuild prompt20,62115 m 57 s21.6closed file—
R2 fix/polishreview prompt + own file20,69716 m 59 s20.3closed; calls undefined buildStatic()—
fix 1console error: buildStatic is not defined20,71917 m 35 s19.6file echoed unchanged—
fix 2literal brief: rename the one call to bakeStatic()20,39916 m 39 s20.4exactly the one-token rename; update loop then throws on lvv
fix 3 (final)literal brief: bind lv in the gun/cannon branch20,51617 m 12 s19.9exactly the two bindings; playable (4 waves cleared)v Final

The tower-defense benchmark: prompts, briefs and harness

The two build-off prompts are byte-identical to the August ones. The fix templates and every brief a fix round was given are below, verbatim, so each row of the version table can be reproduced.

Round 1 — build prompt (identical for all configs)
Build a complete, polished single-file HTML5 tower-defense game (index.html: all CSS and JS inline, canvas-based, no external assets or libraries). Requirements:
- 20x12 grid map with a fixed winding enemy path (visually distinct road) and buildable cells
- 3 tower types: Gun (fast, single-target), Cannon (slow, splash AoE), Frost (slows enemies). Each has cost, range, and 2 upgrade levels (better damage/range/fire-rate). Towers can be sold for 70% of invested cost.
- Enemies: normal, fast, tank; a boss every 5th wave. Health bars above enemies. HP/speed/count scale each wave.
- Economy: start with 100 gold and 20 lives; kill bounties; wave-completion bonus; a leaked enemy costs 1 life (boss costs 5).
- UI: top bar (gold, lives, wave, score); tower shop panel with prices; click a shop tower then click the map to place (show range preview and valid/invalid cell highlight); click a placed tower to select it and show upgrade/sell panel; Start Wave button plus auto-start toggle; game speed x1/x2 button.
- Targeting: enemies closest to the exit first. Render projectiles. Show splash radius on cannon impact and a frost visual on slowed enemies.
- Win after beating wave 20; lose at 0 lives. Proper end screens with a Restart button.
- Polish: hover states, particle effects on kills, hit flashes, clean modern dark theme.
Output the COMPLETE index.html in one ```html code block. No placeholders, no TODOs — fully working code only.
Round 2 — review / fix / balance / polish prompt
Below is your current index.html for the tower-defense game. 1) Review it for bugs: logic errors, undefined variables, broken event handlers, targeting/pathing mistakes, economy bugs, rendering issues — fix them all. 2) Ensure every item of the original spec is implemented: 3 tower types with 2 upgrade levels + sell; 4 enemy types incl. boss every 5th wave; wave scaling; gold/lives/wave/score UI; place/select/upgrade/sell interactions with range preview; start-wave + auto-start + speed toggle; closest-to-exit targeting; projectiles + splash + frost visuals; win at wave 20 / lose at 0 lives with restart; particles and polish. 3) Balance: tune costs, bounties and wave scaling so waves 1-20 are beatable but challenging. 4) Polish: particles, hit flashes, nicer end screens. If the file below is cut off mid-code, complete it properly. Output the COMPLETE final index.html in one ```html code block.

Current file:
```html
{FILE}
```
Fix round — console-error template
Your single-file HTML5 tower-defense game below fails to load. The browser console shows this error:

    {ERR}

Find and fix the error. Also scan the whole file for any other syntax mistakes (unbalanced brackets/parentheses, broken statements) that would prevent it from running. Do not redesign anything — make the minimal corrections needed so the game runs. Output the COMPLETE fixed index.html in one ```html code block.

Current file:
```html
{FILE}
```
Fix round — playtest / literal-brief template
Your single-file HTML5 tower-defense game below now loads without console errors, but a playtest found these bugs:

{ERR}

Fix every one of them. Do not redesign anything and do not rewrite from scratch — make the minimal, targeted corrections needed so the game plays correctly end to end (place towers where the player clicks, towers acquire and shoot enemies in range, waves end and the next wave can start, HUD/toasts behave). Output the COMPLETE fixed index.html in one ```html code block.

Current file:
```html
{FILE}
```
How the rounds were run (sampling, protocol, serve line)
Sampling (all rounds, all configs): temperature 0.7 · top_p 0.8 · max_tokens 26,624 per round · thinking
disabled (chat_template_kwargs: enable_thinking=false). From ATLAS3 on, the request also pins
presence_penalty 0 · top_k 40 · min_p 0.05 (TDBENCH_EXTRA) so Atlas's model preset (presence 1.5) does not
apply — that is what llama.cpp had been getting all along.

Protocol: Round 1 = the build prompt; Round 2 = the review prompt with the model's own file pasted in. The
harness (tdbench_s.py) extracts the single ```html block from each reply; the last complete file is the
published game. A fix round (tdfix3.py) sends the current file back with either the browser console error
(FIX template) or a playtest brief written by the reviewer (RUNTIME template, texts below); the harness
pads the prompt with whitespace until its tokenised length is even (a workaround kept from the first run).
Every game was loaded in a browser and played through by the reviewer before its result was written down.

Serving (Atlas rounds): prod llama-flashnext stopped first (one 128 GB pool); the exact serve line, env pins
and flags per iteration are in "Serve command per game" above. Prod restored after each run. Publishing: spark → orange ~/sites/bench → Caddy → wg umbilical → edge (TLS).

The briefs, per fix round

ATLAS · fix 1 — console error
Uncaught SyntaxError: Unexpected end of input
ATLAS · fix 2 — playtest brief
RUNTIME:
1. Click-to-cell mapping is wrong on the Y axis: clicking a buildable cell places the tower on a different, lower row (roughly double the clicked canvas Y). The canvas is displayed at CSS size (width:100%) while its internal resolution differs, and the click handler does not scale mouse coordinates by canvas.width/rect.width and canvas.height/rect.height consistently for both axes. The range preview / hover highlight must use the same corrected mapping.
2. Because of (1) (and possibly independently) towers never fire: with two Gun towers placed beside the road, wave 1's six enemies walked the whole path untouched — score stayed 0, gold never increased, no projectiles were drawn. Verify targeting (closest-to-exit in range), fire-rate cooldown, projectile spawn/update/hit and bounty payout actually run inside the main game loop.
3. Wave completion never triggers: after all wave-1 enemies leaked (lives 20 -> 14) the HUD stayed "WAVE IN PROGRESS", the Start Wave button never re-enabled, and the "LANE CLEAR — deploy your towers" toast was re-added on every frame (dozens of stacked toasts). End the wave once all enemies of the wave have spawned and none remain (killed or leaked), pay the wave bonus, advance the wave counter, re-enable Start Wave / honor auto-start, and show status toasts only once per state change.
ATLAS3 · fix 1 — console error
Uncaught ReferenceError: c is not defined at drawOverlayGrid
ATLAS3 · fix 2 — playtest brief
RUNTIME:
1. The game can never be started. Clicking "Deploy Forces" (or anywhere on the canvas) runs handleClick, whose `state==='intro'` branch only toggles the intro overlay's CSS classes ('show'/'hide') and never sets `state='idle'`, so updateControls() keeps the Start Wave button disabled forever and the Space key does nothing. The intro overlay also flickers: it is briefly re-shown then hidden on every click. Fix: on Deploy Forces (and only there), hide the intro overlay for good, set state='idle', call updateControls(), and stop the canvas click handler from treating clicks as intro dismissal.
2. Because state stays 'intro', towers cannot be placed: handleClick returns early (`if(state!=='idle'&&state!=='wave')return;`) before the placement code. After fix 1, verify that picking a tower in the Armory (pendingType) and clicking a buildable cell calls placeTower and deducts gold.
3. The enemy road is nearly invisible: buildTerrain() strokes it with #3a4b58 / #2a3a46 / #33475a on a #101c25 background, so the player sees no path. Make the road clearly visible (higher-contrast fill and edge, e.g. a lighter road with a bright dashed centre line) and make the spawn/goal portals clearly visible.
ATLAS3 · fix 3 — console error
index.html:530:135 Uncaught SyntaxError: Unexpected token '}'
ATLAS3 · fix 4 — playtest brief
RUNTIME:
1. The intro overlay is never shown, so the player cannot press "Deploy Forces" and the game stays in state 'intro' with Start Wave disabled. The `.overlay` CSS rule defaults to opacity:0 and pointer-events:none, and nothing ever adds the 'show' class to the #intro element at startup. Fix: add `intro.classList.add('show')` during init (after updateControls()), and make the Deploy Forces handler remove 'show' (it already sets state='idle' and calls updateControls()).
2. The enemy road is drawn as a tiny squiggle in the top-left corner instead of across the map: PATH from buildPath() holds grid-cell coordinates (0..19, 0..10.5), but strokePath(c) passes PATH[i].x / PATH[i].y straight to moveTo/lineTo as pixels. The portals are correct because drawPortal is called with SPAWN.x*CELL. Fix: in strokePath multiply by CELL (c.moveTo(PATH[0].x*CELL, PATH[0].y*CELL) and c.lineTo(PATH[i].x*CELL, PATH[i].y*CELL)). Then check every other consumer of PATH — computePathCells, pathDist, pointAt, PATH_LEN, enemy movement and the shadow dots in buildTerrain — and make sure they all agree on one unit (pixels) so that enemies walk along the visible road and pathCells marks the road cells as unbuildable.
ATLAS3 · fix 5 — literal brief
RUNTIME:
1. Enemies still move in grid-cell units while everything else is in pixels: spawnEnemy starts them at PATH[0] (cell coords), updateEnemies advances e.dist by spd px/s (55–92) and pointAt(e.dist) interpolates along PATH in cell units, so PATH_LEN is only 25.5 and every hostile "breaches the line" in well under a second, never entering any tower's range (ranges are 125–172 px). Fix it at ONE place: define the path in pixels, i.e. replace `const PATH=buildPath();` with `const PATH=buildPath().map(p=>({x:p.x*CELL,y:p.y*CELL}));`. Then make these three consumers consistent with pixel coordinates: (a) strokePath(c): use PATH[i].x and PATH[i].y directly again (remove the *CELL you added there); (b) computePathCells(): the number of steps should be Math.ceil(Math.hypot(b.x-a.x,b.y-a.y)/CELL*4) and the cell key must be Math.floor(x/CELL)+'_'+Math.floor(y/CELL); (c) pointAt(s) and directionAt(s): the fallback return past the end must be {x:GOAL.x*CELL,y:GOAL.y*CELL} (GOAL/SPAWN stay in cells; drawPortal already multiplies by CELL). pathDist(px,py) and isBuildable() already pass pixels, so they become correct automatically. Do not change the enemy speeds or tower ranges.
2. The HUD counters never update: popStat(el) only replays the 'bump' animation and nothing ever writes the numbers, so the header keeps showing 100 gold / 20 lives / wave 0 / score 0 while the game state changes. Add a function updateHud() that writes gold, lives, the wave counter (current wave / total waves) and score into the existing HUD elements (goldVal, livesVal and the wave/score elements), call it at the start of popStat(el), once at init, and at the end of loop() every frame.
ATLAS3 · fix 6 — playtest brief
RUNTIME:
1. Gold becomes NaN after the first kill, so the HUD shows "NaN" and no further towers can be bought. In updateEnemies(), the kill reward is computed as `Math.round(e.bounty*(1+e.shield/e.shieldMax*SHIELD_PEN))`; for unshielded enemies (normal, fast) shield and shieldMax are both 0, so 0/0 = NaN and `gold+=reward` poisons gold. Fix: compute the shield fraction safely, e.g. `const sf=e.shieldMax>0?e.shield/e.shieldMax:0; const reward=Math.round(e.bounty*(1+sf*SHIELD_PEN));`. Check the other places that divide by shieldMax (the enemy shield bar drawing, damageEnemy) and guard them the same way.
ATLAS3 · fix 7 — console error
index.html:497 Uncaught ReferenceError: c is not defined
    at drawOverlayGrid (index.html:497)
    at render
    at loop
ATLAS3 · fix 8 — literal brief
RUNTIME:
1. Towers can be placed on the road. Cause: pathDist(px,py) returns NaN for every point, so `pathDist(c.x,c.y)<CELL*.5` in isBuildable() is never true. pathDist is NaN because PATH contains consecutive duplicate waypoints (buildPath() calls `push(3,4);corner(3,4);` and corner() pushes (3,4) again — same at (9,5) and (15,6)), which makes zero-length segments; pointSeg() divides by the squared segment length, 0/0 = NaN, and Math.min propagates it. Make exactly two changes and nothing else: (a) in pointSeg(), if the squared segment length is 0, return the plain distance from the point to the segment's start point; (b) in isBuildable(i,j), return false when pathCells.has(i+'_'+j). Do NOT modify buildPath() or the path shape, and do not rewrite any other function. Output the complete file.
ATLAS5 · fix 1 — literal brief
RUNTIME:
1. The top stats panel (gold, lives, wave, score — elements #sGold, #sLives, #sWave, #sScore) never changes while playing: updateStats() is only reached through updateAll(), which runs at init and on shop/inspector actions, so kills, leaks, wave changes and gold income are never reflected. Fix: in frame(now), right after the `for(let i=0;i<steps;i++)update(dt);` loop (inside the same `if(!G.paused&&G.phase!=='start')` block), call `updateStats();` and `updateStartButton();`. Do not change update(), draw(), or anything else.
ATLAS6 · fix 1 — console error
index.html:962 Uncaught ReferenceError: buildStatic is not defined
    at resetGame (index.html)
    at index.html:962
ATLAS6 · fix 2 — literal brief
RUNTIME:
1. The game never starts: at init, resetGame() calls buildStatic(), which does not exist, so the script throws "ReferenceError: buildStatic is not defined" and requestAnimationFrame(loop) is never reached (the board stays black; only the HUD and shop clicks work). The pre-render function that builds roadBase/roadTop/decor/staticBake IS defined — it is named bakeStatic(). Make exactly one change: in resetGame(), replace the call `buildStatic();` with `bakeStatic();`. Do not rename bakeStatic, do not add a new function, do not change anything else. Output the complete file.
ATLAS6 · fix 3 — literal brief
RUNTIME:
1. The game freezes the moment a turret acquires a target: update(dt) throws "ReferenceError: lv is not defined" and the requestAnimationFrame loop dies. In update(), inside `for(const t of gridMap.flat())`, only the frost branch defines `const lv=t.lv`; the other branch (gun/cannon) uses `lv.dmg`, `lv.cd`, `lv.range` etc. without defining `lv` (it calls `lv2(t)` for the range check but never binds `lv`). Make exactly one change: at the start of that `else` branch (before `let best=null,bd=-1;`) add `const lv=lv2(t);`. Also check that the frost branch's `const lv=t.lv` is the intended level object (if `t.lv` is the level INDEX, use `const lv=lv2(t);` there too). Do not change anything else. Output the complete file.

Iteration 1 — ATLAS: serve it as-is and write down every failure

Analysis + comparison

Stock Atlas at main@4df6e609 took eleven serve attempts to complete the two rounds once. Five distinct engine defects: the loop-watchdog rollback desynchronised the sparse-attention indexer (HTTP 500 mid-stream); speculative verify with the indexer active ended responses as a clean stop at 1,679 tokens (silent truncation); prompts over 8,192 tokens were refused twice at the n-gram layer (a cap off by one, then a fixed 65,536-slot row cache); continuation prefill chunks faulted with CUDA_ERROR_MISALIGNED_ADDRESS and left a zombie holding 100 GB; and the indexer's capacity defaulted to 32,768 tokens regardless of context. Against the August llama.cpp run of the same model (config A: one typo, one fix round, playable), the difference was not the model.

Proposed changes + decisions

Two investigations of the same five defects, one by the prod model and one by config A on llama.cpp, each with read-only tools over the tree; config A's was the better-anchored one. A seven-item plan came out, ordered smallest-safe-first. Decision: execute all seven with config A as the executor from briefs carrying exact file:line anchors, accumulate on one branch, then run the benchmark again on default settings. (Wide briefs made the executor loop in code search; anchored briefs of at most three subtasks worked, and two items were finished by the reviewer.)

TD benchmark

17.3 tok/s single stream, ~300 tok/s prefill, with five environment pins and single-chunk prefill as workarounds.

Defects

The game ("IRONGATE") needed a second fix round to boot; the file itself was a symptom of the sampling problem found two iterations later.

Iteration 2 — ATLAS2: the fixed engine on defaults

Analysis + comparison

All seven items landed as separate commits: derived n-gram cache sizing (e623e010), indexer capacity from --max-seq-len (561be215), honest finish_reason="error" instead of a fake stop (16af4d33), the scheduler breaking on a latched CUDA fault instead of 500-ing for hours (35028544), rollback restoring the indexer and n-gram aux state (f4e81291), MTP latched off before the indexer's inert bound (86bdf303), and the prefill scratch laid out on 256-byte boundaries — the misaligned-address fault was an odd-width slab packed by raw byte products (97aa9fa4). The fault attribution came from CUDA_LAUNCH_BLOCKING=1; compute-sanitizer was a hundred times too slow to reach the failing chunk.

Proposed changes + decisions

Run on default chunking with no environment pins and no speculative decoding, and treat any server error as a failed run.

TD benchmark

17.3 → 16.7 tok/s, zero server errors. The fixes remove failure modes, not cost.

Defects

The game ("BULWARK") was inert: no boot loop, no event listeners, a 38 KB file that never ran. Same engine, same model, twice now. The conclusion had to be that the engine was not the quality problem.

Iteration 3 — ATLAS3: the sampling that was never equal

Analysis + comparison

Reading sampling_setup.rs: with the model's use_sampling_presets_for_core flag on, Atlas fills every request field the client leaves unset from the card's [sampling.non_thinking] preset — top_k 20, min_p 0, and presence_penalty 1.5. The August llama.cpp runs had used llama.cpp's own defaults: top_k 40, min_p 0.05, no penalty. A flat −1.5 on every already-emitted token is poison for a 20K-token file: closing braces, repeated identifiers and boilerplate all get suppressed. That is what "dead" and "inert" looked like.

Proposed changes + decisions

Pin the request to llama.cpp's values and re-run on the identical engine. Later, make it a model property: [behavior].use_sampling_preset_penalties_for_core, default true so nothing changes for other models, set false for Flash-Next (953820a0; tool-calling requests keep the tools preset).

TD benchmark

17.0 → 16.0 tok/s; a complete, wired 51 KB game ("Ironline") with a single typo, and a self-repair that changed exactly that token — the same behaviour config A showed on llama.cpp in August.

Defects

Eight fix rounds, each about twenty minutes, each repairing exactly what the brief named and nothing else: the typo; the intro state that never left "intro"; a path stored in cell units but consumed in pixels; a HUD that was never written; a 0/0 kill reward; a road that was buildable because a duplicated waypoint made the path distance NaN. Every unrequested edit the model made introduced a regression (a stray brace, a variable name "corrected" in a sibling function three separate times), and one round emitted end-of-sequence in the middle of the file. The lesson that carried forward: literal briefs, one change each, and name the lines the model must not touch.

RoundTokenstok/sOutcome
R1 / R218,954 / 18,95317.0 / 16.051.2 KB, one fatal typo
fix1 · console error18,98015.5byte-identical except the one token
fix2 · runtime brief19,01416.0state fix correct, stray } added
fix3 · console error30,33414.1one statement changed, 11K tokens of extra output
fix4 · runtime brief19,07315.6fixed the two lines named, ignored the other consumers
fix5 · literal brief19,31815.7playable; kill reward 0/0
fix6 · runtime brief19,25915.9NaN fixed; unasked edit re-broke the typo
fix7 · console error19,04615.7single-token diff; fully playable
fix8 · literal brief19,01214.1road fix correct; unasked edit reverted by hand

Iteration 4 — ATLAS4: precision, and a rollback that was wrong on both sides

Analysis + comparison

The checkpoint ships attention projections and the lm_head in BF16; Atlas requantised both to NVFP4 at load, while llama.cpp keeps them at Q8_0. The repo's own 27B recipe pins --lm-head-dtype bf16 "for exact argmax". The rollback item from iteration 2 still resumed one token behind the snapshot with the watchdog on (pos 6050 but 6049 ingested).

Proposed changes + decisions

TD benchmark

15.2 → 14.0 tok/s. With the watchdog on, the same build rolled back and re-steered three times with zero indexer desyncs — and then the loop detector false-positived on a coordinate array and cut a coherent file at 8.4K tokens. It stays off for code. The speed drop is the precision trade, later measured per half at long context: BF16/BF16 13.8, BF16 head with NVFP4 attention 15.0, NVFP4 head with BF16 attention 14.7, both NVFP4 15.6 tok/s.

Defects

A 57.7 KB game with zero syntax errors and one logic bug: killed and leaked enemies were never removed from the enemy list, so a wave never registered as cleared. Placement, road and combat all worked.

Iteration 5 — ATLAS5: the gap study, and three items that did not move the needle

Analysis + comparison

Two investigations by the prod model: first an Atlas-only decode anatomy, then Atlas versus llama.cpp side by side with both trees cited by line. The picture it drew: llama.cpp keeps every data-dependent decision on the device (MoE ids through the mul_mat_id fast path, the indexer top-k as a device sort), has zero host synchronisations per token, keeps the n-gram table in host memory the unified GB10 serves directly, and carries a chunked Gated-DeltaNet kernel for prefill; Atlas had 48 stream synchronisations per token and vetoed CUDA graphs model-wide. The report's one shaky claim — that llama.cpp's graphs were compile-gated — was wrong (the prod build has them on), so the runtime question was measured: GGML_CUDA_DISABLE_GRAPHS=1 costs llama.cpp about 3%. Capture went to the bottom of the plan.

Before A3, a memory check: a resident 51 GB table cannot fit for either engine (Atlas sits at 115.6 of 121.7 GB after load; llama.cpp has only ~10 GB of its 25.6 GB table in page cache). What matters is the per-sequence working set, at most 16 rows of 160 bytes per token — a 4M-slot cache is 640 MB.

Proposed changes + decisions

Execute A3, A2, A1 with the executor from anchored briefs, verify each contribution separately, then benchmark. A3 (event-scoped n-gram pin release, pinned slot staging; 69cb6625) needed two executor passes. A2 (ATLAS_MOE_DECODE_ARM=grouped, routing every layer's single-token MoE through the tensor-core arm; ad7203f5) was done in one. A1 (device-side indexer selection) produced no change after 33 executor steps and was written by the reviewer: a qsa_select_topk kernel with torch's tie-break, behind ATLAS_QSA_DEVICE_TOPK=1, with a parity mode that runs both arms and fails on the first mismatch (0b9ada13).

TD benchmark

ItemMeasured
A3 — n-gram barrier + stagingneutral (14.5 / 15.1 tok/s): the table sits on one model layer, one barrier per token, not 36
A2 — tensor-core MoE arm at M=13× slower (5.3 vs 15.0 tok/s); scalar arm stays
A1 — device-side selectionparity clean over 1,400 tokens; unchanged below the 2,051-token inert bound; +3–5% once selection is active (13.1/13.5 → 13.8/13.9)
ATLAS5, all three on14.7 → 13.9 tok/s on 23K-token rounds

Host synchronisation, table placement and graph capture together were worth a few percent. The gap had to be inside the decode kernels.

Defects

A 62 KB game ("IRONGATE") that played out of the box — briefing, road rules, three tower types, waves, kills, win and lose — except that its stats panel was refreshed only on shop clicks. One two-line fix round.

Iteration 6 — ATLAS6: measure, then act on the two largest stages

Analysis + comparison

One nsys window over 40 seconds of pure decode (642 tokens, BF16 head, NVFP4 attention, device top-k, 3,882-token prompt, ~69 ms per token). The profile:

StageLaunches / tokenms / tokenReading
dense_gemv_bf167324.0 (42%)one is the BF16 lm_head; 72 are the 36 GDN layers' in/out projections, which the loader keeps in BF16: 4.15 GB/token, bandwidth-bound at that precision
hyper-connections (hc_pre_finish and siblings)38817.5 (27%)~1.25 GB of weights whose floor is ~5.4 ms; stage 3 alone 121 µs × 97
MoE (gate/up, silu/down, top-k, blend)~1008.3 (14%)1.46 GB at ~176 GB/s ≈ 77% of achievable: already near its floor
everything else~450~6recurrence, attention, selection, norms

Analytically Atlas moved more bytes per token than llama.cpp (8.5 GB vs 7.6 GB) because those projections never got quantised, and spent a further ~12 ms in the hyper-connection kernels. The prod model's own reading of the same profile misattributed the MoE term (it recomputed with the wrong intermediate size) and proposed a GEMV rewrite worth "26 ms"; the numbers overruled it.

Proposed changes + decisions

TD benchmark

21.6 → 20.3 tok/s on 20.6K-token rounds: +47% over ATLAS5, past ATLAS3's 17.0 with the quality precision kept on the head. llama.cpp on the same box: 24–25.

Defects

The 52.9 KB game ("IRONWORKS · TD") shipped with two: the reset code called a helper by the wrong name (buildStatic for bakeStatic), so the render loop never started, and the turret loop used a level variable it never bound. A console-error round echoed the file unchanged; two literal-brief rounds made exactly the one-token rename and the two bindings. Then four waves cleared with correct gold, score and lives. Two defects in one sample versus one for ATLAS5: the quality gate on the NVFP4 GDN requantisation is neither passed nor failed.

The decision log, entry by entry

The same six iterations as a log: decision, commits and the files each touched, impact, what was rejected and what is left. This is the record kept on the benchmark page; the PR carries the same for its reviewers.

ATLAS — 2026-09-03 · main @ 4df6e609

Serve the RadixArk NVFP4 checkpoint on stock Atlas2026-09-03R1 17.3 tok/s · 11 serve attempts
Decision
Run the same two-round TD benchmark llama.cpp had run in August, on Atlas main, and record every failure rather than tune around it.
What happened
Five distinct engine defects over eleven serve attempts: watchdog rollback desyncing the QSA indexer (HTTP 500), MTP verify with QSA active ending the response as a clean stop (silent truncation), prompts over 8,192 tokens refused twice at the PLE layer (cap off by one, then a 65,536-slot row cache), continuation prefill chunks faulting with CUDA_ERROR_MISALIGNED_ADDRESS and a zombie process holding 100 GB, and ATLAS_QSA_MAX_TOKENS defaulting to 32,768 regardless of context.
Workarounds used
ATLAS_PLE_MAX_TOKENS, ATLAS_QSA_MAX_TOKENS, ATLAS_PLE_CACHE_SLOTS env pins, single-chunk prefill (--max-prefill-tokens 20480), --content-loop-watchdog false, no speculative decoding.
Output
Game "IRONGATE" needed a second fix round to boot. Speed 17.3 tok/s single stream, ~300 tok/s prefill.
Files
no engine changes; scripts ~/driver_td2*.sh, ~/tdbench_r2.py, ~/tdfix3.py on the bench machine; report bench report.

ATLAS2 — 2026-09-04 · fixed engine, default settings

Fix the five defects on a branch, then run with no env pins2026-09-04R1 17.3 → R2 16.7 tok/s
Decision
Two independent investigations (prod model and config A) produced a seven-item plan; execute all seven with the config-A executor on one branch, then re-run the benchmark on defaults.
Commits
  • e623e010 PLE row cache derived from max_batch_tokens × ngram_heads, env stays an override; cap clamped up with a warning — crates/spark-model/src/weight_loader/qwen4_exp/ple.rs, qwen4_exp.rs
  • 561be215 QSA indexer capacity from --max-seq-len (ModelConfig.max_seq_len) — crates/atlas-core/src/config.rs, config/factory.rs, spark-server/src/main_modules/serve_load.rs, spark-model/src/layers/qsa.rs, attach.rs
  • 16af4d33 engine errors reported as finish_reason="error" (500 for blocking clients) instead of stop; cuBLASLt failure logged with M/N/K — scheduler/types.rs, lifecycle.rs, ir/response.rs, anthropic/translate.rs, qwen3_ssm/trait_prefill_proj.rs
  • 35028544 scheduler breaks on a latched CUDA fault instead of holding 100 GB and 500-ing — scheduler/mod.rs
  • f4e81291 watchdog rollback saves/restores the QSA/PLE aux state with the SSM snapshot, declines when missing — traits/model.rs, model/types.rs, impl_a1.rs, model/trait_impl/mod.rs, scheduler/rollback.rs
  • 86bdf303 MTP latched off before a sequence crosses the QSA inert bound — scheduler/mtp_gate/qsa_latch.rs, scheduler/mod.rs, multi_seq/mod.rs
  • 97aa9fa4 hyper-connection prefill scratch laid out on 256-byte boundaries (odd slabs faulted the GEMM) — spark-runtime/src/buffers/sizes.rs, buffers.rs, ops/hyper_connection_lowrank.rs
  • d32605b2 merge of upstream main; cb6b1a59 test fix
Impact
Zero server errors across the run on default chunking with no env pins. Speed unchanged (the fixes remove failure modes, not cost). The game "BULWARK" was inert (no boot loop, no listeners): the quality problem was not the engine.

ATLAS3 — 2026-09-04 · sampling equalised

Pin the request to llama.cpp's effective sampling2026-09-04R1 17.0 → R2 16.0 tok/s
Decision
Found in spark-server/src/api/chat/sampling_setup.rs: Atlas fills unset request fields from the model card's [sampling.non_thinking] preset, including presence_penalty 1.5. Over a 20K-token file that suppresses every token already emitted. Re-run with presence 0, top_k 40, min_p 0.05 on the same engine.
Impact
A complete, wired 51 KB game ("Ironline") with one typo; the identical engine had produced dead/inert files twice. Then eight fix rounds: the model repairs exactly what a brief names, and every unrequested edit it made introduced a regression (fix2, fix6, fix8 each re-broke one token; one round emitted EOS mid-file; one echoed the file). Playable after fix7; a road-placement bug (path distance NaN from duplicate waypoints) fixed in fix8 with one reviewer token reverted. Per-round data on the bench page.
Commit
953820a0 [behavior].use_sampling_preset_penalties_for_core (default true, Flash-Next opts out) — crates/atlas-kernels/build_parse_behavior.rs, build.rs, build_codegen.rs, src/lib.rs, spark-server/src/api/chat/sampling_setup.rs (+ tests), kernels/gb10/qwen3.8-flash-next/MODEL.toml

ATLAS4 — 2026-09-04 · precision and rollback

BF16 head and attention for quality; make the watchdog rollback correct2026-09-04R1 15.2 → R2 14.0 tok/s (watchdog off)
Decision
Attribute the remaining quality gap to precision: the checkpoint ships attention and lm_head in BF16 and Atlas requantised both to NVFP4 at load. Keep the head BF16 by default and add a flag for BF16 attention projections. Finish the rollback fix (the earlier item left a one-token lag).
Commits
  • 61f1993b BF16 lm_head default for qwen4_exp — crates/atlas-core/src/config/methods.rs
  • c120f479 --attn-proj-dtype bf16 — atlas-core/src/config.rs, spark-server/src/cli/{serve_args,validate,flag_values}.rs, main_modules/serve_load.rs, spark-model/src/weight_loader/qwen35/load_layers/attention_arms.rs
  • 6c5b61c3 + 7db64cca rollback rewinds by fed tokens (the two watchdog sites sit on different sides of the step's push) and drops the token sampled from the discarded context — scheduler/rollback.rs, rollback_tests.rs, decode_logits_content.rs, decode_logits_step.rs
Impact
Rollback verified on the GPU: five rollback + re-steer events, zero indexer desyncs. Two side findings: the content-loop watchdog false-positives on code (it cut a coherent file at a coordinate array), so it stays off for code; and each BF16 choice costs 5–6% decode (both ≈ 12%). Game: 57.7 KB, zero syntax errors, one logic bug (killed enemies never removed). The speed drop from ATLAS3 is the precision trade, measured later per half: BF16/BF16 13.8 · BF16/NVFP4 15.0 · NVFP4/BF16 14.7 · NVFP4/NVFP4 15.6 tok/s at 3.9K context.

ATLAS5 — 2026-09-04 · gap study executed

Compare the two engines from source, execute the three cheapest items2026-09-04R1 14.7 → R2 13.9 tok/s
Decision
Two investigations by the prod model (Atlas-only decode anatomy, then Atlas vs llama.cpp side by side, gap study) proposed six items. Measured first: CUDA graphs are worth ≈3% to llama.cpp (GGML_CUDA_DISABLE_GRAPHS=1: 24.8/25.1 → 24.4/24.1), so capture was dropped to last. Execute A3, A2, A1 with the executor; verify each contribution.
Commits
  • 69cb6625 A3: event-scoped PLE pin release + pinned slot staging (the PLE is one layer, so one barrier per token) — spark-runtime/src/gpu.rs, cuda_backend/gpu_impl.rs, spark-model/src/layers/ple/{aux_state,layer,verify}.rs — neutral
  • ad7203f5 A2: ATLAS_MOE_DECODE_ARM=grouped selector — spark-model/src/layers/moe/forward.rs, MODEL.toml — grouped tensor-core arm at M=1 is 3× slower (5.3 vs 15.0 tok/s); scalar stays
  • 0b9ada13 A1: device-side QSA block selection qsa_select_topk behind ATLAS_QSA_DEVICE_TOPK=1, parity mode ATLAS_QSA_TOPK_VERIFY=1 — kernels/gb10/qwen3.8-flash-next/nvfp4/qsa_indexer.cu, spark-model/src/layers/ops/qsa.rs, layers/qsa.rs, new layers/qsa_decode_select.rs — parity clean; +3–5% only once the sparse selection is active (past 2,051 tokens)
Impact
Decode unchanged on the full benchmark: host synchronisation, PLE placement and graph capture together are worth a few percent. Game "IRONGATE" (62 KB) played out of the box except a stats panel refreshed only on shop clicks; one two-line fix round. Conclusion recorded: the gap is in the decode kernels.

ATLAS6 — 2026-09-06 · measured, not reasoned

Profile one decode window with nsys, act on the two largest stages2026-09-06R1 21.6 → R2 20.3 tok/s
Decision
Stop estimating. 40 s of pure decode under nsys (642 tokens, ~69 ms each): dense_gemv_bf16 73 launches = 24 ms (42%) — 72 of them the 36 GDN layers' in/out projections, which the loader keeps in BF16 (4.15 GB/token, bandwidth-bound); hyper-connection stage 3 hc_pre_finish 97 × 121 µs = 11.7 ms (27% with its siblings); MoE ≈ 8.3 ms at ~77% of achievable bandwidth (already near its floor, so the proposed GEMV rewrite was dropped). The prod model's own reading of the same profile misattributed the MoE term and was overruled by the numbers. Profile: reports/atlas-flashnext-decode-profile-2026-09-06.md.
Changes
  • Flag ATLAS_QWEN4EXP_BF16_GDN=0: requantise the GDN projections to NVFP4 through the loader path that already existed (its comment, written at a 2.2 tok/s era, said GDN bandwidth "is not what bounds decode") — 15.0 → 18.3 tok/s; costs ~6 GB (context 64K → 48K at 0.95 utilisation)
  • e7b6af2f hc_pre_finish_x4: one warp per hyper-connection stream, lanes over 32 output dims (4× the threads), identical FP32 accumulation order under --fmad=false — greedy output hash identical to the previous binary; 18.3 → 20.3 tok/s — kernels/gb10/qwen3.8-flash-next/nvfp4/hyper_connection.cu, spark-model/src/layers/ops/hyper_connection_lowrank.rs, hyper_connection_lowrank_gemm.rs (ATLAS_HC_FIN_X4=0 reverts)
  • 3fff25af hc_pre_finish rank loop unrolled ×8 — neutral (the kernel was thread-starved, not load-starved; kept as documentation)
  • 22321152 qsa_select_topk scores staged in shared memory — neutral on tok/s, bit-exact
Measured, rejected
ATLAS_GDN_FUSED_NORM=1 (recurrence + gated norm fused): no speed change and the greedy output differs. ATLAS_D2H_SPIN_SYNC, ATLAS_PLE_FAULT_THREADS=128: neutral. FP8 / NVFP4 lm_head: +1 / +1.5 tok/s on top, at the documented argmax-flip risk; not adopted.
Impact
Full benchmark 21.6 / 20.3 tok/s (+47% over ATLAS5, ATLAS3 was 17.0 on NVFP4 head and attention). Game "IRONWORKS · TD" shipped with two defects (a helper called by the wrong name, an unbound level variable in the turret loop); a console-error round echoed the file, two literal-brief rounds fixed exactly what was named; then four waves cleared cleanly. Quality gate for the NVFP4 GDN requantisation: two defects in one sample versus one for ATLAS5 — not conclusive.
Left on the table
BF16 lm_head ≈ 6 ms/token (precision decision); hc_pre_down and the other hyper-connection launches ≈ 5 ms; conv1d launched separately from the GDN recurrence ≈ 1 ms (the existing fused kernels are not bit-exact); ~1,000 launches/token of gaps. llama.cpp: 24–25 tok/s.

Totals: main as found to ATLAS6

Same model, hardware and prompts throughout. The first row is the engine as it was on main before any change; the last is the head of the pull request.

R1 tok/sR2 tok/svs llama.cpp (24.8)
ATLAS, main as found17.316.370%
ATLAS4/5, BF16 head + BF16 attention15.214.061%
ATLAS6, final21.620.387%

Net against main: +25% on both rounds. Against the BF16-precision configuration the last two changes were measured from: +44%. Per-token cost on the 3.9K-token single-prompt A/B went from 69 ms to about 49 ms.

Where the speed came from

Change (single-prompt A/B, bit-exact unless noted)tok/sGain
Base: BF16 head, NVFP4 attention, BF16 GDN15.0
GDN projections requantised to NVFP418.3+22%
hc_pre_finish_x4 kernel20.3+11%
Device-side QSA top-k (A1)—+3 to 5% while sparse selection is active
FP8 or NVFP4 head21.2 / 21.7+4 to 7%, not adopted

Tried and rejected: the grouped MoE decode arm (3x slower), the fused GDN norm (not bit-exact), the 4M-slot PLE cache with event-scoped pins (neutral, kept as harmless), D2H spin sync and PLE fault threads (neutral).

Reliability and correctness

ParameterMainFinal
Serve attempts to get one complete run111
Server errors during a benchmarkHTTP 500s; engine errors laundered into a clean stop0; engine errors surface as errors
Prompts over 8,192 tokensrefusedaccepted with the default 8K chunked prefill
Loop-watchdog rollbackdesynced the sparse-attention indexer, HTTP 500works, 0 desyncs on the GPU probe; left off for code because it false-positives on coordinate arrays
Sampling vs llama.cpppresence penalty 1.5 applied silently from the model presetequalised; preset penalties opt-in
Tower-defense gamedead on loadplayable

Cost. Resident memory rose from about 100 to 108 GB with NVFP4 GDN, which cut the KV budget from 21 to 14 GB and the practical context from 64K to 48K on this box.

Other models. The ten regression gates the repository runs on other models (Qwen3.6-35B-A3B-FP8, Qwen3.6 and 3.8 27B NVFP4) all pass at the PR head, with decode-floor at 22.7 tok/s. They did not at first: the BFCL function-calling subset on the 27B missed its bar by 0.22 normalized points, and main's own binary and harness scored the identical 86.23 / 83.10 on the same box and draw. Bisecting from the last passing tree to main found the cause in one file: an upstream commit had changed the greedy tie-break rule in the argmax kernel from lowest index wins, the rule llama.cpp and vLLM use, to highest index wins, and BF16 logits at temperature 0 tie often. Reverting that file alone restored every one of the nine flipped samples, on which the old rule answered five correctly and the new rule two. The PR now restores first-index-wins in the kernel and the two host paths, and the full gate draw came back at 86.53 / 83.38, the earlier record to the digit. The concurrency sweep needed a second run because one prompt looped under greedy decoding and the watchdog cut it below the sweep's delivery floor; on the re-run every floor was met.

Still open. The remaining 13% to llama.cpp is the BF16 head (about 6 ms per token), the other hyper-connection launches (about 5 ms), the separate conv1d launch (about 1 ms) and roughly 1,000 launches per token of gaps.

A long-horizon check: four hours of agentic work on the PR head

Speed on a fixed prompt is one thing; an engine also has to hold up when a model works with tools for hours, carrying 20 to 50 thousand tokens of context into every turn. So the last test on the PR head was agentic and phased: build an idle RPG in Rust compiled to WebAssembly with a Yew interface, from a formalised specification, in seven phases that follow a test-driven plan: formalise the mechanic math; write the public interfaces and a failing test suite; implement the battle core; implement shop, events and save; build the Yew components; integrate the game mechanics; test and harden. One Atlas session at a 98K context served all seven phases. The harness was rebuilt after a first attempt at the same task: each phase now opens with an orientation packet instead of a read loop, reads are budgeted, edits outside the phase's scope need a logged reason, and an independent verify tool reports tests, compile errors, clippy, the wasm check and module wiring after every phase. Every prompt, step trace, tool transcript and crate snapshot is published in the report.

What the engine showed. Native tool calling held for four hours and 118 requests, every one finishing on a stop token, with no engine error at the 98K context. Decode stayed between 18.5 and 20.8 tokens per second median across the phases and between 19.6 and 22 on long generations, including a single 12,155-token file. Time to first token was the cost: 55 seconds median at 20 thousand tokens of context, 122 to 138 seconds at 35 to 50 thousand, with a worst case of 256, which is roughly 330 to 380 tokens per second of effective prefill in 8K chunks. Over the run the model consumed about 3.06 million input tokens to produce 70 thousand. One new engine finding: the chat-level loop detector judged three similar file reads a loop and masked the tool-call token for a turn; agentic serving should disable it with the loop-suppression environment flag.

What the model showed. The three failure modes of the first attempt were gone: phase one made a single read instead of nine to fourteen, phase two declared every module and the verify tool collected 44 tests, and no phase drifted back to the balance test. What replaced them was compile-error gravity. Phase three spent its budget on one 12 thousand token battle module, regenerated it after judging the first version corrupted, and left six compile errors. Phases four and five, whose own scopes were the shop and the interface, spent thirty-two steps on those errors through logged scope overrides, and one lifetime fix cascaded into twelve new errors and was reverted. Phase six, with a fresh context and the same tools, fixed the crate in eleven steps, chose a fix that does not cascade and explained why, and left it compiling natively and for wasm with 17 of 44 tests passing. Phase seven found one real bug and two contradictory constants and ended with 19 passing. No interface was written and no game shipped. The honesty was good in all seven phases: every summary states what was not done and what is still red.

What the harness learned. The orientation packet, the wiring check and the scope fence each did what they were built for, and the fence turned an invisible drift into a logged decision without being able to prevent it. The next gate is obvious in hindsight: a phase must not advance while the crate does not compile. Single writes should be capped at about four thousand tokens, because a ten-minute generation that the model then discards is the most expensive thing that happened in the run. And a fresh context beat persistence, so the harness should reset rather than push on when a phase stalls on the same errors. One harness bug surfaced live, an empty file path that escaped the tool node, and was fixed between phases.

Eighteen more hours, and the game. The crate from that run was then continued with a fourth harness for ten hours, extended by eight at my request: a compile gate that follows any round leaving compile errors with a fresh-context repair round, writes capped at four thousand tokens with an append tool, a git-backed revert, and reads limited to the region about to change. Over 21 measured rounds the model consumed 14.9 million input tokens for 117 thousand output tokens, at a decode rate that never left the 17.6 to 20.6 tokens per second band, while time to first token ranged from 39 seconds in the small-context repair rounds to 218 seconds median and 301 seconds worst in the phases that carried 45 to 56 thousand tokens. The model finished the core to 41 of 44 tests, wrote a self-driving tick engine, and produced the first browser build of the whole effort, but never connected the interface to the engine: the rounds that could have done it went to the three failing balance tests instead, three times in a row despite a written deferral, until the scope fence put the core out of reach. I then wired the interface by hand in about forty-five minutes, labelled as mine in the report, and found two loop bugs on the way: kills banked gold but never awarded experience, and fights restarted on the beat they ended so travel events never rolled. The game is playable at the bench site, and the full account, with every prompt, transcript and per-round crate, is in the second report. The engine conclusion is a single number: on this stack an agent should keep its per-turn context under about twenty-five thousand tokens, because above that it spends its budget waiting for its own history to be re-read.

Reflection: what moved the number and what did not

Code: Atlas-Inf/atlas#23 (draft, 25 commits). Benchmark data and every game: the shootout page. The August baselines this compares against: the tower-defense build-off and the agentic-flows benchmark.