🏰 Tower Defense — model shootout

→ also: the agentic research shootout (LangGraph × Atlas codebase)

One complete HTML5 game per configuration, identical 2-round prompts (full build → review/fix/balance/polish), thinking disabled, temp 0.7 · top_p 0.8 · 26K tokens/round. NVIDIA DGX Spark GB10 (128 GB unified), 2026-08-28.

Round 2 — Atlas engine (2026-09-03/06) · performance & bug-fixing log ↓

Same prompts, same sampling, same hardware. Flash-Next served by Atlas (Rust/CUDA engine, RadixArk NVFP4 checkpoint, QSA sparse attention + PLE n-gram table) instead of llama.cpp — first on main@4df6e609, then on a branch carrying seven engine fixes found during the first run (assist/flashnext-longform-fixes).

D-1  loop-watchdog rollback desynced the QSA indexer ("pos 2864 but 2932 ingested") → 500
D-2  speculative verify + QSA active at 2051 tokens → error laundered into finish_reason="stop"
D-3  prompts > 8192 tokens refused twice (PLE cap off by one vs the 8193 chunk; 65536-slot n-gram row cache)
D-4  continuation prefill chunks faulted CUDA_ERROR_MISALIGNED_ADDRESS; engine lingered with a dead context
D-5  ATLAS_QSA_MAX_TOKENS default 32768 aborted decode mid-stream

Branch assist/flashnext-longform-fixes (7 commits on 4df6e609):
  ple: derive the n-gram row cache from the serve config; clamp ATLAS_PLE_MAX_TOKENS up          (D-3)
  qsa: size the indexer from --max-seq-len instead of a fixed 32768                               (D-5)
  scheduler: report engine aborts as finish_reason "error"; log cuBLASLt fallbacks                (D-2b)
  scheduler: stop the loop once a GPU fault is latched                                             (D-4 aftermath)
  rollback: rewind QSA/PLE aux state with the SSM state, or decline                                (D-1; restore path works, cursor 1 token behind — open)
  scheduler: latch MTP off before a sequence can cross the QSA inert bound                        (D-2a)
  mhc: align the prefill GEMM scratch sub-buffers; odd slabs faulted the GEMM                     (D-4 root cause)

D-4 root cause: hc_pre_gemm packed its BF16 scratch as raw byte products; inj_pre is lay*8 bytes, so an ODD
prefill chunk (1,117 tokens, or the default 8,193) put up_wt 8 bytes off a 16-byte boundary and the tensor-core
GEMM faulted. Found with CUDA_LAUNCH_BLOCKING=1 (compute-sanitizer was ~100x too slow to reach the chunk).
Verified: the failing 13,397-token prompt prefills at 4,096 and 8,192 chunking; ATLAS2 ran end to end with defaults.

Sampling was the quality lever. Atlas fills unspecified request fields from the model's [sampling.non_thinking] preset — top_k 20, min_p 0 and presence_penalty 1.5 (the vendor card's chat value) — while llama.cpp used its own defaults (top_k 40, min_p 0.05, no penalty). A flat −1.5 on every already-emitted token is poison for 20K-token code (closing tokens, repeated keywords, boilerplate all get suppressed): the dead/inert games of ATLAS and ATLAS2. With the request pinned to llama.cpp's values (ATLAS3) the model wrote a complete, wired 51 KB game and, like config A in August, self-repaired its one typo with a single-token diff. What remains is an ordinary logic bug.

Speed did not change with the fixes (16–17 tok/s single-stream, ~430 tok/s prefill): they remove failure modes, not cost. Atlas remains ~3x slower than prod and ~20–30% slower than llama.cpp on this model; its quality-of-output on the task was worse in both runs (dead or inert games vs. llama.cpp's playable ones), which is a sampling/serving-stack difference worth its own investigation.

Round 3 — long-horizon agentic run on the PR head (2026-09-09) · full report with transcripts ↓

Not a prompt benchmark: a four-hour, seven-phase, test-driven build of an idle RPG in Rust + Yew/WebAssembly driven by a tool-using harness against one Atlas session at a 98K context (PR #23 head, ATLAS6 pins). Every phase carries 20–50K tokens of context into every turn, so this measures what an agent actually pays on this engine: prefill. Orientation packet, read budget, scope fence with logged overrides and an independent verify tool after each phase; every prompt, transcript, trace and crate snapshot is in the report.

Engine: 118 requests, 3.06M tokens in / 70.5K out, no engine error at 98K, every request finished with stop. Decode 18.5–20.8 tok/s median across the run, 19.6–22.0 on long generations. TTFT is the agentic cost: 55 s median at 20K context, 122–138 s at 35–50K, worst 256 s (≈330–380 tok/s effective prefill in 8K chunks). One new finding: the chat-level loop detector judged three similar read_file calls a loop and hard-masked the tool-call token for a turn — set ATLAS_LOOP_NO_SUPPRESS=1 for agentic serving.

Model / harness: the three failure modes of the previous run (read-everything, hidden modules, balance-test attractor) are gone; compile-error gravity replaced them — phase 3's 990-line module blocked phases 4–5 until a fresh-context phase 6 fixed it in eleven steps. Honesty held in all seven summaries.

Round 4 — the game, 18 hours later (2026-09-09/10) · full report ↓ · Play

Round 3's crate (compiling, 19/44 tests, no interface) continued with harness v4 — a compile gate with fresh-context repair rounds, capped writes with append, a git-backed revert, region-only reads — for ten hours, then eight more at the user's request: 21 measured rounds, 987 min, 576 requests, 14.9M tokens in / 117K out, one Atlas session per driver at 98K context. The model finished the core (41/44), a self-driving tick engine and a rendering wasm build; the reviewer then hand-wired the interface to the engine in about 45 minutes (labelled in the report) and the game plays.

Engine: decode flat at 17.6–20.6 tok/s median for 19 hours regardless of context; TTFT median 39 s at 6–16K tokens to 218 s at 46–50K, worst 301 s. All 576 requests finished on stop; no engine error. The rule the data gives: keep an agent's per-turn context under ~25K tokens on this engine — the rounds that built the game ran at 8–30K and 20–90 s per step; the phases at 45–56K ran 3–5 min per step and wrote nothing.

Harness: the compile gate fixed the build in 5 of 7 repairs; capped writes ended corrupted generations; the scope fence held with zero overrides — and was the only thing that stopped the model from returning to the three failing tuning tests, which it did in three consecutive rounds despite a written deferral. What the model sees (a failing-test list) beats what it is told.

Round 5 — INKBOARD: a 12-hour TDD whiteboard build on Atlas work/qwen38-nvidia-nvfp4 (2026-09-15) · full report ↓ · Open the whiteboard

New task and a new Atlas line under test: a single-user whiteboard as a three-crate Rust workspace — a pure domain crate, an axum/tokio/sqlx backend on Postgres, a wasm canvas engine with a plain HTML frontend — built test-first by Qwen3.8-Flash-Next (RadixArk NVFP4, 98K context) under harness v5 (workspace verifier, compile gate with fresh-context repairs, scope fence, capped writes, region reads, crash handling). Twelve hours of inference time, restarts excluded. The reviewer formulated the epics and briefs and did not touch the code.

The branch: work/qwen38-nvidia-nvfp4 cannot serve Flash-Next as checked out (no kernel target, config parse error, missing flag; it is 287 commits behind upstream main). Merged with upstream main it resolves the model but leaves 3 kernel lookups unresolved (its DP4A M=4 arms) and destroys the CUDA context on continuation chunks of multi-chunk prefills — 4 crashes in 35 minutes (8.8K, 17.3K, 17.4K, 17.9K-token prompts; CUDA_ERROR_MISALIGNED_ADDRESS 716 in layer 0), intermittent, while the PR #23 head has served the same configuration for 50+ hours without one. The driver fell back to the PR #23 binary after the third crash; the synthetic A/B is in the report.

Totals on the PR #23 binary: 19 measured rounds, 724 min, 450 requests, 10.6M tokens in / 142K out; decode 18–20.4 tok/s median, TTFT 28 s at 12K → 135 s at 40K tokens. Harness lesson: a compile-only gate let a TDD contract phase deliver zero tests; scope-fenced single-purpose rounds (the frontend round: 31 min to a browser build) beat general finish rounds (four of them, 12–13 reads each, no change).

Round 6 — headless pi as the harness (Docker on orange, tools over SSH on spark) (2026-09-15) · full report ↓

Can the self-made langgraph harness be replaced by the pi coding agent (headless, JSON events) plus the s1lver091/pi-agent-config recipe, packaged as a container on orange with the workspace and every build tool on spark over SSH and Claude only orchestrating? A like-for-like replay of INKBOARD phase 4 (E3, backend implementation) with the same model, engine, snapshot, brief and 60-minute budget.

Verdict: feasible and equivalent — same result on the same phase; pi supplies the loop, event stream, multi-edit tool, validation, skills and extension hooks for free. It costs more time only because it does not prune context (Atlas has no prefix cache, so 88 % of the wall time was prefill). Next: a context-pruning extension, parallel-read accounting, and the compile-gate driver moved into the container. The first attempt was aborted at 16 min by a wrong parameter shape in the reviewer's edit override (pi 0.85 uses edits[]); the retry is the reported run. Full permissions on the repo as asked (no confirmation gates); production model restored afterwards.

Round 7 — a pixel world map on canvas, through pi (2026-10-05) · open the map ↓

An art task for the production model (Qwen3.8-Flash-Next NVFP4, blazux vLLM recipe) run headless through the pi harness: draw the world as a grid of squares on an HTML5 <canvas> — no images, no SVG, no fetch; the land data has to come from the model's own geography. Acceptance is a static Python check the model can call as verify: page structure, forbidden elements, data validity, area-weighted land coverage (22–38 %), inland and open-water spot checks by latitude/longitude, polar-band sanity — and, from round 3, self-crossing outlines and undeclared enclosed water. Five rounds on one workspace; the published page is the last one.

Lessons: the model cannot hand-author a dense data table and recovers badly from mid-file edits (r1), but it can author a few dozen polygons from geographic knowledge and the code that derives the big artefact from them (r2–r4). The checks decide what it can see: v2 rasterised the shapes the same way map.js must and printed an ASCII preview; the Sahara hole got through because no spot point sat in it, so v3 added geometry checks (self-crossing edges, flood-filled enclosed water) that name the shape and the lat/lon — r3 cleared them in 13 minutes. The 7200 × 3600 renderer landed in one round once the brief spelled out the two drawing regimes. Delivered as-is: pixelmap/ (the page, land.js, map.js, the r1 data that failed, every brief and round summary, verify.py, metrics). Known limit: the polygons still have vertices every ~5°, so zoomed in you see straight polygon edges, not coastline detail — the next program would be one round per continent with a regional preview in verify.

Performance & bug fixing — decision log (ATLAS → ATLAS6)

Prompts

Build a complete, polished single-file HTML5 tower-defense game (index.html: all CSS and JS inline, canvas-based, no external assets or libraries). Requirements:
- 20x12 grid map with a fixed winding enemy path (visually distinct road) and buildable cells
- 3 tower types: Gun (fast, single-target), Cannon (slow, splash AoE), Frost (slows enemies). Each has cost, range, and 2 upgrade levels (better damage/range/fire-rate). Towers can be sold for 70% of invested cost.
- Enemies: normal, fast, tank; a boss every 5th wave. Health bars above enemies. HP/speed/count scale each wave.
- Economy: start with 100 gold and 20 lives; kill bounties; wave-completion bonus; a leaked enemy costs 1 life (boss costs 5).
- UI: top bar (gold, lives, wave, score); tower shop panel with prices; click a shop tower then click the map to place (show range preview and valid/invalid cell highlight); click a placed tower to select it and show upgrade/sell panel; Start Wave button plus auto-start toggle; game speed x1/x2 button.
- Targeting: enemies closest to the exit first. Render projectiles. Show splash radius on cannon impact and a frost visual on slowed enemies.
- Win after beating wave 20; lose at 0 lives. Proper end screens with a Restart button.
- Polish: hover states, particle effects on kills, hit flashes, clean modern dark theme.
Output the COMPLETE index.html in one ```html code block. No placeholders, no TODOs — fully working code only.

Below is your current index.html for the tower-defense game. 1) Review it for bugs: logic errors, undefined variables, broken event handlers, targeting/pathing mistakes, economy bugs, rendering issues — fix them all. 2) Ensure every item of the original spec is implemented: 3 tower types with 2 upgrade levels + sell; 4 enemy types incl. boss every 5th wave; wave scaling; gold/lives/wave/score UI; place/select/upgrade/sell interactions with range preview; start-wave + auto-start + speed toggle; closest-to-exit targeting; projectiles + splash + frost visuals; win at wave 20 / lose at 0 lives with restart; particles and polish. 3) Balance: tune costs, bounties and wave scaling so waves 1-20 are beatable but challenging. 4) Polish: particles, hit flashes, nicer end screens. If the file below is cut off mid-code, complete it properly. Output the COMPLETE final index.html in one ```html code block.

Current file:
```html
{FILE}
```

Your single-file HTML5 tower-defense game below fails to load. The browser console shows this error:

    Uncaught SyntaxError: Unexpected token ')'

Find and fix the error. Also scan the whole file for any other syntax mistakes (unbalanced brackets/parentheses, broken statements) that would prevent it from running. Do not redesign anything — make the minimal corrections needed so the game runs. Output the COMPLETE fixed index.html in one ```html code block.

Current file:
```html
{FILE}
```

Raw round data

Prompt processing (prefill)

Prefill = server-side prompt eval time from llama.cpp. Short prompts are dominated by fixed overhead on every config; on long prompts prod processes ~1.5× faster than Flash-Next. In practice: feeding a ~50 KB file back for review costs ~18 s of prefill on prod vs ~45 s on Flash-Next, before any generation starts.

Method & pipeline

Sampling: temperature 0.7, top_p 0.8, max_tokens 26,624 per round, thinking disabled
(chat_template_kwargs: enable_thinking=false — mandatory: with thinking on, models burned
the entire budget reasoning; prod produced 52K chars of thought and zero code).

Protocol: each config got the same Round-1 build prompt, then Round 2 received its own
full file back for review/fix/balance/polish. The harness extracted the single ```html
block from each response; the last complete file became the published game.

Config A shipped with `flashColor='#ff4d9d');` — one stray parenthesis that killed the
whole game on load. Given only its file plus the console error, it regenerated all 50 KB
byte-identical except that character. Verified with node --check before republishing.

Serving: prod runs live on spark:8000; Flash-Next configs run on :8001 with prod stopped
(128 GB unified memory is one pool — CPU offload frees nothing while another model is
resident). Publishing: spark → orange ~/sites/bench → orange Caddy (handle_path /bench*)
→ wg umbilical → edge Caddy (TLS) → you.

Speed note: prod got FASTER on this task than on short prompts (49 vs 38.6 tok/s) —
long uninterrupted code generation is ideal for DFlash2 speculative decoding. Flash-Next
slowed as context grew (24.6 → 19.4 tok/s).

Spec: 20×12 grid, 3 tower types with upgrades/sell, 4 enemy types incl. bosses every 5th wave, economy, wave scaling, win at wave 20. Both Flash-Next runs independently branded their game “IRONHOLD” and out-designed prod; prod built a working game 3.5× faster. Verdict: prod for speed and agent duty, Flash-Next (E2) when craft beats latency.