Investigation run on prod · Qwen3.8-Flash-Next executor · 147 tool calls · 84 min · 2026-09-04 · items executed and measured the same day
Same model, same NVIDIA GB10. Where the 18 milliseconds per decoded token go, traced through both source trees, and an ordered item plan for Atlas.
qwen4exp), GGML_CUDA_GRAPHS=ONassist/flashnext-longform-fixes @ 7db64ccaThe report's first finding, that llama.cpp's CUDA graphs are compile-gated with no visible switch, is wrong for the prod build: ggml/CMakeLists.txt:209 declares option(GGML_CUDA_GRAPHS …) and the prod build's CMakeCache.txt has GGML_CUDA_GRAPHS:BOOL=ON, which defines GGML_CUDA_USE_GRAPHS. The agent grepped for the wrong spelling. Whether graphs are actually captured at runtime for this model (a CPU-resident PLE gather splits the graph; a mul_mat_id fallback would veto it) is a runtime question, answered below.
Measured on prod, 2026-09-04 19:04–19:12 (same container and arguments, 528-token prompt, two 1,200-token generations each, llama.cpp's own timings):
| Configuration | Decode tok/s (run 1 / run 2) | Wall per 1,200 tokens |
|---|---|---|
| Graphs enabled (build default) | 24.8 / 25.1 | 48.5 s / 48.0 s |
GGML_CUDA_DISABLE_GRAPHS=1 | 24.4 / 24.1 | 49.3 s / 50.0 s |
Conclusion: CUDA graphs are worth about 3% to llama.cpp on this model. The 18 ms/token gap is not about capture, so Atlas item A1 (per-region graph capture) drops to the bottom of the plan; the items that mirror what llama.cpp actually does per token move up: no host round-trips in MoE dispatch (A2), the PLE table resident in unified memory instead of NVMe rows (A3), and removing Atlas's 48 stream synchronizations per token (12 QSA device-to-host copies, 36 PLE barriers), which llama.cpp does not have at all.
| Item | Commit | Change | Result on the GB10 |
|---|---|---|---|
| A3 · PLE barrier + staging | 69cb6625 | event behind the gather kernel instead of a full stream sync; pinned slot staging; 4M-slot row cache (640 MB) by env | neutral — 14.5 / 15.1 tok/s. PLE sits on one model layer: one barrier per token, not 36 as the agent report assumed |
| A2 · MoE decode arm | ad7203f5 | ATLAS_MOE_DECODE_ARM=grouped: every layer's M=1 MoE through the tensor-core grouped-GEMM arm | 3× slower — 5.3 tok/s vs scalar 15.0. The scalar FP32-FMA gemv stays; the real A2 is a vectorised NVFP4 GEMV after an ncu bytes-per-token floor |
| A1 · device-side QSA selection | 0b9ada13 | qsa_select_topk writes the selection into sel_dev; removes 12 D2H syncs + host sorts per token; parity mode against the host arm | parity clean over 1,400 tokens. Below the 2,051-token inert bound: unchanged (15.4 / 15.3). With a 3,882-token prompt: 13.1 / 13.5 → 13.8 / 13.9 tok/s (+3–5%) |
| Graph capture (llama.cpp side) | — | GGML_CUDA_DISABLE_GRAPHS=1 on prod | 24.8 / 25.1 → 24.4 / 24.1 tok/s: ≈3% |
| Full benchmark ATLAS5 | 0b9ada13 | all three on, BF16 head + attention, watchdog off | 14.7 → 13.9 tok/s on 23K-token rounds (ATLAS4: 15.2 → 14.0); 62 KB game playable out of the box, no defect found (ATLAS4 had one) |
Conclusion after execution: host synchronisation, PLE placement and graph capture together account for at most a few percent. The remaining ~18 ms/token is in the decode kernels themselves — the scalar NVFP4 expert GEMV against llama.cpp's vectorised mul_mat_vec_q, the GDN recurrence and the 96 hyper-connection sites — which is kernel work (A2 subtask 2, A4, A5), not plumbing.
The agent's estimated per-token cost of each attribution item, before measurement; ranges overlap and are not additive. The prefill item (466 vs ≈300 tok/s) is a separate axis and is not on this chart.
Scope / honesty note up front. I read both trees, but the budget ran out before I could finish the Atlas-side trace. Everything below that is labelled llamacpp: is backed by code I actually opened (cited repo:path:line). Everything about Atlas is either (a) carried forward from the earlier Atlas-only anatomy report, which I treat as unverified here, or (b) an inference I flag as such. Two findings materially change the framing of the earlier report, and I say so explicitly:
USE_CUDA_GRAPH is only defined if GGML_CUDA_USE_GRAPHS / GGML_HIP_GRAPHS / GGML_MUSA_GRAPHS is set (llamacppggml/src/ggml-cuda/common.cuh:1227-1229), and the only places that string appears in the whole tree are common.cuh:1227 and the feature-report at ggml-cuda.cu:5444. There is no CMake option() for it anywhere in the tree I searched (grep for GGML_CUDA_USE_GRAPHS, GGML_CUDA_GRAPHS, GRAPHS under ggml/ returns only those two hits plus runtime comments). So in a stock build of this PR, use_cuda_graph stays false (ggml-cuda.cu:4252) and llama.cpp runs eager, ungraphed, per-node launches — and still beats Atlas by ~18 ms/token. I could not confirm the PR #27742 build flags, so this must be checked against the actual server banner (ggml_backend_cuda_get_features prints USE_GRAPHS=1 only if compiled in, ggml-cuda.cu:5444-5446). If graphs are off, the whole "Atlas has graphs, llama.cpp doesn't need them" story inverts: llama.cpp wins on per-op efficiency and per-token host cost, not on capture.llama-context.cpp:554-578: auto_fa, auto_fgdn, auto_flid, auto_fhc) but the fused ops are only emitted by glm-dsa.cpp, deepseek32.cpp, deepseek4.cpp, dots3note.cpp (search: fused_lid / fused_dsv4_hc_pre hit only those four). src/models/qwen4exp.cpp contains zero references to fused_lid, fused_gdn_ar, or fused_dsv4_hc_*. So Qwen4exp gets fused GDN only via the generic delta-net path (delta-net-base.cpp:436-442) and gets no fused indexer and no fused HC. Its sparse attention is built out of generic ggml ops.Model shape per token (n=1 stream): 48 layers total = 36 GDN + 12 full-attention-with-indexer, 512-expert top-10 MoE, 4-stream hyper-connections, PLE n-gram table, lm_head.
| Stage | llama.cpp mechanism | Atlas mechanism | llama.cpp graph-captured? | Atlas graph-captured? | host syncs / token | launches / token |
|---|---|---|---|---|---|---|
| embedding + PLE | dev_input is always CPU: "there is very little benefit to offloading the input layer, so always keep it on the CPU" (llama-model.cpp:1476-1477). PLE table forced to CPU by --override-tensor per_layer_token_embd=CPU; the gather is a GGML_OP_GET_ROWS on a CPU tensor, consumed by build_ple (qwen4exp.cpp:293-295). The CUDA backend does accept CUDA-host bufts on integrated devices (ggml-cuda.cu:5320-5324), i.e. on GB10 the "CPU" table is the same unified DRAM the GPU reads. | PLE rows read off NVMe with a row cache (per findings/prior report) | only if USE_CUDA_GRAPH; see Finding A | prior report: graph vetoed for this model (unverified here) | llama.cpp: 0 explicit syncs for PLE; the CPU gather is a scheduler split point (CPU subgraph → CUDA subgraph), which costs a node-boundary handoff, not a cudaStreamSynchronize | llama.cpp: 1 CPU gather + 1 cpy per PLE layer, per the generic get_rows path |
| GDN (36 layers) | build_qkvz = 2 mul_mat (qwen4exp.cpp:356-361), then GGML_OP_GATED_DELTA_NET — a native CUDA op, always supported on non-MUSA (ggml-cuda.cu:5283-5289), kernel in gated_delta_net.cu. Fused AR variant selected when cparams.fused_gdn_ar (delta-net-base.cpp:436), default true (llama-context.cpp:232-233) and validated at reserve time (llama-context.cpp:559-564). Output gate is rms_norm + sigmoid + mul (qwen4exp.cpp:366-376) | prior report: GDN recurrence spread over more, less-fused kernels | same caveat | no (vetoed) | 0 | llama.cpp ≈ 2 mm + 1 GDN + ~3 elementwise/norm = ~6 per GDN layer → ~216 |
| attention + indexer (12 layers) | build_qsa_top_k (qwen4exp.cpp:402-512): indexer k proj, cpy_k into an indexer cache, get_rows of pooled block members, an unrolled r-way sum loop (:457-462), scale, rms_norm, rope_multi, q proj + norm + rope, mul_mat score, relu, permute, sum_rows, get_rows expansion, add bias, then ggml_top_k (:505). Mask is built with fill(-INF) + fill(0) + set_rows + add (:556-579), then build_attn_mha (flash-attn). Hadamard rotations applied to q/k/v and back on the output (:527-533, :589-591). All of this is device-resident — top-k never touches the host. | prior report: indexer top-k also device-side | same caveat | no | 0 for top-k | llama.cpp: ~25-30 nodes per indexer layer → ~300-360 for the 12 attention layers. This is llama.cpp's noisiest stage. |
| MoE (512 experts, top-10) | GGML_OP_MUL_MAT_ID, one node per projection (gate/up/down = 3 nodes/token). Dispatch: ggml_cuda_mul_mat_id picks mul_mat_vec_q for ne2 <= MMVQ_MAX_BATCH_SIZE on quantized src0 (ggml-cuda.cu:1916-1931), else mmq (:1933-1936), else mmf (:1938-1941). The host-sorting fallback (:1944-2057) does cudaMemcpyAsync D2H of ids + cudaStreamSynchronize (:1972-1973), a CPU loop over all 512 experts (:1975-1988), another H2D + sync (:1993-1994), then a per-expert loop of mul_mat calls. That fallback is exactly what ggml_cuda_mul_mat_id_needs_sync predicts (:1874-1901) and it vetoes CUDA graphs for the whole cgraph (:2560-2571). For NVFP4, mmq is supported (mmq.cu:290-293) and turing_mma_available(cc) → true on Blackwell (mmq.cu:312-314), so decode takes the graph-safe vector/mmq path. | NVFP4 experts, custom grouped dispatch | graph-safe only because the mmid path avoids sync at bs=1 | no | 0 in the fast path; 2 + per-expert launches if it falls back | llama.cpp: 3 launches/token total for MoE (one per projection), regardless of 512 experts / top-10 |
| hyper-connections (4 streams) | build_hc_mix / build_hc_combine per block, 2× per layer (qwen4exp.cpp:298-325), plus hc_init as a repeat_4d (:285-288) and a final head mix (:332-334). GGML_OP_DSV4_HC_PRE/COMB/POST kernels exist (dsv4-hc.cu, support checks ggml-cuda.cu:5290-5299) but qwen4exp does not use them (Finding B) → generic norm/mul/add chains | prior report: HC as separate elementwise kernels | same caveat | no | 0 | llama.cpp: ~4-6 elementwise per HC block × 96 blocks ≈ ~400-500 elementwise nodes |
| lm_head | get_rows(inp_out_ids) on the mixed residual (:336-338) then one build_lora_mm(model.output, ...) (:343) — logits computed on device, only the sampled token (or n_outputs) crosses back | prior report: lm_head inside the (vetoed) graph attempt | same caveat | no | 0 extra | 2 |
| sampling | argmax/sample on device (GGML_OP_ARGMAX ggml-cuda.cu:2062); one D2H of the token id per step at the context boundary | prior report: sampling on host after a logits copy | n/a | n/a | llama.cpp: 1 (token id) | 1-3 |
| host bookkeeping | llama_context::decode → graph_reserve cached (sched_need_reserve, llama-context.cpp:582-586), ubatch split, memory apply (device-side cpy_k/cpy_v into the KV/indexer caches, qwen4exp.cpp:445, :549-550), ggml-backend-sched split/compute | prior report: per-token Rust-side orchestration + graph-update probing | — | — | llama.cpp: ~0 syncs, but a full ggml node walk per token when ungraphed | — |
Bottom line of the table: llama.cpp's per-token launch count is also large (my node-level estimate is 1000+ for this architecture) — it is not "one graph launch vs Atlas's many". What it is, is zero host synchronizations inside the token and no per-token host-side data-dependent control flow. That asymmetry, not capture, is the axis to reason about.
Ranked by plausible contribution. Each entry states what would confirm it.
1. Atlas's graph veto costs it the entire launch/sync tax that llama.cpp simply does not pay (est. 6-12 ms/token). Evidence: the prior Atlas-only report concludes the CUDA graph is vetoed for this model, so Atlas falls back to per-token eager dispatch. On the llama.cpp side the only constructs that would veto a graph are (i) the mul_mat_id host-sorting fallback (ggml-cuda.cu:2560-2571, :1944-1973) and (ii) arch checks (:4237). llama.cpp avoids (i) at bs=1 by routing NVFP4 through mul_mat_vec_q/mmq (:1916-1936, mmq.cu:290-293,312-314). If Atlas's veto is triggered by its own MoE dispatch doing a host round-trip to bucket experts, that is the same bug llama.cpp engineered around ([TAG_MUL_MAT_ID_CUDA_GRAPHS], :1873). Confirm: run Atlas with the veto reason logged; run llama.cpp with GGML_CUDA_DISABLE_GRAPHS=1 (common.cuh:1258) and with graphs compiled in, and diff tok/s. If the two llama.cpp numbers are within noise, Finding A holds and the gap is not about graphs at all.
2. PLE: NVMe row reads vs a resident unified-memory table (est. 4-8 ms/token, high variance). Evidence: llama.cpp keeps the PLE table in host memory (--override-tensor), and on GB10 the CUDA backend treats CUDA-host buffers as first-class on integrated devices (ggml-cuda.cu:5320-5324), so a PLE row read is a DRAM read, not an I/O. Atlas reads rows off NVMe with a row cache (findings). At 36 PLE-touching layers × 4 HC streams, a cache miss is a syscall-class event on the critical path. Confirm: hit-rate histogram of the Atlas row cache at steady state, plus iostat/blktrace during a 200-token decode. If miss rate is ~0, this item drops to the bottom of the list.
3. MoE dispatch shape (est. 2-5 ms/token). Evidence: llama.cpp issues 3 mul_mat_id nodes per token for 512 experts × top-10 (qwen4exp.cpp ffn path via generic build_ffn; dispatch ggml-cuda.cu:1903-1941), with the ids tensor consumed on device. Atlas's expert dispatch was not traced in this session — the prior report is thin here, which is itself the risk. If Atlas launches per-expert or per-top-k-slot (10 × 3 = 30 launches, or worse, a host-side bucketing pass), it pays both launch and bandwidth-fragmentation cost. Confirm: nsys trace of one Atlas decode step, count of _moe/gemv kernels and total bytes read from the expert tensors. Compare against llama.cpp's 3 kernels reading exactly top-10 expert slices.
4. Weight precision / bytes-per-token (est. 2-4 ms/token, and it may favour llama.cpp on GB10). Evidence: llama.cpp runs UD-Q4_K_XL with q8_0 KV; NVFP4/Q4_K both have dedicated mmq tensor-core instances (mmq.cu:275-293, mmq-config-blackwell.cuh, mmvq.cu:21,50,133). At bs=1 decode is bandwidth-bound, so the winner is whoever moves fewer bytes per token with better coalescing. Atlas's NVFP4 experts are nominally smaller, but if its kernel reads scale tensors non-coalesced or dequantizes to BF16 in registers/SMEM per element, effective bytes/token can exceed llama.cpp's mmvq path. Confirm: ncu dram__bytes_read per token for both engines; llama.cpp's mmvq NVFP4 config is :133 (4) / :154 (8) / :241 (5) — compare achieved vs theoretical bytes.
5. Prefill gap (466 vs 300 tok/s): chunked GDN + FA + no per-ubatch host work. Evidence: llama.cpp selects fused_gdn_ch (chunked autoregressive GDN) at reserve time (llama-context.cpp:561-562, delta-net-base.cpp:442) — a chunked scan is what makes 36 linear-attention layers prefill at all. It also reserves pp and tg graphs separately and sizes the compute buffer once (llama-context.cpp:631-672) so prefill never reallocates. Flash attention is auto-validated against the actual fused-op/device placement (:554-557), and FA on Blackwell uses the MMA path (fattn-mma-f16.cuh, BLACKWELL_MMA_AVAILABLE common.cuh:286-288). Atlas at 300 tok/s with BF16 KV is plausibly (a) not using a chunked GDN formulation, and (b) bandwidth-limited by BF16 KV writes being 2× q8_0's bytes. Confirm: Atlas prefill with a chunked GDN kernel if one exists behind a flag; measure KV-write bytes/token.
6. Sparse-attention representation (est. 1-3 ms/token, and this is where llama.cpp is worse-looking but wins anyway). Evidence: llama.cpp materializes a dense [n_kv, n_batch, 1, n_stream] F32 mask and rewrites it every token: fill(-INF) over the whole KV, fill(0) over top-k, set_rows, add (qwen4exp.cpp:556-579). At 128k context that is ~512 KB written per attention layer per token, ~6 MB/token, purely to express a 2048-wide selection. Atlas's top-k gather should beat that. That llama.cpp still wins despite this is the strongest single argument that the gap lives in items 1-3, not in the attention math. Confirm: mask-write bytes/token from ncu; if Atlas's number is already lower, delete this item from the plan.
7. Host bookkeeping / language boundary (est. 0-2 ms/token, unproven). llama.cpp's per-token host work is a cached ggml node walk with no allocation (sched_need_reserve short-circuit llama-context.cpp:582-586). Atlas's Rust orchestration cost was measured in the prior report but I did not re-verify it. Do not spend effort here before items 1-3 are measured.
What I could NOT confirm: (a) whether the PR #27742 build has GGML_CUDA_USE_GRAPHS on — no CMake option exists in the tree; (b) Atlas's MoE decode dispatch, GDN fusion state and lm_head path — not traced this session; (c) the actual n_stream/kv_unified setting used for the llama.cpp run (common/common.h:563 defaults kv_unified=false, i.e. n_stream = n_seq_max, llama-kv-cache.cpp:83; at n_seq_max=1 these coincide, which is presumably the measured config); (d) whether ggml_top_k uses CUB on this build — GGML_CUDA_USE_CUB is defined for CUDA ≥ 11.7 (common.cuh:110-112), so yes, CUB sort rather than the ≤1024-element fallback (ggml-cuda.cu:5261-5267).
A1 — Make the graph veto non-fatal: capture per-stage subgraphs instead of all-or-nothing. (flags only where noted) Change: in Atlas's capture path, replace the single whole-forward veto with per-region capture (GDN block, attention block, MoE block, HC, lm_head), so one unsupported region degrades locally. Mirror: llama.cpp's ggml_cuda_graph_check_compability scans nodes and only bails on the specific offender (ggml-cuda.cu:2548-2579), and its graph cache is keyed per first-node with an LRU sweep (common.cuh:1428-1451). Expected gain: 4-8 ms/token if the veto currently discards capture for everything. Risk: medium — cross-region tensor lifetimes; a wrong aliasing assumption is silent corruption. Guard with a bit-exact-vs-eager diff over 200 tokens. GB10 verify: nsys one decode step; count cudaGraphLaunch vs raw cudaLaunchKernel. Metric: tok/s 15-17 → target 22+; also cudaStreamSynchronize count/token must stay ≤1.
A2 — Remove every host round-trip from MoE decode dispatch (device-side ids, device-side bucketing). Change: Atlas expert dispatch must consume the router ids tensor on device and never read them back — exactly what ggml_cuda_mul_mat_id guarantees on its fast path (ggml-cuda.cu:1916-1941) and what ggml_cuda_mul_mat_id_needs_sync exists to detect before capture (:1874-1901). Target: 3 launches/token (gate/up/down), not 30. Expected gain: 2-5 ms/token; also unblocks A1. Risk: low-medium; grouped-GEMV occupancy on 512 experts needs a persistent-kernel or segmented approach. GB10 verify: kernel count/token in ncu (target 3), and expert-tensor dram bytes/token ≈ top-10 × expert size.
A3 — Replace NVMe PLE row reads with a resident unified-memory table + device gather. Change: pin the n-gram PLE table in (unified) DRAM and do the gather as a device get_rows-equivalent; keep NVMe only for cold-table paging. Mirror: llama.cpp never leaves the PLE gather's data path in the token loop — the table is a plain host tensor the scheduler reads (llama-model.cpp:1476-1477, qwen4exp.cpp:293-295) and on an integrated GB10 the CUDA backend accepts host bufts directly (ggml-cuda.cu:5320-5324). Expected gain: 3-6 ms/token at steady state, and removes the tail-latency spikes. Risk: low — pure memory-placement change; the only cost is DRAM footprint (the table is an embedding table, not the experts). GB10 verify: PLE-stage p50/p99 latency per token before/after; iostat -x 1 must show ~0 reads during decode. This is the item most likely to be a serve/config flag rather than code.
A4 — Fuse the indexer into one kernel (score + top-k + mask emission), and stop materializing a dense mask. Change: Atlas should emit a sparse selection consumed directly by attention. Do not copy llama.cpp here — its fill(-INF)/set_rows/add dense-mask rewrite (qwen4exp.cpp:556-579) plus the unrolled r-way pooling loop (:457-462) is ~25-30 nodes/layer. The kernel Atlas should look like is the one llama.cpp has but doesn't wire into this model: lightning-indexer.cu + GGML_OP_LIGHTNING_INDEXER (ggml-cuda.cu:5312-5313), used by deepseek4.cpp:643 via fused_lid. Expected gain: 1-3 ms/token at long context; larger at 128k. Risk: medium — numerics of top-k tie-breaking change attention output. GB10 verify: mask/selection bytes written per attention layer per token; logprob agreement vs reference over a fixed prompt set.
A5 — Chunked GDN for prefill (closes the 466 → 300 tok/s gap). Change: implement/enable a chunked-scan GDN for n_tokens > 1, keeping the recurrent form only for decode. Mirror: llama.cpp carries both forms and picks at reserve time — fused_gdn_ar vs fused_gdn_ch (llama-context.cpp:559-564, delta-net-base.cpp:436-442), native kernel gated_delta_net.cu, always-supported on CUDA (ggml-cuda.cu:5283-5289). Expected gain: prefill 300 → 400+ tok/s. Risk: medium — chunk-boundary state correctness; validate against the recurrent path at chunk boundaries. GB10 verify: llama-bench-style pp2048 on both engines; Atlas state-vs-recurrent max-abs-diff < 1e-3.
A6 — Reserve-once, reuse-forever buffer sizing keyed by (pp, tg) shapes. Change: precompute the compute-buffer high-water mark for the prompt and token graphs separately at load, so no per-request reallocation. Mirror: llama-context.cpp:631-672 reserves pp → tg → pp specifically to avoid ggml-alloc reallocation during inference, and warns when the actual size disagrees with the expectation (:490-499). Expected gain: 0 ms steady-state, removes first-token and long-context spikes. Risk: low. GB10 verify: allocation-count counter must be 0 after warmup; TTFT variance across 20 requests.
Serve-flag-only items: A3 (table placement) and the measurement of A1 (toggle capture on/off, GGML_CUDA_DISABLE_GRAPHS-equivalent for Atlas). A2, A4, A5 need code.
1Dense mask for sparse attention. It allocates and rewrites a full [n_kv, n_batch, 1, n_stream] F32 mask every token — fill(-INF) across the entire KV, then fill(0) + set_rows + add to unmask top-k (qwen4exp.cpp:553-579). Memory traffic scales with context length, not with the 2048-token selection. Atlas's sparse gather is the right design.
2Unrolled Python-style pooling loop in the graph. The block-mean over r indexer members is a host loop emitting r views + r-1 adds + a scale (qwen4exp.cpp:457-463), i.e. graph size grows with compress_ratio.
3Indexer and HC are unfused for this architecture. dsv4-hc.cu and lightning-indexer.cu exist and are gated behind fused_dsv4_hc_* / fused_lid (ggml-cuda.cu:5290-5299, :5312-5313), but qwen4exp.cpp emits none of them (Finding B) — the fusion exists for DeepSeek/GLM only. Atlas has headroom llama.cpp left on the table.
4A latent per-token host sync in MoE. If mul_mat_id ever falls off its fast path (non-F32 src1/dst, or a type/arch combo where mmq/mmf both decline — ggml-cuda.cu:1874-1900), it does two cudaStreamSynchronize calls plus a CPU loop over all 512 experts and a per-expert launch loop (:1944-2057). That single node silently disables CUDA graphs for the entire forward pass (:2560-2571). Fragile, and shape/quant-dependent.
5CUDA graphs are opt-in at compile time with no visible switch. USE_CUDA_GRAPH requires GGML_CUDA_USE_GRAPHS (common.cuh:1227-1229) and no CMake option surfaces it in this tree (Finding A). A default deployment is ungraphed. Atlas's "graph by default, degrade explicitly" posture is better engineering.
6Many small elementwise nodes around every gated norm. rms_norm → sigmoid → mul (qwen4exp.cpp:366-376), attention output sigmoid → mul (:670-674), plus the Hadamard rotations before and after attention (:527-533, :589-591) — each its own launch. Atlas's fused elementwise chains are preferable.
7ggml_top_k is a full sort-based selection (argsort/CUB path, ggml-cuda.cu:5261-5267, top-k.cu:39-98) rather than a partial top-k over a 128k-wide score vector.
One-line summary: llama.cpp is not faster because it is more modern — it is faster because it keeps every data-dependent decision (MoE ids, indexer top-k, KV slot indices) on the device with zero host synchronizations per token (ggml-cuda.cu:1916-1941, qwen4exp.cpp:402-512), keeps the PLE table in resident memory rather than on NVMe (llama-model.cpp:1476-1477), and carries a chunked GDN kernel selected at load time (llama-context.cpp:559-564). Atlas's plan is A2 → A3 → A1 first; A4/A5 next; and it should keep its own sparse-mask and fusion designs, which are better than llama.cpp's.