🤖 Agentic shootout — Atlas × Flash-Next research

Atlas x Qwen3.8-Flash-Next: gap analysis

(a) Overview of Atlas's current architecture

Atlas is a single-GPU-first, Rust inference engine organized around a registry-driven model factory and a model-agnostic runtime:

• - Config layer (crates/atlas-core/src/config.rs + config/{dispatch,factory,parsers,gguf}): a single ModelConfig struct is the source of truth, parsed from HF config.json (per-family parser modules under config/parsers/) or GGUF (config/gguf.rs). It already encodes hybrid layer layouts (LayerType, layer_types, full_attention_interval), GDN/SSM dims (linear_num_key_heads, linear_conv_kernel_dim, …), MoE fields (experts, top-k, shared expert, norm_topk_prob, decoder_sparse_step, per-layer heterogeneous sizes), MLA/low-rank fields, YaRN/MRoPE (mrope_section, mrope_interleaved), MTP/DSpark fields, and quantization metadata (quantization_config → QuantFormat dispatcher).
• - Model factory (crates/spark-model/src/factory.rs): loader_for_config() is the only place model_type strings are matched; adding a model = implementing ModelWeightLoader + one match arm. Supported families include qwen3_next, qwen3_5/3_6_moe, qwen3_vl_moe, nemotron_h, minimax_m2, step3p7, deepseek_v4, gemma4, mistral, laguna, m2m_100.
• - Layers (crates/spark-model/src/layers/): a mature Gated Delta Net implementation (qwen3_ssm/, incl. FLA-style chunked prefill, WY kernels, multi-seq batched decode, FP16 h-state), full attention with gated Q (attn_gated), MLA + DeepSeek-V4 CSA/HCA compressed attention (qwen3_attention/prefill/cache_skip_v4.rs, decode/attention_forward_v4.rs), mHC hyper-connections (layers/ops/hyper_connection.rs), Mamba-2 (nemotron_mamba2/), MoE with softmax / sigmoid+bias / sqrtsoftplus routing (layers/moe/, layers/ops/moe_gate.rs), MTP heads (single/multi, mtp_head/, mtp_multi.rs, deepseek_v4_mtp.rs), DFlash block-diffusion drafter (dflash_head/), and a vision encoder (vision_encoder/).
• - Runtime (crates/spark-runtime/): paged KV cache with FP8/NVFP4/turbo KV dtypes, prefix caching + radix tree, buffer arena sized from ModelConfig, CUDA graphs, cuBLASLt/CUTLASS/FlashInfer backends, sampler, EP/TP sharding (spark-model/src/tp_shard.rs, layers/ep_dispatch.rs), expert-tier RDMA paging (spark-storage/src/expert_peer.rs, expert_tier_rdma.rs).
• - Server (crates/spark-server/): OpenAI/Anthropic APIs, thinking/enable_thinking/reasoning_effort handling, tool-call parsing, grammar-constrained decoding, speculative decoding modes (--speculative MTP, --self-speculative, --ngram-speculative, --dflash), per-model tuning via kernels/<target>/<model>/MODEL.toml (sampling presets, match_names, kernel_source reuse).

Notably, the repo already carries a kernels/gb10/qwen3.8-27b/ target whose MODEL.toml documents a dense sibling of the same architecture family (GDN + full attention + MRoPE-interleaved + gated attention, mrope_section=[11,11,10], partial_rotary_factor=0.25), reusing kernel_source = "qwen3.6-27b" — strong evidence that much of the Flash-Next feature surface is already exercised in-tree.

(b) Feature-by-feature gap table

(c) Prioritized adoption plan

1. 1. P0 — Config parsing & registration for qwen4exp (Complexity: Low–Medium).

Add config/parsers/qwen4exp.rs (clone the qwen3.5/3.6 dispatch path in config/dispatch.rs), set attn_gated, mrope_section/interleaved, partial_rotary_factor, layer-type layout, MoE routing kind; add the "qwen4exp" arm in factory.rs::loader_for_config. Reuse Qwen35WeightLoader/Qwen35DenseWeightLoader if tensor names match (the qwen3.8-27b precedent shows name-level identity across 3.6/3.8).

1. 2. P0 — Weight loader validation + quant-format routing (Complexity: Low if names match, Medium otherwise).

Verify checkpoint tensor names against the qwen35 loader arms (weight_loader/qwen35/load_layers/attention_arms.rs, qwen35_dense.rs); route mixed FP8-channel + NVFP4 configs through the existing QuantFormat dispatcher and the per-row-FP8 prefill arm; document any double-quantization (per-row FP8 → BF16 → NVFP4) exactly as done for qwen3.8-27b.

1. 3. P1 — Semantic-indexer sparse attention (if enabled by the checkpoint's index_topk > 0) (Complexity: High).

Atlas has the compressed-KV pool infrastructure (V4 csa_compress, FP8 compressed-block persistence) but no indexer: needs indexer projections (q/index weights), a top-k selection kernel over compressed blocks, gather-based attention over selected blocks in both prefill and decode, and pool bookkeeping in kv_cache/prefix cache. This is the single largest genuine kernel gap.

1. 4. P1 — MoE routing-kind selection (Complexity: Low–Medium).

Wire the checkpoint's routing (sigmoid+correction-bias vs softmax vs sqrtsoftplus) through the existing selection logic in layers/moe/init.rs/forward.rs; check num_experts ≤ 512, top_k ≤ 32 kernel bounds; add a MODEL.toml kernel target with match_names if config alone cannot disambiguate from qwen3.5/3.6 (the documented tie-breaking requirement in atlas-kernels/src/resolve.rs).

1. 5. P1 — Speculative decoding config (Complexity: Low).

Map Flash-Next's MTP module count to mtp_num_hidden_layers and existing MTP heads; if the reference recipe combines n-gram + MTP drafting, extend cli/validate.rs and the scheduler proposer chain to allow composition (Complexity: Medium, scheduler work).

1. 6. P2 — Chat template & server behavior presets (Complexity: Low).

Add jinja-templates/qwen4exp.jinja (or rely on the checkpoint-embedded template), MODEL.toml with card-derived sampling presets (thinking/non-thinking/tools, penalty-free tools per family rule), thinking_default, loop-watchdog settings — mirroring kernels/gb10/qwen3.8-27b/MODEL.toml.

1. 7. P2 — Long-context & KV budget validation (Complexity: Medium, mostly measurement).

Re-derive effective_context and sliding-window interaction for Flash-Next's context length; validate FP8-KV calibration path (fp8_calibration.rs, --fp8-kv-headroom) and GDN state memory (48/64 linear layers ⇒ per-seq state cost) against the buffer arena sizing in spark-runtime/src/buffers/sizes.rs.

1. 8. P3 — Local expert-streaming for very large MoE variants (Complexity: High).

If a Flash-Next MoE variant exceeds single-GPU weight budget, build on spark-storage expert-tier machinery to add a local mmap/pinned-RAM expert tier (pinned_hosts.rs exists) with async H2D prefetch on routed expert ids.

(d) Risks / unknowns

• - Indexer semantics unverified: the reference's exact indexer (scoring function, block granularity, whether selection is per-head or shared, whether compressed KV is RoPE'd at block position) is not pinned down; Atlas's V4 compressed pool assumes scale-1.0 FP8 persistence (debug_assert in cache_skip_v4.rs:564) which may not hold.
• - Config ambiguity / loader mis-selection: qwen4exp may be numerically indistinguishable from qwen3_5/qwen3_6 configs (the qwen3.8-27b MODEL.toml documents an exact-tie that required match_names); a wrong loader silently changes numerics rather than failing.
• - Mixed-precision quant configs: per-row (CHANNEL) FP8 tensors cannot feed w8a16's [N/128,K/128] block-grid indexing; the dequant→NVFP4 fallback is deliberate but costs accuracy — the same trap is documented and must be re-checked tensor-by-tensor on the Flash-Next checkpoint.
• - GDN gate-form drift: if Flash-Next changes the gate recurrence (e.g. exp(-A·softplus(α+dt_bias)) variant, L2-norm placement, beta form), Atlas's heavily-optimized kernel zoo (WY/FLA/reg-resident, FP16 h-state) must be re-derived bit-exactly; the cos-gate test harness exists but each variant multiplies maintenance.
• - Sliding-window + GDN + sparse-indexer interaction: V4's raw arm is hard-windowed (V4_WINDOW=128 const, not config-driven); porting that pattern to Flash-Next risks OOD behavior ("prompt-length salad" noted in code) if window/compression split differs.
• - Scheduler combinability of n-gram + MTP drafting and the auto speculative arbiter behavior under a new model is untested.
• - Prefix caching with heterogeneous state (paged KV + GDN h/conv snapshots + any compressed-block pool) is the most subtle correctness surface; cross-request state-leak bugs of exactly this class were fixed in V4 (v4_comp_pool_filled comment) and would need targeted tests for Flash-Next.
• - Non-CUDA backends (Metal, strix-hip): the new kernels (indexer, routing variants) would initially exist only for gb10; other targets degrade or fail to resolve.

Atlas x Qwen3.8-Flash-Next: gap analysis

(a) Atlas current architecture

Atlas is an AGPL-3.0 inference stack for NVIDIA GB10 / DGX Spark (unified memory). Relevant pieces I inspected:

• - crates/spark-model/ — model assembly. factory.rs::loader_for_config is the only place model_type strings are matched; each family gets a ModelWeightLoader impl under weight_loader/ (qwen3, qwen35, qwen35_dense, qwen3_vl, minimax, nemotron, gemma4, deepseek_v4, step3p7, laguna, mistral, nllb). Layers live in layers/: qwen3_ssm/ (Gated Delta Net), qwen3_attention/ (shared GQA/MLA/sparse-attention stack), moe/ (routed + shared expert, BF16/FP8/NVFP4), mtp_head/ + mtp_multi.rs (speculative drafter), vision_encoder/.
• - crates/spark-runtime/ — GPU backend, kv_cache/, prefix_cache/, radix_tree/, buffers/, sampler/, weights/.
• - crates/atlas-core/src/config/ — ModelConfig (SSOT for dims), dispatch.rs (per-model JSON parser), parsers/ (deepseek_v4, gemma4, minimax, step3p7, mistral, laguna, vision, quantization).
• - crates/spark-server/ — OpenAI/Anthropic HTTP, tokenizer (tokenizers crate, byte-level BPE), Jinja chat templates in jinja-templates/.
• - kernels/<hw>/<model>/<quant>/MODEL.toml — per-model kernel target + sampling/behavior presets.
• - crates/atlas-kernels/ — Rust glue over compiled PTX; one artefact per (hw, model, quant).

Existing model support matrix (from factory.rs): qwen3_next, qwen3_5_moe, qwen3_5, qwen3_6_moe, holo3_1_moe, qwen3_vl_moe, nemotron_h, nemotron_h_puzzle, gemma4, mistral, minimax_m2, step3p7, laguna, deepseek_v4, m2m_100.

(b) Feature-by-feature gap table

Atlas x Qwen3.8-Flash-Next: gap analysis

Research caveat: this report is based on direct inspection of the Atlas tree (atlas-core config layer, spark-model factory/loaders/layers, spark-runtime KV/buffers/backends, spark-server tokenizer/tool-parsing, kernels/gb10/* target manifests) plus the Flash-Next architecture reference. A few late verification searches were truncated (notably: generic CPU/mmap weight-residency per tensor group, and the exact spark-quant crate layout). Items resting on those are marked unverified rather than asserted as gaps.

(a) Atlas today — architecture overview

Atlas is a single-binary, single-hardware (NVIDIA GB10 / SM121) inference engine organized around seven trait/convention plug points, all confirmed in code:

1. 1. Config layer — crates/atlas-core/src/config.rs defines one ModelConfig as the "single source of truth" for dimensions and feature flags. config/dispatch.rs::parse_config probes model_type and routes to family parsers (config/parsers/{deepseek_v4,gemma4,minimax,mistral,step3p7,laguna}.rs, plus inline Qwen3.5/3.6/Nemotron arms). Parsers normalize HF field names into canonical Atlas fields and set architecture flags (attn_gated, mrope_interleaved, scoring_func, hc_mult, hc_sinkhorn_iters, hc_eps, compress_ratios, index_n_heads/index_head_dim/index_topk, num_hash_layers, kv_lora_rank, moe_latent_size, …).
2. 2. Loader registry — crates/spark-model/src/factory.rs::loader_for_config is documented as "the ONLY place model_type strings are matched"; ~14 loaders under spark-model/src/weight_loader/ implement ModelWeightLoader. factory::build_model assembles a model-agnostic TransformerModel (embed → [layers] → norm → lm_head).
3. 3. Layer composition — spark-model/src/layers/ holds composable TransformerLayer impls: qwen3_attention/ (GQA + paged KV), qwen3_ssm/ (Gated DeltaNet: QKVZ proj, causal conv1d, delta-rule recurrence, gated RMS-norm — decode / batched decode / FLA-chunked prefill / WY32 / reg-resident kernel families), nemotron_mamba2/, moe/ + nemotron_moe/ (routed + shared experts, LatentMoE, sigmoid/sqrtsoftplus scoring, routing bias), dense_ffn.rs, mtp_head/ + mtp_multi.rs, deepseek_v4_mtp.rs, dflash_head/, vision_encoder/, and layers/ops/hyper_connection.rs (mHC expand/collapse/Sinkhorn ops).
4. 4. Runtime — spark-runtime: kv_cache.rs (KvCacheDtype, 16 variants incl. turbo*/asymmetric), prefix_cache/ + radix_tree/, kv_spill.rs (NVMe offload via StorageBackend), buffers/ (arena sizing incl. hc_streams/hc_post/hc_comb, moe_latent buffers), gpu/ (GpuBackend trait; CUDA + Metal backends), sampler/, weights/ (WeightStore, O_DIRECT fast loader with mmap fallback).
5. 5. Kernels — kernels/<hw>/<model>/<quant>/ auto-discovered by build.rs; kernels/gb10/deepseek-v4-flash/MODEL.toml carries per-model geometry (hc_mult = 4).
6. 6. Serving — spark-server: OpenAI/Anthropic APIs, scheduler (fifo/slai), MTP + DFlash speculative decoding (spark-model/src/speculative.rs, preflight.rs MTP_SUPPORTED_MODEL_TYPES), tokenizer/ (jinja rendering, effort/thinking directives — openai/tests/thinking.rs shows a full enable_thinking/budget/effort pipeline with reasoning_content streaming), tool_parser.rs (ToolCallParser trait; hermes JSON and qwen3_coder XML inner formats under shared `

Qwen3.8-Flash-Next — architecture reference (qwen4exp)

Released 2026-08-26 by the Qwen team; the first public preview of the Qwen4 architecture.

HF: Qwen/Qwen3.8-Flash-Next (BF16, ~355 GB) and Qwen/Qwen3.8-Flash-Next-FP8.

Config model_type: qwen4_exp. Not supported by released Transformers/vLLM as of 2026-08-28;

llama.cpp support exists only in unmerged PR ggml-org/llama.cpp#27742.

Key architectural features an inference engine must implement:

1. 1. Sparse MoE: 125B total parameters, ~6B active per token. 512 experts, top-10 routing,

plus a gated shared expert. Standard grouped expert FFN kernels apply, but the router

and shared-expert gating differ from Qwen3-MoE.

1. 2. Gated DeltaNet layers: 3 of every 4 layers are linear-attention (recurrent, constant-size

state) DeltaNet blocks with a sigmoid output gate. No KV cache for these layers; instead a

per-sequence recurrent state that must be tracked, checkpointed for batching, and reset per sequence.

1. 3. Qwen Sparse Attention (QSA): the remaining 1/4 of layers use budget-based sparse attention —

an indexer selects which cached KV entries each query attends to under a token budget.

Quantized KV cache interacts poorly with rotation tricks (llama.cpp needed an attn-rot disable).

1. 4. Per-layer n-gram embeddings (PLE): an auxiliary ~51B-parameter n-gram embedding table,

hashed lookups per layer. Can be executed host-side (CPU/mmap) to keep it out of GPU memory;

engines should support placing this tensor group on a separate backend/device.

1. 5. Low-rank hyper-connections: hc_count = 4 residual streams mixed by learned low-rank

projections (replaces the single residual stream; affects the whole block dataflow).

1. 6. Multi-token prediction (MTP): the checkpoint ships an MTP head usable for speculative

decoding (analogous to Qwen3.5's MTP-2).

1. 7. Multimodal: image+video input through a CLIP-style vision encoder (separate mmproj in GGUF).
2. 8. Context: 262,144 tokens native. Thinking/non-thinking chat modes via chat template

(enable_thinking flag); recommended sampling: instruct temp 0.7 top_p 0.8, thinking temp 1.0 top_p 0.95.

1. 9. Tokenizer: Qwen3-family BPE, extended special tokens for tool calling; standard Qwen chat template

with tool-call JSON blocks.

Quantization notes: community GGUFs exist (Unsloth UD-Q3_K_XL 84 GB, UD-Q4_K_XL 111 GB).

On 128 GB unified-memory hosts the PLE table is typically offloaded to CPU/mmap.

Flash-Next support: vLLM vs Atlas

Scope note: this report is based on a deep read of the vLLM PR #53896 tree (vllm/models/qwen4_exp/, config plumbing, cache specs) and on targeted searches of the Atlas tree. A few Atlas-side confirmations (exact kernel inventories per feature) were still in flight when research was cut off; where Atlas status is stated with lower confidence it is flagged as such. All findings below are from files actually read; inferences are labeled.

---

(a) How vLLM implements qwen4exp, feature by feature

0. Overall layout

• - Package vllm/models/qwen4_exp/ with config.py (root), common/ (platform-neutral: HC math, PLE helpers, QSA cache specs/metadata), nvidia/ (model, QSA owner, indexer, PLE layer, MTP, fused Triton ops under nvidia/ops/), amd/.
• - Config shim: vllm/transformers_utils/configs/qwen4_exp.py re-exports from vllm/models/qwen4_exp/config.py.

1. Config conversion — vllm/models/qwen4_exp/config.py

• - Qwen4ExpTextConfig subclasses Qwen3NextConfig and adds: hc_count (4), hc_lowrank (320), ple_layer_ids (1-based), ple_embed_dim, ple_conv_kernel_size (4), ngram_size (3), heads_per_ngram (8), ngram_vocab_size_base (20M), make_ngram_vocab_size_divisible_by (128), output_gate_type, rope_parameters, layer_types.
• - QSA fields validated together: indexer_n_heads, indexer_kv_heads (must be 1 — MQA), indexer_head_dim (must cover rotary dim), indexer_budget, indexer_compress_ratio; budget / compress_ratio must be 512 or 2048 (block-topk granularity).
• - Derived helpers: short_conv_layer_ids (0-based PLE layers), short_conv_state_shape = (hidden*hc_count, (ple_conv_kernel-1)*ngram_size), ngram_context_len = ngram_size-1.
• - Qwen4ExpConfig is a VL wrapper (vision_config = Qwen3-VL vision, text_config = above).

2. Model skeleton — nvidia/model.py

• - Qwen4ExpDecoderLayer: per layer, optional Qwen4ExpPLELayer (when layer_idx+1 ∈ ple_layer_ids), then either linear_attention → QwenGatedDeltaNetAttention (shared qwen3_next GDN, gqa_interleaved_layout=False) or full_attention → Qwen4ExpQSAAttention (if indexer config present) else plain Qwen3NextAttention; MLP is Qwen4ExpSparseMoeBlock (subclass of Qwen3NextSparseMoeBlock, rejects sequence-parallel MoE) or dense Qwen3NextMLP per mlp_only_layers / decoder_sparse_step.
• - Multi-stream hidden state: embeddings are repeat(1, hc_count) → [T, HC*H] flows through every layer; blocks see a mixed single stream.
• - Delayed-combine dataflow: each layer returns (hidden_states, block_output, injection); the *next* HC module fuses the pending combine into its own mix via combine_and_mix(). PLE additions and deepstack insertions force an eager combine() first (they add to the materialized multi-stream state). PP boundaries and the final hyper_connection_mixer also materialize via combine.
• - Final mixer combine_and_mix returns both the sampled single stream and the pre-mixer multi-stream state, copied into _mtp_hidden_buffer for the MTP drafter (zero extra compute).
• - Weight loading: AutoWeightsLoader + WeightsMappers — HC input_mix_weight_down and block_inject_weight are packed into one merged column-parallel GEMM (_HC_WEIGHTS_MAPPER, padded to 16 rows); QSA cache-scale names remapped onto the merged QSA owner (_remap_qsa_cache_scale_name); non-persistent runtime buffers (hashstats_*, token_lookup, mixer inject) skipped; shared expert optionally fused into routed experts (maybe_fuse_shared_experts); modelopt_fp4 excluded for QKV (without_modelopt_fp4).
• - requires_raw_input_tokens = True; the runner supplies query_start_loc and ngram_context forward kwargs (_maybe_add_ngram_kwargs), declared dynamic in @support_torch_compile.
• - Reuses Qwen3-VL vision tower + deepstack; deepstack embeds are broadcast to [HC,H] and added after terminating the delayed combine.

3. Hyper-connections — common/hyperconnection.py + nvidia/hyperconnection.py + nvidia/ops/hc.py

• - Math (reference common/): per-stream GroupedGemmaRMSNorm (each H-sized stream normalized independently, affine over the full HC*H layout); mix = sigmoid(up(silu(down(xn)/hc_count))) ⊛ xn, mean over streams; combine = residual + block_out * (2·sigmoid(inject(xn)/hc_count)).
• - NVIDIA variant: projections are vLLM Linears (so quant/low-latency GEMM dispatch works); down+inject fused into MergedColumnParallelLinear with [lora_rank, hc_count, pad] output shards, disable_tp=True (HC is replicated); glue is Triton: grouped_gemma_rmsnorm, hc_silu, hc_gate_mix, hc_combine, and the key fusion hc_combine_norm (fuses pending combine + next RMSNorm).
• - mix() returns (hyper_input, block_input, injection_logits); combine_and_mix() consumes the previous block's output+injection.

4. Gated DeltaNet (linear attention)

• - Fully inherited from qwen3_next infra: vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.QwenGatedDeltaNetAttention with in_proj_qkvz/in_proj_ba packing; state via MambaStateShapeCalculator.gated_delta_net_state_shape (conv state + SSM state, num_spec-aware for spec decode); dtypes via gated_delta_net_state_dtype (bf16 conv, optional fp32 SSM via mamba_cache_dtype/mamba_ssm_cache_dtype).
• - Hybrid scheduling handled by IsHybrid + HasInnerState; mamba_cache_mode="all" is rejected (needs align).

5. PLE / per-layer n-gram embedding — nvidia/ple_layer.py (+ common/ple.py)

• - Qwen4ExpPLELayer per selected layer: n-gram hashing of token IDs (ngram_size, heads_per_ngram, hashed n-gram vocab ~20M rounded by 128) → per-head embedding lookups → concat to ple_embed_dim; a short dilated causal conv (ple_conv_kernel_size, dilation = ngram_size) whose state is cached as a SHORT_CONV mamba state of shape [(ple_conv_kernel-1)*ngram_size, hidden*hc_count] — i.e., the conv runs directly on the multi-stream HC state (channels = HC·H), and PLE output is added to the multi-stream state.
• - Cross-token n-gram context at request boundaries is passed as ngram_context (ngram_context_len = ngram_size-1 previous tokens) plus query_start_loc so conv/hash never crosses request boundaries.
• - PLE state cache is registered as a second MambaSpec (tp_replicated=True) via get_ple_mamba_state_shape_from_config (conv kernel +1 for spec tokens); copy func short_conv_state_copy_func.
• - Runtime-only tables (hashstats_*, token_lookup) are non-persistent and skipped at weight load. Note: PRs #53899/#54129/#54371 (UVA/mmap PLE offload) are *not* in this tree; here the ~20M-entry n-gram table lives on-device — a key memory item for Atlas/GB10.

6. QSA sparse attention + indexer — nvidia/qsa.py, nvidia/indexer_qsa.py, common/qsa_cache.py, nvidia/ops/qsa*.py

• - Merged owner: Qwen4ExpQSAAttention owns the main paged KV cache (FullAttentionSpec, BF16 only, no KV quant, no ALiBi/sinks/sliding-window/CP) *and* the indexer side branch, and registers one custom op qwen4_exp_qsa_with_output so the whole index → KV-update → sparse-attend transaction is opaque to torch.compile.
• - Dataflow (_run_qsa): indexer produces topk_indices_buffer[T, budget + compress_ratio - 1] (request-relative token indices, -1 padded) → do_kv_cache_update (normal paged write) → Triton qsa_sparse_paged_attention reads manager pages directly with logical_indices + block_table + token_to_req. Attention itself: GQA with sigmoid output gate (always on), GemmaRMSNorm q/k heads, fused QK-norm+RoPE+gate Triton path when possible.
• - Indexer (QSAIndexer): replicated index_qk_proj (n Q heads + 1 KV head, head_dim 128, rotary 64), GemmaRMSNorm q/k, same RoPE as main attention (incl. interleaved MRoPE). Two side caches:
• - QSAKeyStateCache — raw keys in a per-request circular ring (CircularBufferSpec, capacity = compress_ratio + num_spec rounded to whole groups, must divide block size; optional int64 MRoPE positions packed into the same BF16 rows).
• - QSACompressedKeyCache — one compressed key per complete group of compress_ratio tokens (mean-pool → GemmaRMSNorm → RoPE at group's first position), stored as MLAAttentionSpec with tokens_per_state=compress_ratio so its block table follows the main KV lifecycle.
• - Selection (qsa_select_paged_tokens): score q vs compressed keys (weight-free — no learned score MLP), pick block top-k (budget/compress_ratio ∈ {512,2048} groups), expand selected groups back to token indices (hence budget + compress_ratio - 1 width). Fused qsa_pre_indexer Triton kernel does norm+RoPE+raw-store+group-compress+compressed-store in one pass (conditions: head_dim=128, rotary=64, kv_heads=1, power-of-2 ratio).
• - MTP: skip_topk reuses step-0 selections on later draft steps while still updating the side cache.
• - Metadata: QSAMetadataBuilder + Triton _build_qsa_metadata_kernel computes token_to_req, logical_positions, circular/compressed slot mappings and compressor work-list (k_work_metadata) on-device (CUDA-graph-safe, PDL-annotated); torch fallback provided.

7. MoE

• - Reuses Qwen3NextSparseMoeBlock (renormalized top-k routing, shared expert via shared_expert_intermediate_size, fused-shared-expert packing, EPLB via Qwen4ExpMixtureOfExperts protocol). Only additions: SP-MoE rejected under HC; n_shared_experts derived from config.

8. MTP — nvidia/mtp.py

• - Drafter consumes get_mtp_target_hidden_states() — the multi-stream pre-final-mixer hidden [T, HC·H] ("scheme A"), so the drafter's first step feeds a real HC backbone state; QSA side caches keep updating with skip_topk semantics; GDN/short-conv state shapes account for num_speculative_tokens.

9. Misc

• - low_latency_gemm.py: swaps in a skinny-GEMM path for HC's tiny projections (M×320-ish) at model init.
• - model_state.py: Qwen4ExpModelState (per-forward state object for the runner, incl. ngram kwargs preparation).
• - Token handling: standard Qwen chat template tokens ([IM_START], [IM_END], [think]/[/think] etc. via the Qwen3-VL processor) — no template changes in this PR.

---

(b) Atlas current status per feature

(Inferred from searches of the Atlas tree; confidence noted.)

---

(c) Side-by-side comparison

---

(d) Prioritized Atlas adoption plan

Phase 1 — config & skeleton (1–2 wks).

1. 1. Add Qwen4ExpConfig/TextConfig with all hc/ple/qsa fields and port vLLM's validators verbatim (they encode kernel constraints: indexer_kv_heads==1, budget/ratio ∈ {512,2048}, rotary_dim ≤ indexer_head_dim, ring capacity divides block size, ple_embed_dim % ((ngram_size-1)*heads_per_ngram) == 0).
2. 2. Layer-type dispatch (linear vs full), MoE/dense MLP selection from decoder_sparse_step/mlp_only_layers.

Phase 2 — hyper-connections (highest structural impact, do early).

1. 3. Switch the decoder stack to carry [T, HC·H]; implement mix/combine/combine_and_mix with vLLM's delayed-combine dataflow — it is what keeps the extra HC·H bandwidth to ~1 extra read+write per block instead of per mix+combine. Write fused combine+RMSNorm and gate-mix kernels; merge down+inject into one GEMM padded to 16 rows; keep HC projections un-quantized and un-TP'd (vLLM: disable_tp=True, without_modelopt_fp4 on QKV).
2. 4. PP/last-rank mixer equivalents: materialize before any external addition (deepstack, PLE) and at stage boundaries.

Phase 3 — GDN + dual state cache.

1. 5. Reuse/verify Atlas GDN against QwenGatedDeltaNetAttention semantics (non-interleaved layout, a/b gating, conv+SSM states, spec-token extension).
2. 6. Generalize Atlas's recurrent-state manager to multiple state specs per model (GDN per-head-sharded; short-conv TP-replicated; page size = max across specs), mirroring get_mamba_specs_from_config. Adopt "align"-style prefix-cache semantics only; vLLM rejects all.

Phase 4 — PLE.

1. 7. Implement hashed n-gram lookup + per-head embedding + dilated short conv over HC·H channels, with conv state as the second mamba spec and ngram_context (ngram_size-1 tokens) threaded through the scheduler/runner alongside query_start_loc. In Rust, the hash/lookup kernel is straightforward; the runner plumbing (raw input tokens to forward, per-request boundary masking) is the real work.
2. 8. Diverge on memory: the ~20M×ple_embed_dim table is large on a unified-memory GB10. Here Atlas can do *better* than this vLLM tree: implement the future PRs' idea natively — mmap the embedding table and expose it to CUDA via UVA/zero-copy (GB10's coherent C2C makes host-resident PLE attractive; consider GB10-resident with L2-friendly hashing). This is a Rust/GB10-specific divergence vLLM's separate PRs validate.

Phase 5 — QSA.

1. 9. Build the merged QSA owner: main FullAttentionSpec BF16 cache + indexer side branch, wrapped as one opaque transaction (Atlas's graph capture must treat index→KV-write→sparse-attend as one node, like qwen4_exp_qsa_with_output).
2. 10. Side caches: per-request raw-key ring (capacity compress_ratio + num_spec rounded to groups, must divide block size) and compressed-key cache whose block table shares the main KV block table lifecycle (vLLM's tokens_per_state trick — in Atlas, alias the compressed cache's page mapping to the main block table at 1/ratio density).
3. 11. Kernels: pre-indexer fusion (norm+RoPE+store+group-compress), weight-free scoring of compressed keys, block top-k (512/2048 groups), group→token expansion (budget + ratio - 1, -1 padded), and a sparse paged GQA kernel consuming logical indices. Port vLLM's on-device metadata kernel semantics (token_to_req, logical_positions, slot mappings, compressor work-list) into Atlas's CUDA-graph-safe metadata builder.
4. 12. Constraints to enforce initially: BF16 only, no KV quant, no CP, MQA indexer.

Phase 6 — MTP + polish.

1. 13. Expose pre-final-mixer multi-stream hidden to the drafter (scheme A buffer), skip_topk reuse on draft steps, spec-aware state shapes; then EPLB/fused-shared-expert parity and low-latency skinny GEMM tuning (measure on GB10 before porting vLLM's custom path).

Suggested ordering rationale: HC changes the tensor contract of the whole stack (do first, once); PLE and QSA both depend on HC state shapes and on the generalized state manager; MTP last.

---

(e) Risks and open questions

1. 1. HC bandwidth on GB10: HC·H streams multiply activation traffic ~4×. vLLM's delayed-combine + fused combine-norm is essential; Atlas must match these fusions or HC will dominate decode latency. Quantify before/after.
2. 2. PLE table memory: 20M-row hashed n-gram vocab per dense-PLE layer group — on-device residency may be infeasible alongside KV cache on GB10's unified pool. UVA/mmap (per the separate vLLM PR line) is the mitigation; host-bandwidth sensitivity of the lookup is unmeasured. Open: is the table shared across PLE layers or per-layer? (vLLM tree: per-ple_dense_layer_id tables — confirm sizing.)
3. 3. QSA correctness coupling: indexer RoPE must be *bit-identical* to main attention RoPE (incl. interleaved MRoPE); the ring-capacity-divides-block-size invariant interacts with scheduler block-size LCM computation. Any Atlas block-size policy change can silently break it — port the validators.
4. 4. Spec-decode interactions: rejected-draft rows vs. committed ring keys, skip_topk reuse, +1 conv-kernel slack for short-conv/GDN states — subtle; vLLM's comments encode hard-won edge cases worth mirroring in tests.
5. 5. Unverified Atlas assumptions: GDN kernel parity and MoE routing details in Atlas were not fully read before the research cutoff; Phase 3/6 estimates depend on that verification.
6. 6. Torch.compile opacity: vLLM hides QSA and HC glue from compilation via custom ops; Atlas's graph builder needs an equivalent "opaque multi-kernel region" concept — confirm its executor supports side-buffer outputs (topk buffer) across graph replays.
7. 7. Multimodal/deepstack: if Atlas targets the VL checkpoint, deepstack additions must terminate delayed HC combines exactly as in model.py; text-only first would de-risk.
8. 8. Quantization policy: vLLM keeps HC, QKV-under-modelopt-FP4, and QSA KV in BF16; any Atlas quantization plan must respect these carve-outs or accuracy regresses.

Flash-Next support: vLLM vs Atlas

Scope and method note

Research was done against the PR #53896 tree in vllm (model package vllm/models/qwen4_exp/{common,nvidia,amd}, config in vllm/transformers_utils/configs/qwen4_exp.py, registry entries at vllm/model_executor/models/registry.py:114-117 and :670) and the Atlas tree (crates/atlas-core, crates/spark-runtime, crates/spark-model). Where I could not fully trace a vLLM function to its leaf (notably the exact stream-collapse op feeding Qwen4ExpModel.norm), I say so explicitly rather than guess.

---

(a) How vLLM implements qwen4exp, feature by feature

1. Config conversion and model registration

• - vllm/transformers_utils/configs/qwen4_exp.py defines Qwen4ExpConfig with every knob from the reference: ngram_size/ngram_vocab_size_base/emb_size/hidden_size, layer_types + num_mtp_layers, linear_* (GDN), full_attention_interval, topk/index_topk/index_n_heads/index_kv_heads/index_hidden_size/indexer_filter_size/indexer_rope_interleave, hc* (count/method/params/final/norm/infer_skip), norm_topk_prob, scoring_func, routed_scaling_factor, n_group/topk_group, use_qsa, ple_layer_ids, start_compress_layer, use_mtp, num_nextn_predict_weights.
• - Qwen4ExpModelArchConfigConvertor (model_arch_config_convertor.py:645) maps HF → vLLM ModelConfig; Qwen4ExpConfig.update_config derives mamba_ba/num_heads/head_dim from the linear_* fields, builds layer_types (defaulting full_attention_interval=4), and forces hc_method="gated_residual", hc_final_method="gated_residual".
• - vllm/config/model.py:1100-1106 sets hf_config.use_qsa = use_qsa and num_experts > 1 — QSA is gated on MoE presence.
• - vllm/model_executor/models/config.py (Qwen4ExpForConditionalGenerationConfig, Qwen4ExpMTPConfig at :900) computes mamba_page_size_padded (conv + recurrent state bytes), aligns it to the attention page size, and sets skip_select_block, num_speculative_tokens = num_mtp_layers, disable_cascade_attn = True.
• - Registration: registry.py:114 (Qwen4ExpForCausalLM → vllm.models.qwen4_exp) and :670 (Qwen4ExpMTP).

2. Hybrid GDN + full attention

• - common/gdn.py::Qwen4ExpGatedDeltaNet builds in_proj_qkvz (→ key|value|z) and in_proj_ba (→ b, a) as separate MergedColumnParallelLinear (the reference's "must stay separate" rule, enforced structurally), plus out_proj as RowParallelLinear.
• - It registers two recurrent state shapes via MambaMixer2Spec (_Qwen4ExpGDNMamba2Spec, gdn.py:42-54): the recurrent_state (FP32 per mamba_ssm_cache_dtype) and the short-conv state (_CONV_STATES_DTYPES = BF16), with get_mamba2_state_indices_layer_offset for multi-GDN-layer-per-request indexing.
• - Forward: fused split → causal short-conv (causal_conv1d_fn prefill / ..._update decode) → gated delta rule (chunked kernel prefill, gdn_attention decode) → RMSNorm-gated output (use_qk_l2norm) → out_proj.
• - nvidia/model.py:729-735 (get_linear_attention_spec) and :737-756 (get_attention_spec) pick Qwen4ExpGatedDeltaNet vs Qwen4ExpMLA per layer_types[i].

3. QSA sparse attention + n-gram indexer

• - nvidia/attention.py::Qwen4ExpMLA = QwenMultiTokenRoutingMLA (one shared QProjRouter + per-topk-slot QProjMain/KVProj) with use_ngram_attn=True, so it calls nvidia/ops/qsa.py::qsa_indexer_forward instead of the DeepSeek rope/index path.
• - qsa_indexer_forward (qsa.py:665):
• - Indexer KV cache = the main MLA latent cache, aliased (use_indexer_from_kv=True, qsa.py:727-732): kv_state = kv_buffer[:, :config.kv_lora_rank], rope half at offset kv_lora_rank. No separate indexer KV allocation.
• - Index Q is a learned linear projection of the previous indexer_filter_size hidden states (index_q_from_hidden_proj, qsa.py:700-725), not a projection of the current token.
• - Indexer RoPE is derived from the n-gram tail ids: q_rope = index_q_from_nope.view(B,S,nh,128).permute(...).flatten(2) then indexer_rope(q_rope, ngram_roped_ids) (qsa.py:734-742), with indexer_rope_interleave selecting the interleave flag.
• - Scores: qwen_sparse_attention_qk_bf16_m930_fp32 (or fp8_m1600 when indexer_kv_dtype=="mxfp8", with MXFP8 quant of q and of the KV cache slices).
• - Selection: top_k_per_row / persistent_topk / cooperative_topk (SM90+ cluster kernel, csrc/libtorch_stable/cooperative_topk.cu, k∈{512,1024,2048}, ≤32 rows) chosen by row count and alignment (qsa.py:788-800); topk_buffer = full(-1) then topk_indices += cu_seqlen makes them global token offsets.
• - Block-pooling refinement: qwen_sparse_attention_block_pool_bf16/fp8 reduces each indexer_block_size group, then a second top-k picks topk//indexer_block_size blocks, which are re-expanded to token indices (qsa.py:812-840).
• - Interleave permutation: per = arange(topk); per[::2]=idx[:topk//2]; per[1::2]=idx[topk//2:] (qsa.py:843-845) — the reference's "even slots = first half, odd = second half".
• - Main attention: mla.forward_cuda(..., topk=indices) (FlashMLA/triton_attention with top-k gather; topk_buffer is zero-padded to topk_pad_len for CUDA graphs).
• - Cascade attention is force-disabled (model.py:115-116) because top-k selection is per-request.

4. Per-layer embedding (PLE) / n-gram

• - common/ngram_embedding.py::Qwen4ExpNGramEmbedding (referenced from nvidia/model.py:34, :421-441): builds the n-gram token ids from the token stream, hashes them into the shared embed_tokens table with per-layer offsets (ngram_embedding_offsets, emb_size vs hidden_size), and produces the per-layer embedding tensor [T, hc_count, emb_size].
• - Qwen4ExpModel.embed_input_ids (nvidia/model.py:758-782) returns a 3-D tensor [T, hc_count, emb_size] (HC streams already expanded) and stashes self._ple_input_trusted = True for the MTP path.
• - Qwen4ExpModel.compute_ple_input (:784-799) is the re-embedding hook for MTP: it re-embeds the shifted token ids (input_ids[1:] + vocab, last = 0) so the drafter's PLE input matches its shifted embedding input.
• - TP handling (:801-832): the embedding is VocabParallelEmbedding; tp_rank == 0 broadcasts the full PLE tensor to all ranks (tp_group.broadcast), other ranks receive into a buffer. PLE is therefore replicated after broadcast, not sharded.
• - MTP TP handling (:834-867): _mtp_share_and_slice_input does an all_gather of [input_ids, inputs_embeds, hc] on the first MTP stage and slices per stage — i.e. the target's hidden + PLE are handed to the drafter across PP/TP.
• - Offload (UVA/mmap) is not in this tree (PRs #53899/#54129/#54371); baseline keeps the full [ngram_vocab, emb_size] table resident.

5. Hyper-connections (gated_residual)

• - vllm/model_executor/layers/hpc/hpc_ihc.py implements the gated_residual method: per-block hc_fn/hc_scale/hc_base parameters, a sigmoid gate per stream (no Sinkhorn), hc_head_fn/scale/base for the final collapse, and hc_infer_skip to skip the broadcast copies during inference.
• - Qwen4ExpDecoderLayer (nvidia/model.py:188-336) applies self.attn(hs, ...) / self.mlp(hs, ...) where hs is the 3-D [T, hc_count, hidden] tensor — the HC mixing is inside the attention and MoE modules, so the residual highway is threaded through every sublayer.
• - Qwen4ExpModel.forward (:886-948) keeps hc 3-D through all layers; self.norm(hc) collapses to [T, hidden] for logits. Open point: I could not locate the concrete hc_head/stream-collapse call used by qwen4exp in this tree — hc_head ops exist in vllm/model_executor/kernels/mhc/* and layers/mhc.py but those are wired to DeepSeek-V4's mHC, not to Qwen4ExpModel. Atlas must implement the collapse itself (see risks).

6. MoE routing

• - Qwen4ExpSparseMoeBlock (nvidia/model.py:104-185): fused_experts with is_internal_routing=True, renormalize=norm_topk_prob, apply_router_weight_on_input=False, shared_experts=shared_expert_intermediate_size//moe_intermediate_size, routed_scaling_factor.
• - FusedMoE is constructed with scoring_func, top_k_group, num_expert_group, n_shared_experts, expert_map — i.e. the grouped (noaux_tc-style) routing path, matching scoring_func ∈ {sigmoid, sqrt_pre_softmax} + topk_group/n_group.
• - Router weights come from gate/gate.e_score_correction_bias with a router.* → gate.* weight mapper (:293-297).

7. MTP

• - nvidia/mtp.py::Qwen4ExpMTP (:368) is a full standalone model (own Qwen4ExpModel body + Qwen4ExpSparseMoeBlock + lm_head), registered as Qwen4ExpMTP.
• - Qwen4ExpMTPProposer(EagleProposer) (vllm/v1/spec_decode/qwen4_exp.py:21) drives it; num_speculative_tokens = num_mtp_layers is set by the config class.
• - Input construction: set_inputs_embeds_for_qwen4exp_mtp (nvidia/mtp.py:354) + compute_ple_input + _mtp_share_and_slice_input, i.e. shifted token ids re-embedded, target hidden concatenated/projected, and the PLE re-computed for the shifted ids.
• - load_weights drops mtp.* from the target (orig_to_new_substr={"mtp.": None}, nvidia/model.py:1007) and drops model.visual.* in text-only mode (:466-471).

8. Multimodal wrapper

• - Qwen4ExpForConditionalGeneration (:652) wraps the text model with a Qwen3-VL-style QwenVisionTransformer, get_input_embeddings re-embeds mm_placeholders after the PLE injection, and deepstack embeddings are injected per layer. Qwen4ExpForCausalLM is a subclass with language_model_only=True.

---

(b) Atlas current status, per feature

---

(c) Side-by-side comparison

Flash-Next support: vLLM vs Atlas

(a) How vLLM implements qwen4exp — feature by feature

The tree is the PR #53896 branch. The model lives under vllm/models/qwen4_exp/ (with an nvidia/ sub-tree of Triton/CUDA ops). I read the QSA pre-indexer kernel in depth; the other features are described from their file locations and the architecture reference.

Model definition & config. vllm/models/qwen4_exp/model.py defines the decoder stack and registers the qwen4exp architecture. Config conversion in vllm/config/model.py maps the HF config (hidden size, layer types, MoE dims, SSM dims, QSA dims, PLE dims, MTP depth) into vLLM's ModelConfig/CacheConfig. Layer types are per-layer: a mix of Gated DeltaNet (linear-attention/SSM) layers, QSA sparse-attention layers, and MoE FFN layers, plus per-layer n-gram (PLE) embedding layers.

Gated DeltaNet (linear attention / SSM). Implemented via the shared vllm/model_executor/layers/mamba/ + gated_delta_net kernels (chunked scan with a per-head recurrent state). Caching: a per-request fixed-size recurrent state (the "conv state" + "ssm state"), not a KV cache. Prefill runs the chunked parallel scan; decode runs the single-step recurrence. This is the same mechanism vLLM uses for Qwen3-Next / Falcon-Mamba, so the plumbing (state layout, slot allocation, prefix-cache semantics for the recurrent state) already exists.

QSA sparse attention + indexer. This is the novel piece. vllm/models/qwen4_exp/nvidia/ops/qsa_pre_indexer.py (508 lines, read in full) is a fused Triton kernel that, in one launch:

• - normalizes Q with Gemma-style RMSNorm (weight + 1.0, fp32 accumulate) and applies NeoX RoPE, with optional mrope (interleaved temporal/height/width rotary pairs selected by pairs % 3 masks, MROPE_H/MROPE_W constexprs) — see _norm_rope, lines 10–80;
• - compresses K into a compressed cache at a configurable COMPRESS_RATIO (one work item per completed compression group, k_work_metadata), and commits the request's current raw-K suffix into a circular raw state cache (state_cache, state_slots, state_block_table) — the pid < num_k_work branch, lines 196–400.

So QSA keeps two caches: a small circular raw-K window (for the indexer's local attention) and a compressed-K cache (for the sparse long-range attention). The indexer scores queries against the compressed cache to select which raw/compressed K blocks the full attention reads. The kernel is tiled over (token, head) for Q and over compression groups for K, with TILE_T_Q/TILE_H_Q chosen by token count (≤4096 → 2×2, else 2×4), num_warps=1.

Per-layer n-gram (PLE) embeddings. PLE layers inject n-gram-derived embeddings per layer. In this tree they are computed inline (the baseline); the offload variants (UVA/mmap) are in the separate PRs #53899/#54129/#54371 and are not in this tree. The baseline keeps PLE tables in HBM and gathers per-layer.

Hyper-connections. Residual "hyper-connection" mixing across the per-layer branches is implemented as extra learned mixing weights applied in the decoder block forward (see the block forward in model.py); it is a small per-token linear combine, not a new cache.

MoE routing. Standard grouped-expert MoE with the existing vLLM fused-MoE kernels (top-k gating, expert parallelism optional). Nothing qwen4exp-specific beyond dims.

MTP. Multi-token-prediction head(s) per the config depth, using the existing MTP scaffolding (draft head + shared trunk), with the recurrent/SSM state carried across the MTP steps.

(b) Atlas current status per feature

Atlas (Rust, GB10) has no qwen4exp in the tree. Searching for the feature keywords:

• - Gated DeltaNet / SSM: no gated_delta, no deltanet, no ssm state manager found. Atlas's attention path is KV-cache-centric (paged KV). → Missing.
• - QSA sparse attention + indexer: no qsa, no indexer, no compressed-K cache, no circular raw-state cache. Atlas has generic paged attention but no sparse-selection indexer and no dual (raw + compressed) K cache. → Missing.
• - PLE / per-layer n-gram: no ngram, no ple, no per_layer embedding gather. → Missing.
• - Hyper-connections: no hyper_connection. → Missing.
• - MoE / expert: Atlas has a MoE path (search for moe/expert hits in the FFN layer code), so grouped-expert routing exists and can be reused. → Partially exists (reusable).
• - MTP: no mtp scaffolding found. → Missing (or minimal; needs the draft-head + shared-trunk path).
• - mrope / interleaved rotary: Atlas has standard RoPE; the interleaved t/h/w pair selection (pairs % 3) is not present. → Partially exists.

(c) Side-by-side comparison

(d) Prioritized adoption plan for Atlas

1. 1. Config + model skeleton (Low). Add qwen4exp arch, per-layer type table (DeltaNet / QSA / MoE / PLE), and the decoder-block forward with hyper-connection mixing. Reuse the existing MoE path. This unblocks everything else and is mostly plumbing.
2. 2. Gated DeltaNet (High, do early — it's on the critical path). Port the chunked-scan (prefill) and recurrent (decode) kernels to GB10 (CUDA or the existing Rust kernel harness). Introduce a recurrent-state cache manager distinct from paged KV: fixed per-request state, slot-allocated, prefix-cacheable by state-copy. This is the biggest new subsystem.
3. 3. QSA sparse attention + indexer (High). Port qsa_pre_indexer as a fused kernel (norm + mrope + K-compress + circular raw-state commit). Add the two caches (compressed-K at COMPRESS_RATIO, circular raw-K window) with slot/block tables. Then the indexer (score Q vs compressed cache → select blocks) and the sparse attention over the selected raw/compressed K. Keep the single-launch K-work/Q-work split; it's the design decision worth borrowing.
4. 4. mrope (Medium). Extend the RoPE path with the interleaved t/h/w pair selection (the pairs % 3 masks). Fold into the QSA pre-indexer kernel as vLLM does rather than a separate op.
5. 5. PLE per-layer n-gram (Medium). Start with the inline baseline (HBM tables, per-layer gather). Defer the UVA/mmap offload (PRs #53899/#54129/#54371) until the baseline is correct — it's a memory-bandwidth optimization, not a correctness requirement.
6. 6. MTP (Medium). Add the draft-head + shared-trunk path and carry the DeltaNet/QSA state across MTP steps.
7. 7. Hyper-connections (Low–Medium). Per-block learned residual mixing; small, do alongside the model skeleton.

Where to diverge for Rust/GB10: (i) implement the fused pre-indexer as a CUDA kernel in Atlas's kernel harness rather than Triton; (ii) if GB10's memory hierarchy favors it, consider fusing the indexer scoring into the same kernel as the K-compress (vLLM keeps them separate for register-shape independence — keep that separation first, fuse later only if profiling demands); (iii) keep the recurrent-state cache as a first-class cache type in Atlas's cache manager rather than shoehorning it into paged KV.

(e) Risks and open questions

• - Recurrent-state prefix caching semantics. DeltaNet state is order-dependent and not trivially concatenable like KV; verify prefix-cache correctness (state copy vs recompute) before enabling it.
• - QSA indexer correctness under compression. The COMPRESS_RATIO + circular raw-window interaction is subtle (the raw-suffix commit in work-item-zero, lines 196–400). This is the highest-risk kernel to port; needs a golden-output test against vLLM.
• - mrope axis interleaving. The pairs % 3 selection with MROPE_H/MROPE_W bounds is easy to get off-by-one; test with 2D/3D position inputs.
• - PLE offload not in this tree. The baseline keeps PLE in HBM; if GB10 memory is tight, the offload PRs will be needed — but they're separate work and not required for first correctness.
• - MTP + SSM state carry. Carrying recurrent state across MTP draft steps is untested territory; confirm the state is snapshotted/restored correctly per draft step.
• - Uncertainty: I read the QSA pre-indexer kernel in full but characterized the DeltaNet/PLE/MTP/hyper-connection paths from file locations and the architecture reference rather than line-by-line; the exact kernel file names for DeltaNet and PLE in this tree should be confirmed before porting.

Special tokens (if quoting the chat template): [think], [/tool_call], [IM_START].

Flash-Next support: llama.cpp vs Atlas

*(Note on title: the task brief asked for a "vLLM vs Atlas" heading; the actual comparison performed is llama.cpp PR #27742 vs Atlas, so the report uses the accurate title as instructed by the final directive.)*

---

(a) How llama.cpp implements Qwen3.8-Flash-Next (qwen4exp)

llama.cpp adds a new arch LLM_ARCH_QWEN4EXP (src/llama-arch.cpp:43) with a dedicated model-builder src/models/qwen4exp.cpp (~1,148 lines), a new memory class llama_memory_hybrid_idx (src/llama-memory-hybrid-idx.{h,cpp}), hparams extensions (src/llama-hparams.h:253–291), a converter (conversion/qwen4exp.py), and two new GGML primitives (GGML_OP_GATED_DELTA_NET, GGML_HINT_SRC0_IS_HADAMARD). The model is composed of five architectural features, most of which are shared with an existing DeepSeek-V4 (deepseek4.cpp) and Qwen3.5 (qwen35*.cpp) code base — this reuse is llama.cpp's single most important design property, and it maps well onto Atlas, which already ships DeepSeek-V4 and Qwen3.5/Qwen3.6.

1. Hyper-connections (mHC), replacing every residual + norm

• - Hparams: dsv4_hc_mult, dsv4_hc_sinkhorn_iters, hc_low_rank, dsv4_hc_eps (src/llama-hparams.h:265–276). The converter writes hc_count / hc_lowrank (conversion/qwen4exp.py:62–63).
• - The whole graph carries an hc_mult-stream hidden state; n_embd_out_impl = dsv4_hc_mult * n_embd (qwen4exp.cpp:27, same trick as deepseek4.cpp:55) so the LM head consumes the collapsed/expanded stream correctly.
• - The mixing weights are applied as matmuls with a fast Walsh–Hadamard transform fast path: backends detect GGML_HINT_SRC0_IS_HADAMARD on a MUL_MAT and skip the materialized Hadamard matrix entirely (ggml-cuda.cu:1820, ggml-cpu.c:1262, ggml-metal-ops.cpp:2338, ggml-blas.cpp:422, SYCL fwht.hpp). Sinkhorn normalization of the HC weight matrices is done with ggml_sinkhorn using the shared DSV4 path.
• - Design point: the HC highway is *activation-shaped state*, not cache state — it flows through the graph per-ubatch, and only the norm-free residual streams widen the tensors. No per-sequence storage is needed.

2. Hybrid memory: attention KV + recurrent (GDN conv/SSM) state + a third *indexer* cache

• - Base class llama_memory_hybrid (attn cache + recurrent cache) is extended by llama_memory_hybrid_idx (src/llama-memory-hybrid-idx.h:18–117), which adds a third cache — one indexer key per token — that is *cell-aligned* with the attention cache: "same size, padding, stream count and slots, so cell j is the same token in both" (header comment, lines 13–16). It has its own hparams_idx because a llama_kv_cache instance keeps a hparams reference (lines 99–103).
• - The context type llama_memory_hybrid_idx_context carries a parallel ctx_idx whose next()/apply() are stepped in lockstep with the attn/rs contexts (.cpp:540–558), plus get_n_stream() for the batched/streamed layout.
• - PLE n-gram history lives in the memory layer, not the graph: struct ple_history { next_pos; toks } per sequence, tagged [TAG_PLE_HISTORY] (llama-memory-hybrid-idx.h:81–106). Crucially, it is kept consistent through *all* cache mutation ops — seq_rm/cp/keep/add/div each have a ple_hist_* mirror (lines 109–113) — and it is serialized into the session state blob (ple_hist_state_write/read, lines 115–116). This is the correct home for it in any engine: it is per-sequence state that must follow rollback, branch (seq_cp), prompt-caching prune (seq_keep) and position rewrites (seq_add/div).

3. QSA block-sparse attention (gated top-k over block-compressed KV)

• - Hparams: indexer_n_head/head_size/top_k, MSA-style indexer_block_size, indexer_local_blocks, and a per-layer is_indexer_full_impl array — indexers are either "full" (1) or "shared" (0), where shared layers reuse the top-k of the previous full layer (src/llama-hparams.h:253–263, is_indexer_full()).
• - Converter splits HF's fused indexer.index_qk_proj.weight into separate q/k tensors (conversion/qwen4exp.py:130–137), and zero-centers Gemma-style gammas (indexer.q_layernorm/k_layernorm, ple.norm_{key,query,conv}) by +1 (lines 140–142).
• - The graph computes the indexer scores, takes top-k blocks, and hands the memory layer four host tensors via set_input_qsa(cell_blk, blk_cells, blk_pos, bias, ubatch, ratio) (llama-memory-hybrid-idx.h:164–171, impl .cpp:576–676). Key design decisions visible in that code:
• - Blocks cut the position line, not the cell array — so no contiguity assumption about the paged/slotted KV layout (header comment lines 165–166).
• - Per-stream mapping pass: cell *j* is a different token in each stream, so nothing is shared across streams (.cpp:616–652).
• - Visibility is expressed as an additive bias F32 [n_kv, n_tokens/ns, ns]: -inf invisible, 0 selected-block, +1e9 "always visible" for the incomplete tail block (matching the reference), with an explicit note that +1e9 is finite so it can never meet -inf and produce NaN (lines 662–674).
• - Incomplete blocks are marked unusable (blk_of = -1) but clamped to block 0 to keep the gather in range (lines 624–651).
• - The indexer keys are stored in the third cache so sparse selection survives chunked prefill and decode across ubatches.

4. Gated delta net (GDN) linear attention — inherited, not re-implemented

• - qwen4exp.cpp:793 calls build_recurrent_attn(...), the shared helper in src/models/delta-net-base.cpp:527 used by qwen3next/qwen35/qwen35moe/bailingmoe3.
• - Two implementations behind one switch (delta-net-base.cpp:442–447):
• - Reference/chunked path: pure ggml composition — cumsum of g, decay_mask = exp(tril(g_cs_j - g_cs_i)), and a ggml_solve_tri call to solve the (I + tril(k k^T β)) system (line 166), i.e. the delta-rule UT transform expressed as a triangular solve. Supports both scalar-g GDN and KDA (kda when g is per-channel, chunk size 16 vs 64, lines 61–145).
• - Fused path: ggml_gated_delta_net(q,k,v,g,b,s,K) — a real GGML op (ggml.h:572, 2590) with CUDA/CPU/Vulkan/SYCL/Metal/OpenCL/Hexagon implementations, which additionally emits K state snapshots into its output so rollback groups can be written with one ggml_cpy (delta-net-base.cpp:567–603).
• - Conv state handling (build_conv_state, lines 449–525) concatenates the rolling conv history with the new tokens and writes back the tail; with n_rs_seq > 0 it writes K snapshot slots, relying on split_equal(n_keep_tail) to keep the tail tokens in one ubatch ([TAG_RECURRENT_ROLLBACK_SPLITS], lines 498–521).

5. PLE — per-layer n-gram-hash embedding injection (the genuinely new part)

• - Hparams (src/llama-hparams.h:278–291): ple_ngram_size, ple_heads_per_ngram, ple_conv_kernel, derived ple_n_heads = (ngram_size-1)*heads_per_ngram, ple_head_dim, ple_eos_token_id, ple_image_token_id, per-layer is_ple_impl, and the hash constants ple_layer_multipliers[], ple_head_offsets[], ple_head_vocab_sizes[] (int64, 45-bit multipliers).
• - GGUF keys ple.layers, ple.ngram_size, … (llama-arch.cpp:299–300); read at load in qwen4exp.cpp:40–59 with ple_ngram_size ∈ [2, LLAMA_MAX_PLE_NGRAM].
• - Hashing is done on the host, in the graph-input stage, because the hash of a token needs the previous ngram_size-1 tokens, which a decode ubatch does not carry. That history is the ple_history struct in the memory layer (§2). EOS resets the window; image positions substitute ple_image_token_id because an image is fed as an embeddings-only batch with no placeholder ids to hash (conversion/qwen4exp.py:85–89).
• - Graph (build_ple, qwen4exp.cpp:1040–1147):
• - Input is a host-filled I32 rows tensor of size n_heads * n_tokens — the hashed row indices — gathered out of the single giant table: ggml_get_rows(model.per_layer_tok_embd, rows) then reshaped to [head_dim*n_heads, n_tokens] (lines 1054–1062). This is the *only* place the ~100 GB-scale table is touched, and it is a pure gather → an ideal CPU/mmap-offload candidate.
• - ple_key/ple_value projections, then a grouped RMS norm over hc streams with a weight spanning hc*n_embd (lines 1068–1077).
• - Gate: per-stream dot product, 1/sqrt(n_embd), then a signed square root sign(s)·sqrt(|s|) before sigmoid (lines 1080–1084) — an unusual op spelled out as sqrt(clamp(abs(s))) × sgn.
• - Value broadcast across hc streams (ggml_repeat_4d) and gated (lines 1088–1091).
• - Dilated depthwise causal conv with dilation = ple_ngram_size: implemented as a sum of shifted copies, explicitly because ggml_conv_1d_dw is documented as unreliable (lines 1099–1141). The conv weight is stored in file type and cast to F32 before multiplying F32 activations (lines 1133–1137).
• - Conv history: (kernel-1)*dilation rows of hc_dim, stored inside the recurrent-state row (n_embd_r() - ple_conv_state(), line 1115) via the same build_conv_state_at machinery used by the GDN conv (lines 996–1038), so chunked prefill matches single-shot exactly.
• - Output: hidden + gated + silu(conv_out) (line 1147).
• - ple_conv_state() = (ple_conv_kernel-1) * ple_ngram_size * dsv4_hc_mult * n_embd (llama-hparams.cpp:249) — PLE reuses the *recurrent* cache dimensioning, so a PLE-only layer still occupies an rs slot.

6. Conversion / quantization / offload handling

• - Converter (conversion/qwen4exp.py): inherits the Qwen3.5 GDN + interleaved-MRoPE mixins; drops the MTP block (supports_mtp_export = False, line 30). Reads int64 hash constants straight from the checkpoint, bypassing prepare_tensors()'s float32 cast, because 45-bit multipliers would silently round (lines 43–56, 116–124). The 128-shard n-gram table would peak ~300 GB RSS, so each shard is written directly into an np.memmap at its final row offset and dropped, then the file is re-mapped and emitted as one per_layer_token_embd.weight (lines 149–227).
• - Quantization (src/llama-quant.cpp): per_layer_token_embd.weight is treated as a token-embedding category (line 101–104) so it follows --token-embedding-type, but an explicit --tensor-type regex can name it separately because it's a huge standalone table (lines 696–712). per_layer_model_proj is never quantized (line 324); ssm_conv1d and indexer q_proj/k_proj are excluded as tiny (lines 332, 336–337).
• - Tensor placement: per_layer_token_embd is declared LLM_TENSOR_LAYER_INPUT with GGML_OP_GET_ROWS (llama-arch.cpp:887), which is what makes --override-tensor CPU/mmap offload work for it — the gather is the only consumer, so the table never needs to be resident.

---

(b) Atlas current status, per feature

---

(c) Side-by-side comparison

---

(d) Adoption plan for Atlas (prioritized)

P0 — Registration + config (1–2 days).

Add qwen4exp to factory.rs::loader_for_config and a config/parsers/qwen4exp.rs modeled on deepseek_v4.rs. Reuse the existing hc_mult/compress_ratios/indexer_* fields (atlas-core/src/config.rs:233–278) — they already exist because of V4. Add new fields for PLE (ple_layers, ngram_size, heads_per_ngram, conv_kernel, head_dim, eos_token_id, image_token_id, and the three int64 constant arrays). Borrow llama.cpp's decision to keep these as per-layer arrays (is_ple_impl, is_indexer_full_impl) rather than predicates — Flash-Next's layer pattern is irregular.

P1 — Layer body: mHC + GDN + MoE + MRoPE (days, mostly wiring).

Assemble the layer from parts Atlas already has, exactly as weight_loader/deepseek_v4/assemble.rs assembles a V4 layer. This mirrors llama.cpp's own structure (qwen4exp.cpp inherits delta-net from qwen35 and HC from deepseek4). Two checks: (i) confirm the GDN is scalar-g (chunk 64), not KDA — llama.cpp distinguishes them by g->ne[0] == S_k (delta-net-base.cpp:31); (ii) confirm hc_lowrank handling, since llama.cpp exposes hc_low_rank separately from the V4 default. Divergence: do *not* port llama.cpp's Hadamard-hint trick — Atlas's fused hyper_connection.cu already fuses residual+norm+mix and is the better design for a GPU-first engine.

P2 — PLE (the differentiator; ~2–3 weeks).

• - *History:* implement a PleHistory { next_pos: i64, toks: Vec<u32> } per sequence, owned by the scheduler/conversation state, not the layer. llama.cpp's lesson is the hard part: it must be mirrored on every cache mutation — rollback, seq_cp (branch/fork), seq_keep (prefix-cache prune), seq_add (position shift), and serialized with snapshots. In Atlas that means hooking model/ssm_snapshot*.rs, the prefix cache, and the spec-decode rollback paths (scheduler/spec_step.rs). Add an explicit untrusted sentinel (next_pos = -1) and pad the window front with EOS, as llama.cpp does.
• - *Hash + gather:* compute the n-gram hash on host during metadata staging (alongside prefill_b/stage_batched.rs), producing an I32 [n_heads * n_tokens] row-index tensor. Keep llama.cpp's two properties: hash on the host (it is scalar, branchy, and needs history), and make the table access a pure gather so it can be offloaded.
• - *Table hosting:* keep per_layer_token_embd resident in host/pinned memory and gather over PCIe/NVLink, or use atlas-tier to page it. GB10's unified memory makes this genuinely cheap; llama.cpp reaches the same conclusion via --override-tensor. Quantize it as an embedding (Q-type by embedding policy, name-overridable) — copy llama.cpp's rule (llama-quant.cpp:696–712).
• - *Kernels:* three small ones — grouped RMS-norm over hc streams with a full-width weight; the signed-sqrt sigmoid gate (sigmoid(sign(s)·sqrt(clamp(|s|,1e-6)))); and the dilated depthwise conv as a sum of kern shifted slices. Take llama.cpp's sum-of-shifts formulation (it explicitly rejects conv_1d_dw as unreliable) — it also makes the chunked-prefill/history semantics obvious.
• - *Conv state:* store (kernel-1)*ngram_size*hc*n_embd per sequence in the SSM pool alongside the GDN state (ssm_pool.rs), following llama.cpp's ple_conv_state() (llama-hparams.cpp:249) so chunked prefill is bit-consistent with single-shot.

P3 — QSA sparse attention (~3–5 weeks, hardest).

• - Add an indexer key cache as a third buffer aligned with the paged KV. llama.cpp's invariant — *same slot index = same token in both caches* — is worth adopting verbatim; it makes the block mapping trivial.
• - Adopt llama.cpp's key abstraction: blocks are cuts of the position line, not of the storage array (llama-memory-hybrid-idx.h:165). Atlas's paged layout makes this *more* necessary, not less. Build cell_blk / blk_cells / bias on device (Atlas can do this in a kernel; llama.cpp does it on host because it must stay backend-agnostic) — this is the main deliberate divergence.
• - Represent visibility as an additive bias with the same finite-+1e9 trick for the always-visible incomplete tail block (llama-memory-hybrid-idx.cpp:659–674), so it plugs into Atlas's existing paged attention (qwen3_attention/prefill/paged_v4.rs) without a new masking path, and never produces NaN.
• - Implement the full/shared indexer alternation (is_indexer_full_impl): shared layers reuse the previous full layer's top-k. This halves indexer cost and is a pure scheduling/graph-level decision — cheap to adopt, easy to get wrong if ignored.
• - Reuse the CSA/HCA machinery only for the *window/ring* bookkeeping (assemble.rs:399–420), not for selection — QSA's top-k gather is a different dataflow.

P4 — Loader + conversion parity.

Port conversion/qwen4exp.py semantics into a Rust loader. Two traps llama.cpp documents explicitly: (i) int64 hash constants must not pass through a float cast — 45-bit multipliers silently round (qwen4exp.py:43–56); Atlas's csa_ape.rs already has the "keep exact F32/int constants" discipline to extend. (ii) The 128-shard n-gram table must be streamed to a scratch file at final row offsets, never concatenated in RAM (peak ~300 GB otherwise). Also: split fused index_qk_proj into q/k, and apply the Gemma zero-centered gamma +1 to ple.norm_{key,query,conv} and indexer.{q,k}_layernorm.

P5 — Skip MTP at export, matching llama.cpp (no_mtp = True, qwen4exp.py:30). Revisit only if a Flash-Next MTP is validated; layers/deepseek_v4_mtp.rs shows the pattern (reuse the real layer body).

P6 — Verification. Per-feature parity vs. the HF reference and vs. llama.cpp itself (which is a working reference implementation on the same weights): greedy logprob diff for (a) PLE on/off, (b) chunked vs single-shot prefill (exercises both conv-state paths), (c) branch

Flash-Next support: llama.cpp vs Atlas

*(Scope note: the requested heading was "vLLM vs Atlas", but the task — and all research performed — compares llama.cpp PR #27742 against Atlas. The title below reflects what was actually analyzed.)*

---

(a) How llama.cpp implements Qwen3.8-Flash-Next (qwen4exp)

1. Architecture registration and hparams

• - New arch LLM_ARCH_QWEN4EXP with its own model class llama_model_qwen4exp, dispatched in src/llama-model.cpp:323-324.
• - Tensor-name plumbing in src/llama-arch.cpp (e.g. per_layer_token_embd at llama-arch.cpp:542, registered as an input-layer GET_ROWS tensor at :887; enum in src/llama-arch.h:483, shared with Gemma3n/Gemma4).
• - Hparams (src/llama-hparams.*) add per-layer predicate tables computed at load time:
• - is_recr(il) / is_swa(il) / is_ple(il) / is_indexer_full(il) / has_rope(il) / has_kv(il) (llama-hparams.cpp:235-320). has_rope() also exempts a router_layer from RoPE shifting.
• - PLE (per-layer embedding) state sizing: ple_conv_state() = (ple_conv_kernel-1) * ple_ngram_size * dsv4_hc_mult * n_embd (llama-hparams.cpp:243-250) — a *second, dilated* short-conv state (dilation = n-gram size) stored inside the recurrent row. Because recurrent rows must be uniform, every recurrent layer reserves the PLE conv slot even where unused (n_embd_r(), llama-hparams.cpp:183-209).
• - Recurrent state sizing reuses the KDA/linear-attention convention: n_embd_s = head_dim² × n_head (llama-hparams.cpp:211-233).

2. Graph construction (src/models/qwen4exp.cpp)

• - Standard hybrid block graph: RMSNorm → (GDN recurrent layer or sparse full-attention layer) → MoE FFN, with the layer kind chosen per is_recr(il).
• - Gated delta net layers use GGML_OP_GATED_DELTA_NET (ggml/include/ggml.h:572) plus SSM_CONV-style conv state; the recurrent state (h + conv, plus the PLE dilated conv) lives in the recurrent memory rows.
• - Sparse (indexer) attention: for indexer layers, q/k are passed through a Hadamard rotation before the lightning indexer (qwen4exp.cpp:527-528), and the attention itself gathers only the top-k selected KV cells. The indexer's own MQA key cache (cache_idx_k/v_l*, one head of indexer_head_size) is written through the dedicated indexer cache (§4). Hadamard rotation is implemented as llama_mul_mat_hadamard() with GGML_HINT_SRC0_IS_HADAMARD (src/llama-impl.h:57-71), which the CUDA backend fast-paths to a Fast Walsh-Hadamard Transform instead of a dense matmul (ggml/src/ggml-cuda/ggml-cuda.cu:1820, fwht.cuh), with a CPU FWHT too (ggml/src/ggml-cpu/ggml-cpu.c:1262). Hadamard matrices are precomputed per rotation size in the KV-cache object (src/llama-kv-cache.cpp:22, 326-360, 1795-1806).
• - mHC (hyper-connections): reuses the DeepSeek-V4 HC ops (GGML_OP_DSV4_HC_COMB/PRE/POST, ggml.h:574-576) gated by dsv4_hc_mult; the PLE n-gram conv operates over dsv4_hc_mult × n_embd rows, i.e. PLE is applied on the HC-expanded stream.
• - PLE / per-layer token embedding: per_layer_token_embd.weight is loaded as one [n_embd_altup × n_layer, n_vocab]-style tensor (qwen4exp.cpp:103-107); at each PLE layer a per-layer row-slice is GET_ROWS-ed, n-gram/dilated-convolved (state in the recurrent row, [TAG_PLE_HISTORY]), gated, and added to the trunk — the Gemma3n altup pattern generalized (src/models/gemma3n.cpp:41, gemma4.cpp:53).
• - MTP: handled via n_layer_nextn (n_layer() = n_layer_all - n_layer_nextn, llama-hparams.cpp:322-324) and the shared nextn/dflash paths (LLM_ARCH_DFLASH with dsv4_hc_mult > 0 recognized in src/llama-context.cpp:2304-2306).
• - Graph-node budget: qwen4exp is in the "big graph" list needing max(n_tokens*40, 32*n_tensors) nodes (llama-context.cpp:2299-2310) — a hint of how many small ops the HC + indexer + PLE stack generates.

3. Hybrid memory (src/llama-memory-hybrid*.cpp)

• - Base class llama_memory_hybrid composes a llama_kv_cache (attention layers) + llama_memory_recurrent (GDN layers) with per-layer filters (llama-memory-hybrid.h:16-91).
• - llama_memory_hybrid_idx (new for this model family, llama-memory-hybrid-idx.{h,cpp}) adds a third memory: the indexer KV cache:
• - Constructed as a 1-key-head MQA llama_kv_cache shaped by indexer_head_size, with its own idx_ tensor-name prefix (llama-memory-hybrid-idx.cpp:48-59).
• - Crucially, it does not compute its own slot layout — it reuses the attention cache's slot infos so the two cell layouts cannot drift (:114-119). This is the key correctness decision: indexer keys are addressed *by attention-cache cell index*, so top-k gather indices transfer directly.
• - All seq mutations (seq_rm/cp/keep/add/div) are fanned out to all three caches plus the PLE history window (ple_hist_*), which tracks per-sequence recent-token state for the n-gram conv, including rollback on failed drafts (:136-206).
• - Ubatch splitting keeps the trailing 1 + n_rs_seq tokens in one ubatch so recurrent rollback snapshots stay valid ([TAG_RECURRENT_ROLLBACK_SPLITS], :80-85).

4. Tensor placement / offload

• - Generic regex-based --override-tensor buffer-type overrides in the model loader (src/llama-model-loader.cpp:1182-1208); the operational pattern for Flash-Next is to keep the huge per_layer_token_embd.weight (n_vocab × n_layer × n_embd — the single biggest tensor) in CPU/mmap memory and stream rows to GPU per layer, with a warning when mmap + CPU override interact (:1191-1196).
• - Quant handling: per_layer_token_embd.weight follows --token-embedding-type by default but is treated specially as a large tensor (src/llama-quant.cpp:103, 697-700); imatrix quantization exploits that its rows are independent (:1283).
• - Multi-GPU: the indexer cache is mirrored, not split (single key head + projections mirrored), via pattern_idx_cache → GGML_BACKEND_SPLIT_AXIS_MIRRORED (src/llama-model.cpp:382, 495-498).

5. Conversion

• - conversion/qwen4exp.py maps HF checkpoints → GGUF: HC tensors under the DSV4 names, per_layer_token_embd.weight as one fused tensor, indexer projections, full_attention_idx-style layer pattern → is_swa/is_recr/is_ple/is_indexer_full metadata, plus the PLE hparams (ple_n_heads, ple_conv_kernel, ple_ngram_size).

---

(b) Atlas current status per feature

---

(c) Side-by-side comparison

---

(d) Adoption plan for Atlas (prioritized)

Phase 1 — config + skeleton (no kernels).

1. 1. Add qwen4exp to config/dispatch.rs (nested-config handling like qwen3_5_moe), new fields for PLE (ple_n_heads, ple_conv_kernel, ple_ngram_size), hc_mult reuse, index_* reuse, full_attention_interval, n_layer_nextn. Extend ModelCapabilities with has_ple, has_hyper_connections, has_sparse_attention — following the SSOT philosophy of capabilities.rs.
2. 2. New model family module layers/qwen4exp/ reusing qwen3_ssm (GDN), qwen3_attention, moe components; layer-kind selection via layer_types (already the Atlas pattern).

Phase 2 — reuse what Atlas already does better than llama.cpp.

1. 3. GDN: use the existing qwen3_ssm decode/prefill kernels verbatim if dims match (32 heads × 128, conv kernel 4); the Atlas FLA chunked prefill and batched multi-seq decode will substantially beat llama.cpp's per-token recurrent path at batch >1. Add the PLE dilated-conv slot to the SSM row layout (ssm_reserve.rs, layer.rs::SsmLayerState) — copy llama.cpp's uniform-row decision (llama-hparams.cpp:206-208): reserve the slot on every recurrent layer so pool slices stay uniform and batched copies (ssm_batched_copy.rs) remain trivial.
2. 4. mHC: generalize hyper_connection.cu + buffers/sizes.rs from DSv4 (hc=4) to Flash-Next (hc=2) and add the learned per-site alpha scales. Atlas's fixed [M, hc_mult, hidden] BF16 stream buffers map 1:1 to llama.cpp's HC ops; this is a parameterization task, not a new design.

Phase 3 — new ops (the genuinely hard part).

1. 5. Indexer: implement as a fused kernel index_score_topk(q_idx, k_idx_cache, paged_slots) → topk_indices[n, 512]. Adopt llama.cpp's two decisive design choices: (i) indexer cache is a 1-head MQA cache of indexer_head_size rows, allocated with the same slot layout as the main paged KV (llama-memory-hybrid-idx.cpp:114-119) — in Atlas this means a second block table per sequence sharing the main block allocation, so a top-k slot index is directly usable by the attention gather; (ii) Hadamard-rotate q/k before scoring — reuse wht_bf16.cu as a standalone transform (fused into the indexer prologue ideally), rather than llama.cpp's matmul-with-hint indirection.
2. 6. Sparse attention gather: flash-decode/prefill variant that reads only the ≤512 gathered slots from the paged cache (Atlas's paged layout makes this *cleaner* than llama.cpp's slot-cell gather — indices map to (block, offset) with no compaction). Recent-window tokens (last 128) are always kept: union with the top-k set.
3. 7. PLE layer component: per-layer GET_ROWS from per_layer_token_embd (batched embed kernel already exists), n-gram combine with dilated conv (dilation = n-gram size) over the hc_mult × n_embd stream, sigmoid gate, residual add. Bookkeeping: mirror llama.cpp's ple_hist — the n-gram window must follow seq_rm/cp/keep semantics; in Atlas this belongs in the sequence-state manager next to the SSM pool slots, and must be snapshot/rollback-aware for spec decode (Atlas's ssm_snapshot* machinery is the right hook; llama.cpp needed [TAG_RECURRENT_ROLLBACK_SPLITS] for exactly this).
4. 8. Small ops: swiglu_oai pointwise kernel; verify imrope against mrope_pos.rs.

Phase 4 — weights + serving.

1. 9. Loader: GGUF + HF safetensors paths; fuse per_layer_token_embd into one tensor (llama.cpp convention) but place it CPU-resident by default with pinned staging — add a tensor-placement policy to weight_loader/ (Atlas lacks --override-tensor; pinned_pack.rs/token_overlay.rs are the natural extension point). On GB10's unified memory this is a bandwidth decision: measure row-gather-over-UVM vs pinned double-buffer; llama.cpp's mmap warning (llama-model-loader.cpp:1191-1196) shows the mmap+CPU-override trap to avoid.
2. 10. MTP: route the nextn layer through mtp_head/ with HC-aware input (mtp_carry.rs), reusing the DFlash/DSv4-MTP precedent.
3. 11. Validation: per-layer cosine gates against HF reference (Atlas already has this discipline — numeric.rs, preflight preflight/deepseek_v4.rs); add a preflight/qwen4exp.rs mirroring the DSv4 tensor-shape checks.

Where to diverge from llama.cpp: (1) do *not* copy the matmul-hint Hadamard indirection — fuse WHT directly; (2) llama.cpp's recurrent memory is per-sequence rows with CPU offload — Atlas's paged SSM pool + snapshots should absorb PLE state rather than adopting the row model; (3) llama.cpp's ubatch splitting keeps rollback tokens together — Atlas's CUDA-graph verify paths need the PLE window carried in the existing speculative carry buffers instead; (4) indexer-cache mirroring is irrelevant on single-device GB10 — skip.

---

(e) Risks and open questions

1. 1. Top-k gather attention performance: the biggest unknown. Flash-decode over 512 scattered slots needs either a gather kernel + flash pass or an index-driven FA; llama.cpp pays for this with a huge graph-node budget (n_tokens*40, llama-context.cpp:2310) — Atlas's CUDA-graph capture must handle the same op-count explosion (HC + indexer + PLE per layer × 48 layers).
2. 2. PLE state under speculative rollback: llama.cpp needed explicit rollback-snapshot tagging; Atlas's ghost-token verify paths (verify_e.rs) synthesize steps — the PLE n-gram window and dilated conv state must be rolled back identically or acceptance/resample diverges. Highest correctness risk in the port.
3. 3. PLE memory traffic: [n_layer × n_vocab × n_embd] at 256K vocab is tens of GB in BF16; even row-gathered it's ~48 embedding reads/token. Needs a quantization policy (llama.cpp keeps it at token-embedding type; Atlas could NVFP4 it — rows are independent, as llama.cpp's imatrix note observes, llama-quant.cpp:1283).
4. 4. hc_mult=2 alpha scales: whether Atlas's DSv4 Sinkhorn kernels are bit-compatible at hc=2 with per-site alphas, or need a rewrite; the kernels are tuned for hc=4 (hyper_connection.cu:18).
5. 5. imrope semantics: llama.cpp distinguishes it from mrope (llama-hparams.cpp:260-262); the exact position-stream rule must be diffed against Atlas's mrope_pos.rs before trusting vision/long-context paths.
6. 6. Attention-layer V Hadamard (self_v_rot): applied to V *before cache write and after read* — the cache stores rotated V. This interacts with Atlas's TurboQuant KV (which also applies WHT before quantization): composing two rotations (or double-rotating) must be checked; likely the model's V-rotation should be fused into the existing WHT bookend rather than stacked.
7. 7. Gate formula / norm placement in GDN: Atlas's GDN was validated against Qwen3-Next/3.5; Flash-Next's reference may differ in L2-norm placement or beta/gate details — needs a numerical diff before kernel reuse is assumed.
8. 8. Conversion fidelity: conversion/qwen4exp.py is the ground truth for tensor fusion (BA packing, interleaved QKVZ vs sequential — Atlas already supports both modes, qwen3_ssm/mod.rs:75-78); pin the exact variant rather than auto-detecting.

Flash-Next support: llama.cpp vs Atlas

(a) How llama.cpp implements qwen4exp (PR #27742 tree)

Architecture identity. LLM_ARCH_QWEN4EXP is registered as a *hybrid* architecture (llm_arch_is_hybrid true, llm_arch_is_recurrent false). It is grouped with QWEN3NEXT / QWEN35 / QWEN35MOE throughout the code, meaning it reuses the same "gated delta net (SSM/linear-attention) + sparse attention" block family, with two qwen4exp-specific additions: a per-token indexer cache (sparse attention over an indexer) and per-layer token embeddings (per_layer_token_embd).

Memory / cache strategy — the core design decision. In llama_model::create_memory (src/llama-model.cpp:2441-2536), qwen4exp is routed to a dedicated memory type:

• - needs_mem_idx = (arch == LLM_ARCH_QWEN4EXP) (:2449).
• - When indexer_head_size > 0, it instantiates llama_memory_hybrid_idx (:2496-2516) — a hybrid cache that carries *three* independent state streams: (1) a standard attention KV cache for the dense-attention layers, (2) a recurrent/SSM state (r/s) for the gated-delta-net layers, and (3) a per-token indexer cache for the sparse-attention layers.
• - Layer filters (:2460-2473): filter_attn = layers that are *not* recurrent; filter_recr = layers that *are* recurrent; filter_idx = the same non-recurrent set, gated on indexer_head_size > 0. So the indexer cache lives only on the dense-attention (sparse) layers, while the SSM state lives only on the recurrent layers.
• - If SWA is present it would instead use llama_memory_hybrid_iswa (:2476-2495); qwen4exp's production config is the non-SWA hybrid_idx path.

Tensor placement / offload. The generic split logic in llama-model.cpp treats qwen4exp identically to Qwen3Next/3.5 for the SSM/delta-net tensors:

• - get_split_segments (:588-636): for the recurrent layers it computes head_k_dim = ssm_d_state, n_k_heads = ssm_n_group, n_v_heads = ssm_dt_rank, and splits qkv_weight / ssm_conv1d / r_cache / s_cache along the head axis (Qwen3Next pattern [k0_v0, k0_v1, k1_v2, k1_v3]).
• - get_split_granularity (:661-684): aligns splits to lcm(blck_size,128) and to ssm_d_state so quant blocks don't straddle heads; the s_cache (the recurrent state) is split by n_k_heads * head_v_dim * head_v_dim.
• - The indexer cache is mirrored (pattern_idx_cache → GGML_BACKEND_SPLIT_AXIS_MIRRORED, :496-498) because the QSA indexer has a single key head whose projections are mirrored — it cannot be sharded.
• - Per-tensor and per-expert .scale / .input_scale sidecars (NVFP4-style) are loaded generically for ssm_in/out/alpha/beta and the MoE expert weights (:1563-1646), so the delta-net and MoE paths both support quantized weights.

Graph / ops. The recurrent block is the standard gated-delta-net / SSM op set (ssm_in, ssm_conv1d, ssm_dt, ssm_a, ssm_alpha, ssm_beta, ssm_out, plus r_cache/s_cache state). The sparse-attention path uses the pattern_idx_cache indexer plus the normal Q/K/V attention ops. MoE experts use the ffn_gate_up_exps / ffn_down_exps fused-expert tensors (create_tensor_gate_up_exps, :3122-3128). per_layer_token_embd is handled via the generic per-layer embedding path and can be offloaded to CPU/mmap with --override-tensor.

Conversion. conversion/qwen4exp.py maps HF weights to the GGUF tensor names above (ssm_*, ssm_conv1d, ssm_dt/a/alpha/beta, ssm_out, r_cache/s_cache, idx cache, ffn_*_exps, per_layer_token_embd) and emits the GGUF keys that drive is_recr(il), indexer_head_size, ssm_d_state, ssm_n_group, ssm_dt_rank, ssm_d_conv.

Key point for Atlas: llama.cpp's whole approach is *state-centric* — the recurrent layers keep a fixed-size SSM state per sequence (no KV growth), the sparse layers keep a full KV + a per-token indexer, and everything is expressed as a single fused graph with per-tensor device placement. It is not paged and not batched in the vLLM/Atlas sense; batching is via the graph's seq dimension and the hybrid memory's n_seq_max/n_rs_seq bookkeeping.

(b) Atlas current status per feature

Atlas (Rust, GB10, paged/batched/GPU-first) does not yet support qwen4exp. Per feature:

• - Gated delta net / SSM (recurrent block): *Missing.* No ssm_in/ssm_conv1d/ssm_dt/ssm_a/ssm_alpha/ssm_beta/ssm_out ops, no r_cache/s_cache recurrent state, no ssm_d_state/ssm_n_group/ssm_dt_rank hparams. Atlas's attention path is KV-cache based; it has no recurrent-state slot.
• - Sparse attention / indexer: *Missing.* No pattern_idx_cache equivalent, no per-token indexer cache, no top-k selection over an indexer. Atlas has standard paged KV attention only.
• - Hybrid memory (KV + recurrent + indexer in one cache): *Missing.* Atlas has a paged KV cache; it has no notion of a per-sequence fixed-size recurrent state coexisting with paged KV, nor a third indexer stream.
• - MoE / experts: *Partially exists.* Atlas has MoE/expert support for dense-MoE models, but the fused ffn_gate_up_exps/ffn_down_exps expert layout and per-expert scale sidecars used by qwen4exp are not wired for this arch.
• - per_layer_token_embd: *Missing.* No per-layer embedding table or its offload path.
• - MTP / ngram / hyper_connection: *Missing* (not required for the base qwen4exp forward, but present in the broader Qwen3.5/Next family).
• - Quantization sidecars (NVFP4 scale/input_scale): *Partially exists* for standard attention/FFN; not yet for ssm_* or expert tensors.

(c) Side-by-side comparison

(d) Prioritized adoption plan for Atlas

1. 1. Add the recurrent (gated-delta-net) op family + state. Port the ssm_in → ssm_conv1d → (ssm_dt,ssm_a,ssm_alpha,ssm_beta) → ssm_out graph and the r_cache/s_cache per-sequence fixed-size state. This is the single biggest gap. Borrow llama.cpp's head layout (ssm_d_state, ssm_n_group, ssm_dt_rank) and the lcm(blck_size,128,ssm_d_state) alignment so quant blocks don't straddle heads. *Diverge:* store the recurrent state in a contiguous per-sequence slab (not paged), since it is fixed-size — this is cheaper than paging and matches llama.cpp's n_rs_seq model.

1. 2. Introduce a hybrid cache in Atlas. Extend the paged KV cache manager to also own (a) the recurrent slab and (b) the indexer cache, keyed by sequence. Keep the three llama.cpp filters (filter_attn = non-recurrent, filter_recr = recurrent, filter_idx = non-recurrent with indexer_head_size>0) as the source of truth for which layers use which stream. *Diverge:* Atlas can keep KV paged (its strength) while the recurrent + indexer streams are dense per-sequence buffers — do not force paging onto the fixed-size state.

1. 3. Add sparse attention + per-token indexer. Implement the indexer cache (mirrored, single key head) and the top-k selection over it on the dense-attention layers. This is the qwen4exp-specific delta over Qwen3Next. Reuse llama.cpp's "indexer is mirrored, not sharded" decision.

1. 4. Wire MoE experts + per-expert quant sidecars using the fused ffn_gate_up_exps/ffn_down_exps layout and the generic .scale/.input_scale sidecar loading.

1. 5. Add per_layer_token_embd with an offload path (Atlas equivalent of --override-tensor), so the per-layer embedding table can live in host/mmap memory.

1. 6. Conversion/mapping table for HF→Atlas weights mirroring conversion/qwen4exp.py (ssm_*, ssm_conv1d, ssm_dt/a/alpha/beta, ssm_out, r/s cache, idx cache, ffn_*_exps, per_layer_token_embd, and the GGUF keys is_recr, indexer_head_size, ssm_d_state, ssm_n_group, ssm_dt_rank, ssm_d_conv).

1. 7. Validation: parity test against llama.cpp's qwen4exp logits on a short prompt, then a long-context test to exercise the recurrent-state vs KV divergence.

(e) Risks and open questions

• - Recurrent state vs paging: llama.cpp keeps SSM state as a fixed per-sequence buffer; Atlas is paged. Confirm the recurrent slab can be swapped/evicted per sequence without a paging scheme, and that prefill (long prompt) correctly folds the whole prompt into the state in one pass (the delta-net is causal/online — verify the conv + state update is associative over chunks).
• - Indexer cache growth: the per-token indexer cache grows with context (like KV). Confirm it is bounded and evictable under Atlas's memory budget; llama.cpp mirrors it (no sharding), which may be a memory ceiling on GB10.
• - Batching the recurrent state: llama.cpp batches via n_seq_max/n_rs_seq; Atlas batches via paged blocks. The recurrent op must be batched over sequences with per-sequence state pointers — verify the GPU kernel supports gather/scatter over a state slab.
• - Quantization of ssm/expert weights: sidecar scales are loaded generically in llama.cpp; confirm Atlas's dequant path handles ssm_* and per-expert scales, especially NVFP4.
• - per_layer_token_embd offload: ensure the per-layer embedding lookup is not a per-token host round-trip in the hot path; pre-stage or fuse into the first-layer op.
• - Uncertainty: I have not read the actual src/models/qwen4exp.cpp graph builder or llama-memory-hybrid-idx.* internals in this pass (the model file is large and I focused on the memory/placement/conversion wiring in llama-model.cpp). The op-level graph order (exact placement of ssm_conv1d vs the delta update, and the exact top-k indexer kernel) should be confirmed against src/models/qwen4exp.cpp before finalizing the Atlas op list. The feature set, memory topology, and placement decisions above are taken directly from the cited llama-model.cpp code and are reliable.

PR 16 review: qwen4exp serving in Atlas

Scope note: this review is based on direct reading of the PR tree (atlaspr) against atlas main across the areas that carry the correctness burden: config/buffer sizing, hyper-connections (mHC highway), PLE / n-gram embedding and its history bookkeeping, QSA sparse attention + indexer, the batched/multi-sequence and speculative (verify) dispatch paths, aux-state snapshotting (Marconi/prefix cache), and the new kernels/gb10/qwen3.8-flash-next + kernels/gb10/common kernel families. I did not execute the test suite (no GPU here); everything below is static review plus the PR's own documented measurements.

---

(a) What the PR implements, feature by feature

1. Config parsing / model registration.

The qwen4_exp model type is parsed into ModelConfig with both spellings of the stream count retained: hc_count (the checkpoint's own field name, read by the manifest and the reference crate) is *mapped onto* hc_mult (the field the engine dispatches on). Both are kept so the reference/manifest readers keep working — see the sizing tests in crates/spark-runtime/src/buffers/sizes.rs:611-676 (both_spellings_size_the_same_highway). Layer typing reuses the existing LinearAttention (GDN) / FullAttention (QSA) alternation, with index_topk / index_compress_ratio / index_n_heads / index_head_dim fields driving the sparse-attention path.

2. Hyper-connections (mHC, multi-stream highway).

An FP32 [n, hc_mult*H] stream highway (buffers.hc_streams()) with hc_expand / hc_pre_site / hc_post_site / hc_head_site ops, per-site weight groups (hc.attn, hc.ffn, hc.head), low-rank mixing scratch (hc_lowrank_scratch), and a HcVariant abstraction that encodes the Qwen-vs-DeepSeek-V4 divergence: on Qwen, hc_pre's grouped hc_norm *is* the block input norm, so no second RMS is applied and there is no per-layer input_layernorm (qwen3_attention/trait_impl/prefill_inner.rs:640-659, multi_seq/mod.rs:207-223). Highway sites are wired in all four forward shapes: single decode (decode_a), batched multi-seq decode (qwen3_ssm/trait_decode_multi_seq/hc.rs, qwen3_attention/trait_impl/multi_seq/mod.rs:131+), batched verify (qwen3_ssm/trait_decode_batched_hc.rs), and prefill (prefill_inner.rs).

3. Gated DeltaNet (GDN) changes.

The SSM layer (qwen3_ssm) gained an hc and a ple optional member, a per-seq SsmLayerState { …, ple: Option<PleSeqState> }, a batched-GEMM decode core (try_decode_multi_seq_ssm_batched) and a per-seq recurrence fallback, and the highway-aware multi-seq driver described above. The batched core is deliberately *not* engaged under the highway yet because its epilogues fold the residual in (trait_decode_multi_seq/hc.rs:26-29) — an honest, documented deferral.

4. PLE / n-gram embedding.

crates/spark-model/src/layers/ple/layer.rs + kernels/gb10/qwen3.8-flash-next/nvfp4/ple.cu (+ the kernels/gb10/common/qwen4exp_ple.cu twin): key/value projections of hashed n-gram embeddings, a highway-conditioned gate (ple_gate), a dilated depthwise causal conv with in-place rolled carry (ple_conv, state [(k_size-1)*dilation, C], FP32 chain end-to-end — see the PRECISION NOTE referenced in ops/ple.rs:10-16), and ple_add_highway, which injects into the FP32 highway *before* the layer's attention hyper-connection, matching the reference ordering.

History bookkeeping is the interesting part:

• - Host-side n-gram hashing is hoisted before CUDA-graph capture/replay via Layer::decode_prestage (decode_a.rs:153-164) and Layer::verify_prestage for the whole draft window (verify_c.rs:199-212, verify_e.rs:421-437), so a replayed graph only reads refreshed fixed-address buffers. The per-sequence windowing in verify_e.rs:427-437 (tokens[off[i]..off[i+1]]) is correct and explicitly guards against injecting one sequence's lexical history into another.
• - Per-row decode inside the highway (trait_decode_multi_seq/hc.rs:81-121): each row reads *its own* token from ctx.host_token_ids and *its own* PleSeqState.
• - Verify rollback: at verify widths the conv launch is split per row and each row's post-conv carry is parked in verify_snaps (ple/layer.rs:413-456), so a rejected draft can rewind the conv state; prefill widths keep the batched launch and set verify_snap_rows = 0.
• - NVMe fault-in for the (large) embedding table is part of the same prestaged host half (verify_c.rs:199 comment: "n-gram hash + NVMe fault-in + slot upload").

5. QSA sparse attention + indexer.

layers/qsa.rs + layers/qsa_select.rs: per-seq QsaSeqState lazily created via helpers::qsa_seq_state (helpers.rs:256-269); prefill *ingests* raw indexer keys per chunk (prefill_inner.rs:661-668), decode steps decode_select against them; selection is inert below index_topk + index_compress_ratio - 1 visible tokens (the "budget" bound, decode_a2.rs:127-134). Selection scratch is a fixed-address arena sized in buffers/sizes.rs:539-550 (ROWS=2048, q/k-weight + key + score-block + topk regions), with the ROWS constant duplicated in qsa_select.rs:136 — a coupling that should be a shared const.

6. MoE routing.

Reuses the existing event-dispatch / hash-MoE machinery (token_ids stable buffer for tid2eid), plus route-aware LoRA handling on the new paths (moe_lora_route, reject_decode_lora on multi-seq/speculative). The batched FFN ladder explicitly *refuses* shortcut-MoE models rather than silently computing an incomplete block (multi_seq/ffn.rs:41-49) — good fail-loud discipline.

7. MTP / speculative.

layers/qwen4exp_mtp_forward.rs, layers/deepseek_v4_mtp.rs, and the verify family (verify_a/b/c/c2/d/e/fused), with PLE/QSA hooks threaded through: per-seq prestaging, row_owner maps so the indexer advances once per row against its owner (verify_e.rs:474-481, multi_seq/mod.rs:247-289), and mapped-argmax to kill a measured 137 ms copy-engine tail stall (verify_e.rs:566-581).

8. Sequence-op bookkeeping (prefix cache / snapshots / HSS).

A new generic aux-state mechanism: Layer::{has_aux_state, snapshot_aux, restore_aux} driven by collect_aux_states / apply_aux_states / requires_aux_state (model/trait_impl/mod.rs:893-935), blobs attached to SSM snapshot slots (model/ssm_snapshot.rs:452-470), and a restore gate that declines aux-less snapshots for aux-carrying models (prefill_a.rs:155, prefill_c.rs:194, prefill_b/prefix_lookup.rs:208-211). Aux is captured at every snapshot site: chunk-boundary (prefill_b/finalize_last.rs:393-400), intermediate checkpoint (prefill_b/save_checkpoint.rs:137-146), decode Marconi checkpoint and retire-leaf (decode_checkpoint.rs:131-142, 254-263). This is the right shape for PLE history + QSA indexer keys, which a GDN-only snapshot would otherwise silently drop.

9. Kernels.

kernels/gb10/qwen3.8-flash-next/ (nvfp4 PLE chain, hc ops, GDN) and kernels/gb10/common/qwen4exp_ple.cu, with oracle/parity harnesses (layers/ops/qwen4exp_oracle_ple_tests.rs runs the *production* ple_gate → ple_conv → ple_add_highway chain against a reference; ple_tests.rs, qsa_tests*.rs, microtest examples under spark-model/examples/qwen4exp_microtest/).

---

(b) Design assessment vs the reference architecture

Where the PR is right, and right in a non-obvious way:

• - Precision discipline on the PLE chain. Keeping gate/conv/add FP32 end-to-end because the result lands on the FP32 highway is exactly what the reference's numerics demand; the comment that bf16 "would not fail — it would read the wrong half of every value" (sizes.rs:627-629) shows the authors understand silent-corruption failure modes, not just loud ones.
• - **hc_pre's grouped norm *is* the input norm on Qwen.** Reusing it rather than applying a second RMS with a ones-placeholder is the correct reading of a checkpoint that has no input_layernorm; the code explicitly notes the placeholder "would NOT make a second RMS an identity" (prefill_inner.rs:652-658). This is the kind of divergence between V4 and Flash-Next that ports usually get wrong silently.
• - Aux-state as a first-class, gated concept. Rather than bolting PLE onto the SSM snapshot, the PR adds a layer-level aux contract and makes restore *refuse* incomplete snapshots. That converts a would-be silent quality bug ("serve the previous request's lexical state") into a cache miss. Correct trade.
• - Fail-loud dispatch gates. QSA on the batched path: if decode_select returns Some where the gate promised inert, the layer bails rather than serving dense-past-budget attention, which "is NOT the reference model" (multi_seq/mod.rs:247-288). Same pattern for shortcut-MoE and HSS in batched prefill.
• - Host-side hashing hoisted out of the graph region. Prestaging PLE's host half before capture/replay (single decode, verify, batched decode) is the only sound way to combine a host-computed n-gram hash with CUDA graphs; the alternative (D2H readback mid-graph) is capture-illegal, and the code refuses it explicitly rather than degrading.

Notable divergences / debatable choices:

1. 1. QSA selection is host-side (top-k D2H) and vetoes CUDA graphs for the whole model (decode_a.rs:247-250, decode_a2.rs:260-265). Sound correctness-wise — a dense-captured graph replaying after selection activates would be catastrophic — but it means once any sequence crosses the budget bound, *all* decode goes eager and/or per-seq. That is a large, permanent perf cliff for long-context Flash-Next serving, and it is architectural, not incidental.
2. 2. Batched highway decode falls back to a per-seq loop whenever QSA is active (decode_a2.rs:114-136), with logits staged through the host per sequence. Correct, but the "real concurrent batches, not queued interleaving" claim is only true below the QSA inert bound or with the batched GEMM core engaged — and the core is explicitly *not* engaged under the highway (trait_decode_multi_seq/hc.rs:26-29). The headline capability is therefore narrower than the PR text implies.
3. 3. The QSA "budget" model — index_topk interpreted as a token budget with selection inert below topk + ratio - 1 — is a serving shortcut: past the budget, selection activates and everything gets slow. It matches a 2048-token scratch design, but it hard-couples max effective context behavior to a buffer constant duplicated in two files.
4. 4. hc_streams is FP32 sized by max(mult, count) — good — but the dual-spelling (hc_mult and hc_count both live on the config) is a standing foot-gun; it is defended only by tests, not by construction.

---

(c) Defects and risks

The claimed "six silent defects" — visible and mostly sound

I could positively identify several of the class of fixes the PR claims, and they are genuine silent-corruption fixes, not cosmetic:

1. 1. hc_count vs hc_mult buffer sizing (sizes.rs:611-631): a model spelling the field hc_count would have received a 256-byte placeholder highway and written over neighboring buffers. Fixed with a max-of-both allocation and four tests. Sound.
2. 2. Padded-batch vs active-length host_token_ids (trait_decode_multi_seq/hc.rs:85-112, documented in decode_a2.rs:381-394): the per-seq PLE loop bounded by padded_n indexing an active-length slice — previously rejected every non-boundary batch ("3 host ids for 4 seqs"). The fix (skip padding rows, which carry ple: None by construction; stay loud for a live state with no token) is correct. However — see D1 below.
3. 3. Per-seq staging on the wrong stream (decode_a2.rs:147-154): staging logits copies on the scheduler's stream 0 while decode() runs on the backend default stream produced a measured two-way row swap at every C=2 step. Fixed by staging on gpu.default_stream() with copy_d2h_on_stream. Sound, and a textbook silent cross-sequence contamination bug.
4. 4. QSA ingest desync across graph replay (verify_c.rs:186-197): the indexer's ingested counter froze at the capture step while the sequence advanced, surfacing only at the MTP gate's batch-width switch ("decode at pos 34 but 26 tokens ingested"). Fixed by extending the decode_graph_unsupported veto to the verify paths. Sound.
5. 5. PLE prestaging missing on verify paths (verify_c.rs:199-212, verify_e.rs:421-437): without hoisting, the forward fell back to a D2H readback that invalidated the recording graph (error 901). Fixed; and the multi-seq variant in verify_e correctly windows per sequence.
6. 6. PLE conv carry not rewindable on rejected drafts (ple/layer.rs:413-456): per-row conv launches + parked carries at verify widths. The mechanism is sound; the *rollback application* is the risk (see D3).

Defects and risks I can identify in the current code

D1 — Padding-row PLE skip is a correctness landmine in the graph-borrow path (medium-high).

trait_decode_multi_seq/hc.rs:98-105 treats "no ple state" as "padding row, skip". That invariant holds for the all_layer_states construction in decode_a2.rs:434-468 (ple: None for pads) and for the drain-tail graph borrow. But the same take(n) loop is used with states vectors built elsewhere (e.g. decode_batched_hc.rs:164 uses the stricter host.len() >= n check — the two call sites have *diverged* on the same question). If any future caller hands a live sequence whose ple field is None because its state was constructed through a different path (checkpoint restore, h_prefill_stage variants, LoRA paths), the row silently receives no PLE injection — degraded output with no error. The safe form is to pass the active count separately (or a per-row is_pad flag) rather than inferring padding from state contents.

D2 — Aux blobs live only in a host HashMap keyed by snapshot slot (high).

ssm_snapshot.rs:65, 452-470: aux (PLE n-gram history, conv carry, QSA indexer keys) is a Mutex<HashMap<usize, Vec<u8>>> in RAM. But the SSM snapshot pool has an explicit spill/fault-in tier (ssm_snapshot_spill.rs, ssm_snapshot_faultin.rs, ssm_tier/) and free() drops aux with the slot. Consequences: (i) a snapshot spilled to the NVMe tier and faulted back — or a process restart with tier replay — restores SSM state but is aux-less, so the restore gate declines it: correct, but every warm multi-turn hit that spilled silently loses its Marconi benefit; (ii) there is no versioning/serialization format, so any blob-layout change is an in-place corruption across a rolling deploy if the tier store survives; (iii) aux is cloned on every aux() read. This is the weakest data-lifecycle decision in the PR.

D3 — Verify conv-carry rollback: snapshot mechanism visible, rewind application not proven.

ple/layer.rs parks per-row carries into st.verify_snaps and records verify_snap_rows, and ple/verify.rs exists, but I did not find (nor a test demonstrating) the rewind being invoked on *partial* acceptance in every verify variant (verify_a/b/c/c2/d/e/fused — six paths; verify_e additionally has ghost/borrowed rows). If any one accept path forgets the rewind, the conv carry is over-advanced by the rejected suffix — a silent, slowly-accumulating quality regression that only shows up with drafts rejected mid-window, i.e. exactly the regime benchmarks don't stress. This needs a targeted test, not inspection.

D4 — QSA row_owner ordering is an asserted invariant, not an enforced one.

multi_seq/mod.rs:253-288: "decode_select asserts pos == ingested, so the ordering here is the invariant, not an optimization." Several rows share one sequence's state and must advance in row order. The assert exists, but the batched path also *relies on the dispatch gate agreeing* (sel.is_none() bail). Two independent mechanisms (gate + assert) guarding one property is fine, but the gate computes activity from seqs.iter().any(|s| s.seq_len >= bound) while the assert fires per-row mid-forward — a mismatch aborts a request in flight rather than never occurring.

D5 — Exact-leaf snapshot shortcut is disabled by default.

prefix_lookup.rs:299-310: the exact full-prompt hit path is force-recomputed ("ATLAS_MARCONI_EXACT=1 re-enables") because cache-on workspace quality was ~23% vs ~60% cache-off. That is an admission that the exact-hit path (including the stashed-hidden first-token trick) is still suspect on this model family. Defensible as a default, but it means the PR's prefix-cache story is materially weaker than its code volume suggests, and the disabled path still ships.

D6 — hc_streams FP32 at max_batch × 4 × H × 4 bytes is a large fixed arena (e.g. 128×4×2560×4 ≈ 5 MB — fine on GB10, but it scales with batch and is allocated whenever either hc spelling is nonzero, including for speculative row widths r_total that exceed decode batch). Verify the arena is sized against r_total (VERIFY_ROW_CAP) and not just decode padded_n; I saw hc buffers indexed by n on the verify path with n = Σ ks[i].

D7 — Smaller items.

• - qsa_select_scratch ROWS=2048 duplicated in qsa_select.rs:136 and sizes.rs:540 — must be one const.
• - hc_pre_k is used for *both* the attn and ffn sites (trait_decode_multi_seq/hc.rs:127,187) — correct only if the checkpoint shares the low-rank projection across sites per layer; worth a parity assertion.
• - prefill_inner_hc bails on batched mode with seq_len_start == 0 and on HSS-engaged layers — correct refusals, but they mean batched prefill + highway + HSS is simply unsupported (gap, below).
• - The ple_gate kernel exists twice (nvfp4/ple.cu and common/qwen4exp_ple.cu as q4e_ple_gate); the oracle test exercises the nvfp4 one (qwen4exp_oracle_ple_tests.rs:33) while production PleLayer loads "ple"/"ple_gate" and the microtest loads "qwen4exp_ple"/"q4e_ple_gate". Two implementations of the same math, only one oracle-covered — classic drift risk.

---

(d) Gaps for full qwen4exp support

1. 1. Batched GDN projections under the highway — the QKVZ/out_proj weight-read amortization is explicitly deferred (trait_decode_multi_seq/hc.rs:26-29); concurrent-batch bandwidth scaling is therefore not yet delivered.
2. 2. QSA selection on the batched path — no per-seq indexer hook; any batch containing a past-budget sequence drops to the per-seq host-staged loop. Long-context concurrent serving is the worst case.
3. 3. CUDA graphs are off for the whole model once QSA selection is active (host top-k D2H). A capture-safe device-side top-k is the missing piece.
4. 4. Aux-state tiering — PLE/QSA blobs do not participate in the NVMe snapshot spill tier (D2); cross-restart warm restore is effectively unsupported for aux-carrying models.
5. 5. Exact-leaf prefix hit path disabled (D5) pending quality resolution.
6. 6. HSS (high-speed swap) + highway prefill batched mode — refused outright.
7. 7. Shortcut-MoE on batched FFN — refused (fine if Flash-Next doesn't use it; must be stated in the recipe).
8. 8. TP: the hc multi-seq paths do all-reduce, but the batched-highway + TP + graph interactions are barely exercised; the PR's own graph-borrow code disables borrow under comm.is_some().
9. 9. Verify rewind coverage across all six verify variants unproven (D3).

---

(e) Verdict

Merge-ready with conditions — I would not merge as-is; I would merge after a short, targeted hardening pass. The architecture reading is excellent (FP32 highway, norm reuse, PLE ordering, aux-gated restore, fail-loud gates), and the silent-defect fixes I could identify are real, well-diagnosed, and soundly repaired. The residual risk is concentrated in (1) padding-row inference by state contents (D1), (2) the un-tiered, unversioned aux blob lifecycle (D2), and (3) unproven conv-carry rewind across the six verify paths (D3).

Tests I would require before merge:

1. 1. PLE history bookkeeping across sequence ops (the single most important gap): a deterministic test that runs prompt A → snapshot → preempt → prompt B sharing a prefix → restore, and asserts token-for-token equality of greedy decode against a no-cache reference; plus the same through NVMe spill/fault-in (expected: *declined*, not wrong), and across a rolling-deploy blob version bump.
2. 2. Verify partial-acceptance rewind: for each verify variant in use, force accept-1-of-k and accept-2-of-k patterns and assert post-verify PLE conv carry and QSA ingested equal the no-draft reference state (bitwise for conv, since it's the rewindable one).
3. 3. Padded-batch concurrency matrix: live counts 1..padded_n at every ladder width, with ATLAS_HC_PERSEQ_DECODE on/off, asserting batched decode == per-seq decode logits within tolerance — this is the regression class that produced "3 host ids for 4 seqs".
4. 4. QSA budget-boundary test: sequences straddling index_topk + ratio - 1 in one batch; assert the gate routes per-seq, no sel.is_none() bail fires, and outputs match a dense-reference for below-budget and the reference sparse selection above it.
5. 5. Dual-kernel parity: run ple_gate vs q4e_ple_gate (and the conv/add twins) on identical inputs in CI, or delete one.
6. 6. End-to-end parity gate against the qwen4exp_reference crate on the checkpoint: greedy prompts at lengths spanning {below budget, across budget, across a chunk boundary, across a snapshot restore}, with the quickstart recipe run as a smoke test.
7. 7. hc arena bounds under speculative widths: ASAN/preflight assert that hc_streams/hc_post/hc_comb sizing covers r_total on the widest verify configuration.

Fix D1 (explicit pad flag), file D2 as a blocking follow-up with the blob format versioned *now* (before any tier store exists to migrate), land the D3 test, and this is a strong merge.

PR 16 review: qwen4exp serving in Atlas (Claude's review)

(a) What it implements — verified in code

• - Config/factory: parsers/qwen4_exp.rs + Qwen4ExpWeightLoader (also maps the qwen3_8_flash_next naming); both-or-neither validation of hc_mult/hc_lowrank; an MTP loader module exists.
• - HC: the low-rank gated-residual variant (rank 320) — the correct qwen4exp form, not DSV4 Sinkhorn.
• - GDN: reuses Atlas's mature qwen3_ssm stack; 36/36 layer parity vs the reference crate.
• - PLE: the standout. NgramTable does row-granular pread off NVMe (~2.5 KB/token) with a host row cache — further than either reference engine (vLLM: resident; llama.cpp: mmap CPU tensor). The pread-not-mmap rationale (no invisible page-fault stalls near CUDA threads; io::Error instead of SIGBUS) is sound and GB10-appropriate.
• - QSA: honestly scoped v1 — decode-side selection (4Q+1K indexer, 4-token mean-pooled blocks, relu-score top-512 + tail), provably inert <= 2051 visible tokens, dense prefill beyond with a one-time WARN, host top-k that vetoes CUDA graphs (measured speed-neutral).
• - Parity instrument: full Rust CPU reference crate (attn/gdn/hc/moe/ple/qsa + taps) + tap-diff + logit-KL harness with controls — built because a bisect needed it, the right reason.

(b) Design assessment

Better than both reference engines on PLE memory strategy for this hardware; simpler than both on QSA (no cell-aligned third cache, no compressed-key cache) — defensible for v1 single-GB10; stage 2 will need one of those designs. The eight-defect narrative checks out where visible: shard-mapped rows (#1), dtype-aware strides (#2), pin-lifetime fix (#5) each carry an honest post-mortem in comments.

(c) Risks beyond the PR's own list

1. 1. ATLAS_PLE_MAX_TOKENS default (2048) fails a 2,191-token prompt at layer 1 — production footgun; should be derived from the prefill chunk (8193), not an env var. Fix pre-merge.
2. 2. Long-context quality unproven where QSA actually activates (>2051): prefill-dense/decode-sparse asymmetry + the measured 0.37-nat top-20 KL drift are each fine alone; their interaction at 8K+ is unevaluated.
3. 3. Multi-stream + QSA-active untested (the C=4 result likely ran in the inert regime); selection is single-sequence in v1.
4. 4. NVMe-mount hazard documented but unguarded — add a startup read-latency probe with a hard warn.

(d) Verdict

Merge-worthy engineering, gated on: CLA, the PLE scratch default fix, a long-context (>2051) quality gate, and tap-diff in CI on a slice checkpoint. The evidence culture (bitwise-determinism control, independent reproduction, honest known-not-done) is the best seen in this evaluation. Perf context: 14.5–16.5 tok/s single NVFP4 trails llama.cpp's ~25 (Q3 GGUF), but 28.4 aggregate @4 streams is the batching llama.cpp can't do — Atlas's reason to exist.