← back to the article · Andrei Vogulkin

2026-09-03 · Atlas main @ 4df6e609 · read-only agents over the tree

Two independent defect investigations, verbatim

Report P — prod model (abliterated Qwen3.8-27B + DFlash, llama.cpp)

120-call budget, five defects, fix plan.

Atlas Flash-Next defect investigation

Defect 1 — Content-loop watchdog rollback desyncs the QSA indexer cache → HTTP 500

Symptom. A content-loop watchdog rollback during decode desynchronizes the QSA indexer's per-sequence counters; the next decode step fails the pos == st.ingested invariant and the request dies with a 500.

Code path (citations).

Root cause. rewind_verify is only correct if the verify path that ingested the k rows is the same path being rolled back. The comment at qsa_aux.rs:77-91 states the pairing assumption explicitly: the MTP path commits through commit_accepted_prefix while only the self-speculative path goes through checkpoint_ssm_states. The content-loop watchdog rollback is a third caller of rollback_aux_verify (via async_chkpt.rs:269) that does not go through the checkpoint/commit pairing the delta rewind was designed around. Concretely: the watchdog rolls back the sequence's token position by k - num_accepted, but the QSA state's ingested counter was advanced by decode_select (line 321, st.ingested = pos + 1) or prefill_ingest (line 242) at a different width than the k the watchdog passes — so ingested lands one or more tokens ahead of (or behind) the true sequence position, and the next decode_select hits the pos == st.ingested ensure at qsa.rs:291. The raw_keys buffer is left with dead rows past the new end (documented as tolerable at qsa_aux.rs:87-91), but the counter is the thing that desyncs. I could not open the watchdog's exact call site to confirm the precise k it passes; the mechanism (unpaired delta rewind) is confirmed from the code above.

Proposed fix.

Regression test. Serve-level probe: enable the content-loop watchdog, drive a prompt that triggers a rollback mid-decode with QSA active (visible prefix > inert_bound() = budget + ratio - 1), and assert the next decode step does not 500 and pos == st.ingested holds. Unit test in qsa_tests.rs: call rewind_verify with a rejected that does not equal the last verify width, then decode_select at the expected pos, assert no error.

Defect 2 — MTP speculative verify (k2 graphed path) with QSA active → silent truncation (finish_reason=stop)

Symptom. MTP speculative verify on the k2 graphed path, with QSA active, reports an internal error to the client as a clean finish_reason=stop — the generation is silently truncated.

Code path (citations).

Root cause. The k2 graphed MTP verify path captures a decode graph that includes the QSA layer's dense attention (the inert regime), but when the visible prefix exceeds inert_bound() the QSA selection is supposed to activate. Two failure modes are visible in the code: (1) the graph was captured on the dense path but replayed once selection should activate — the comment at trait_impl.rs:107-109 warns exactly this ("a graph captured on the dense path would replay wrong attention once selection activates"); and (2) the verify's error is swallowed and mapped to finish_reason=stop rather than propagated. The verify_context_limit gate (trait_impl.rs:118) is the intended guard, but the MTP k2 path does not consult it before capturing/replaying the graphed verify — it captures the graph unconditionally and the QSA layer's decode_graph_unsupported veto is only honored on the self-speculative path, not the MTP path. I could not open the MTP verify dispatcher to confirm the exact gate that fails to route; the mechanism (graphed verify bypasses the QSA inert-bound gate) is confirmed from the veto/limit pair above.

Proposed fix.

Regression test. Serve-level probe: enable MTP speculative (--speculative with MTP draft head), QSA active, drive a prompt whose visible prefix exceeds inert_bound() (e.g. > 2051 tokens for the default budget), and assert the response is not a silent finish_reason=stop with a truncated completion — either it completes correctly (eager path) or returns a 500. Unit test: assert verify_context_limit() is consulted on the MTP path by checking the graph is not captured when the limit is exceeded.

Defect 3 — Prompts > 8192 tokens: ATLAS_PLE_MAX_TOKENS off-by-one vs 8193-wide chunk, then NgramRowCache 65536-slot cap overflow

Symptom. Prompts > 8192 tokens fail: first the PLE layer refuses the 8193-token chunk (off-by-one vs the 8192-sized scratch), then for an 8193-token chunk the NgramRowCache's 65536-slot cap overflows.

Code path (citations).

Root cause. Two independent sizing bugs compound:

1. Off-by-one: max_batch_tokens = prefill_budget + max_batch_size = 8192 + 1 = 8193 (kv_cache.rs:114), but the serve script sets ATLAS_PLE_MAX_TOKENS=8192. The PLE floor is max(8193, 2048) = 8193 (qwen4_exp.rs:197), so the 8192 override is below the floor and the 8193-token chunk is refused at layer.rs:267. The off-by-one is that max_batch_tokens includes max_batch_size (the decode slot) but the PLE scratch is sized for the prefill chunk width, which is prefill_budget + 1 (the +1 being the decode token fused into the mixed step). The serve script should set ATLAS_PLE_MAX_TOKENS >= max_batch_tokens, i.e. ≥ 8193.

2. Slot cap: The NgramRowCache defaults to 65536 slots (ple.rs:47), but a single 8193-token prefill pins 8193 × 16 = 131,088 rows (one per token per ngram head). The victim() function (ngram_cache.rs:431-454) cannot find an unpinned slot and bails. The slot count must be ≥ max_prefill_tokens × ngram_heads for the largest chunk.

Proposed fix.

Regression test. Serve-level probe: send a prompt of exactly 8193 tokens (or 8192 + 1) with the default serve config and assert it does not 500 with "PLE: 8193 tokens exceeds the 8192" or "NgramRowCache: every one of 65536 slots is pinned". Unit test in ple_tests.rs: construct a PleLayer with max_tokens = 8192 and call forward with num_tokens = 8193, assert the error message names ATLAS_PLE_MAX_TOKENS; then with max_tokens = 8193, assert no error. Unit test in ngram_cache/tests.rs: open a cache with 65536 slots, resolve 131,088 distinct row ids, assert victim() bails with the expected message; then with 131,088 slots, assert no bail.

Defect 4 — Continuation prefill chunks (chunk 2+) fault with CUDA_ERROR_MISALIGNED_ADDRESS at layer 0; chunk 0 never faults

Symptom. Continuation prefill chunks (chunk 2 and later) fault with CUDA_ERROR_MISALIGNED_ADDRESS at layer 0. Chunk 0 never faults. The process survives with a dead CUDA context.

Code path (citations).

Root cause. I could not open the GDN prefill kernel's launch code to confirm the exact pointer/stride that is misaligned for chunk 2+. The mechanism is most likely one of:

1. Arena buffer offset: The continuation chunk's input tensor is a slice of the arena buffer (offset by chunk_offset × hidden × dtype_bytes), and the GDN prefill kernel assumes 16-byte (or 256-byte) alignment of the base pointer. Chunk 0 starts at offset 0 (aligned); chunk 2+ starts at a non-aligned offset if chunk_offset × hidden × 2 is not a multiple of the alignment requirement.

2. Paged KV cache block table: The continuation chunk's block table is a continuation of the chunk-0 block table, and the GDN kernel's block-table lookup assumes the table is aligned to a 16-byte boundary. The chunk-0 table is freshly uploaded (aligned); the chunk-2+ table is appended (potentially misaligned).

3. SSM state pointer: The GDN layer's SSM state is a per-sequence device pointer that is advanced by chunk 0 and then used by chunk 2+. If the SSM state pointer is not re-aligned for the continuation chunk, the kernel reads a misaligned address.

The fact that chunk 0 never faults and the process survives with a dead CUDA context (rather than crashing) is consistent with a single misaligned read that poisons the context but does not trigger a synchronous error (CUDA errors are sticky and asynchronous).

Proposed fix.

Regression test. Serve-level probe: send a prompt that requires ≥ 3 prefill chunks (e.g. 25000 tokens with --max-prefill-tokens=8192), and assert chunk 2+ does not fault with CUDA_ERROR_MISALIGNED_ADDRESS. Check the CUDA error code after each chunk in the serve log. Unit test: construct a GDN layer, call prefill with chunk_offset = 8192 (chunk 1) and chunk_offset = 16384 (chunk 2), assert no misalignment error.

Defect 5 — ATLAS_QSA_MAX_TOKENS default 32768 aborts decode mid-stream with a 500

Symptom. With the default ATLAS_QSA_MAX_TOKENS=32768, decode aborts mid-stream with a 500 once the sequence position reaches 32768.

Code path (citations).

Root cause. The QSA indexer's per-sequence key cache (raw_keys, block_keys) is allocated once at layer construction for max_tokens = 32768 tokens. Once the sequence position pos reaches 32768, the decode_select guard at qsa.rs:297 fires and the request 500s. The default of 32768 is a hard cap on the sequence length, not a soft limit — there is no growth path. For long-form generations (20K+ tokens) that continue past 32768 total tokens (prompt + completion), the decode aborts.

The default should derive from the model's max_position_embeddings (262144 for this model) or from the serve config's --max-seq-len, not from a hardcoded 32768. The cost is max_tokens × hd × 2 bytes for raw_keys and max_tokens/ratio × hd × 2 bytes for block_keys — at hd = 128, ratio = 4, that is 32768 × 128 × 2 + 8192 × 128 × 2 = 8.4 MB + 2.1 MB = 10.5 MB per sequence, which is negligible. At 262144 tokens it is 67 MB per sequence, still reasonable.

Proposed fix.

Regression test. Serve-level probe: send a prompt of 30000 tokens with max_tokens = 5000 (so the total sequence length exceeds 32768), and assert decode does not 500 at pos = 32768. Unit test in qsa_tests.rs: construct a QsaIndexer with max_tokens = 32768, call decode_select at pos = 32768, assert the error message names ATLAS_QSA_MAX_TOKENS; then with max_tokens = 65536, assert no error.

Fix plan

Ordered by smallest-safe-first:

1. Fix the PLE off-by-one (Defect 3a).

2. Fix the NgramRowCache slot cap (Defect 3b).

3. Fix the QSA max_tokens default (Defect 5).

4. Fix the QSA rewind pairing (Defect 1).

5. Fix the MTP k2 graphed verify gate (Defect 2).

6. Fix the continuation prefill misalignment (Defect 4).

Prerequisites for serving long-form (20K+ token) generations reliably: items 1, 2, 3, and 6. Items 4 and 5 are required for speculative decode with QSA active, which is a performance optimization, not a correctness requirement for long-form generation.

Report A — config A (Qwen3.8-Flash-Next UD-Q4_K_XL, llama.cpp)

Same brief, same budget; the better-anchored of the two and the basis of the fix plan.

Atlas Flash-Next defect investigation

Scope note on evidence quality: everything below is grounded in code I opened in this tree (cited path:line). Where I could not confirm a mechanism from source I say so explicitly; those are marked [unconfirmed]. Two greps returned empty for strings I expected (rollback_aux_decode, has_aux_state consumers outside model/trait_impl/mod.rs, verify_context_limit inside crates/spark-server other than scheduler/mod.rs:726) — I treat those absences as findings (missing wiring), not as tool failure, except where noted.

Defect 1 — content-loop watchdog rollback desyncs the QSA indexer cache → HTTP 500

Symptom

QSA: decode at pos 25 but 26 tokens ingested — the indexer cache lost sync (prefix-cache skip or a rewound sequence) (crates/spark-model/src/layers/qsa.rs:291-296), surfaced as a 500 on a non-streaming request after the content-phase loop watchdog fired.

Code path

Root cause

The watchdog rewind is a cursor rewind: it lowers seq_len/tokens and restores the SSM recurrent state from a ring snapshot. QSA carries a second, independent cursor — QsaSeqState::ingested (plus pooled, raw_keys, block_keys, table_len, qsa.rs:53-64) — that is only ever moved forward by prefill/decode ingest and is rewound only by the MTP verify-commit hook. So after a rollback of dropped tokens, seq.seq_len is dropped lower while st.ingested is unchanged; the very next decode calls decode_select(pos = seq_len) and trips pos == st.ingested (qsa.rs:291). The error message itself names this case ("or a rewound sequence"), i.e. the author knew rewind was required but only wired it for speculative verify.

Two contributing facts I confirmed:

1. has_aux_state() (qwen3_attention/trait_impl.rs:114-116) is consumed only by requires_aux_state() (model/trait_impl/mod.rs:915-917), which is a snapshot-restore soundness gate. There is no equivalent "this model needs aux rewind on a watchdog rollback" signal for the scheduler, and grep has_aux_state shows no scheduler consumer.

2. The SSM ring gate is a correctness gate that declines (falls back to a hard stop) when no snapshot exists (rollback.rs:88-95, :229-253). There is no analogous gate for aux state, so nothing declines; the rollback proceeds and the 500 lands one step later inside the model.

[unconfirmed] Whether the SSM ring is enabled for qwen4_exp in this serve config (the ring is a.ssm_rollback_ring.is_enabled(), rollback.rs:229) — I did not open the config plumbing. If it is disabled, the rollback also silently corrupts GDN h_state, which would make the 500 the visible half of a larger desync.

Proposed fix

Make aux rewind a first-class part of the rollback contract, mirroring the SSM treatment:

1. Model API: add Model::rewind_aux_decode(&self, seq: &mut SequenceState, dropped: usize) -> Result<()> next to save_decode_ssm_snapshot/restore_decode_ssm_snapshot, implemented in model/trait_impl by looping self.layers[i].rewind_aux_decode(state, dropped). On Qwen3AttentionLayer, delegate to a new QsaIndexer::rewind_decode(st, dropped) that does st.ingested -= dropped; st.pooled = st.ingested / ratio; st.table_len = 0; (the last line is required: table_len is a monotone high-water mark, qsa.rs:410-413, and a stale value would skip the identity-table upload for a wider later selection). Add the PLE counterpart in layers/ple/aux_state.rs (truncate st.history by dropped, restore st.conv from st.verify_snaps-style carry if a per-token carry exists; otherwise the layer must report "not rewindable").

2. Exactness, not approximation: prefer the existing snapshot mechanism over arithmetic. snapshot_aux already produces a byte-exact blob (qwen3_attention/trait_impl.rs:149-167); extend SsmDecodeRing entries to carry (ssm_slot, Vec<(layer_idx, blob)>) so rollback_to_boundary restores SSM and aux from the same boundary, and restore_aux (:169-188) is the restore. This reuses the audited D2D path instead of inventing a second rewind.

3. Gate: in rollback_to_boundary, compute let aux_ok = !model.requires_aux_state() || ring entry carries aux and fold it into the existing eligibility predicate (rollback.rs:230-253) with a new RollbackFallback::NoAuxSnapshot. Declining → legacy hard stop is safe and stream-safe; a desynced indexer is not.

4. Cheap belt-braces (not a fix): decode_select's assert could, when pos < st.ingested and the delta is small and the tail was never re-pooled, self-heal by rewinding and logging WARN. I would ship this only behind ATLAS_QSA_AUTOREWIND=1 for canary, because silent self-heal hides real desyncs.

Regression test

Defect 2 — MTP verify (k2 graphed) with QSA active reports an error as finish_reason=stop (silent truncation)

Symptom

A verify-path failure is delivered to the client as a clean finish_reason=stop; the generation is truncated with no error.

Code path

Root cause

Two independent defects compose:

(a) The gate is a snapshot, not a pin. verify_ctx_limit is compared against a.seq.seq_len at dispatch time (scheduler/mod.rs:761-762). A sequence whose seq_len is just below inert_bound (2051 on this card: index_topk=2048, ratio=4) is admitted to the MTP lane; the K-row verify then advances the indexer across the bound, and the next step's verify runs with QSA selection active — the exact configuration the comment says "the batched verify refuses". The gate therefore leaks precisely at the boundary crossing, which is why it looks intermittent and length-dependent. [unconfirmed] the exact refusal site inside the graphed k2 verify: I read the gate and the aux-ingest comments but did not open the k2 graph capture body in verify_a.rs/verify_fused.rs, so I cannot cite the bail! that fires.

(b) The error channel collapses into "finish". The in-tree comment states the consequence outright — "a verify error finishes the request" (scheduler/mod.rs:757-759) — and the observed finish_reason=stop confirms the step error is consumed by the request-completion path rather than routed to the SSE/HTTP error channel. I did not locate the specific match ... Err(_) => finish(...) in this session (grep for the refusal string returned nothing, so the refusal is likely a generic Context-wrapped error, not a greppable literal). This is the severity driver: an engine error is being laundered into a successful-looking response.

Note also that decode_a2.rs:120-137 keeps QSA-active batches on the per-seq loop because "the ms attention path has no per-seq indexer hook yet — dense past the budget is NOT the reference model". The same reasoning ("this configuration cannot produce reference-correct output, so refuse it") was applied to the batch path but not to the verify path, where refusal instead becomes a client-visible lie.

Proposed fix

1. Route the error honestly (highest priority, smallest diff). In the MTP step error path, a verify failure must produce finish_reason + an OpenAI-style error payload (or HTTP 500 for non-streaming), never a bare stop. Concretely: in scheduler/mtp_step.rs / the step_mtp caller in scheduler/mod.rs, distinguish "step error" from "sequence finished" and mark the request failed; add a tracing::error! with seq_id, seq_len, k, and the Caused by chain.

2. Make the gate sticky and hysteresis-free. Replace the instantaneous predicate with a per-sequence latch: once seq.seq_len >= lim - k_max (where k_max = num_drafts + 1, i.e. the widest verify the lane can issue), set a.disable_mtp = true for the remainder of the request (the field already exists and is already honored at scheduler/mod.rs:774). This removes the crossing race entirely at zero correctness risk, and costs only the MTP speedup for the tail of long sequences — which is exactly what the existing gate already does past 2051.

3. Or make verify QSA-aware (larger, better): the verify already ingests aux per row (verify_b.rs:354, verify_e.rs:476) and already rewinds it (async_chkpt.rs:266-277), so the missing piece is only the selection application for rows past the inert bound. Until that exists, the refusal must be a decline at dispatch (item 2), not an error mid-forward.

4. Add a debug assertion in the verify entry: if any layer has an active QSA selection (visible > inert_bound), bail before any state mutation with a distinct error type VerifyUnsupportedWithActiveQsa, so the scheduler can convert it into a serial-decode retry instead of a request failure.

Regression test

Defect 3 — prompts > 8192: ATLAS_PLE_MAX_TOKENS off-by-one vs the 8193-wide chunk, then NgramRowCache 65536-slot cap

Symptom

Two sequential failures on a >8192-token prompt: first PLE: {num_tokens} tokens exceeds the {max_tokens} this layer was sized for. Raise ATLAS_PLE_MAX_TOKENS (costs tokens*10240*14 bytes of scratch) or lower the prefill chunk size. with num_tokens = 8193 and max_tokens = 8192; after raising it, an NgramRowCache 65536-slot overflow for the same 8193-token chunk.

Code path

Root cause

max_tokens is a capacity constant with no derivation: it is a hand-set default (8192) that must happen to be ≥ the widest chunk the prefill chunker can emit, and the two are configured by different knobs. Any prompt > 8192 with a 8192 chunk size produces a chunk of 8193 (off-by-one in the chunker) and the ensure fires. Raising ATLAS_PLE_MAX_TOKENS moves the buffer capacity but exposes the second, independent cap: the n-gram row cache is sized 65536 slots, which is a fixed number, not f(max_tokens × heads), so 8193 rows × heads > 65536 overflows. Two hard-coded numbers, neither derived from the other or from the chunker.

Proposed fix

1. Fix the chunker's off-by-one (the actual bug): the chunker must emit chunks of ≤ chunk_size tokens; find the ceil/block-rounding in model/trait_impl/prefill_b/ (chunk loop feeding prefill(..., num_tokens, ..., seq_len_start, ...)) and replace end = min(ceil(len/chunk)*chunk + 1, len)-style arithmetic with end = min(start + chunk, len), keeping the final-token row as its own explicit step if the pipeline needs it. Add a property test: for every len ∈ [1, 3*chunk+2], max(chunk lens) <= chunk and sum == len.

2. Derive the capacity, don't hand-set it. At model build time compute

ple_max_tokens = max(chunk_size, max_model_len.min(chunk_size)) rounded up to block_size, + 1 — i.e. max_tokens must be ≥ the widest chunk the chunker can emit by construction, so the two can never disagree. Keep ATLAS_PLE_MAX_TOKENS as an override only, and log the derived value at startup (INFO ple sized: max_tokens=... chunk=... scratch=... GiB). Do the same for ATLAS_QSA_MAX_TOKENS (qsa.rs:131), which today is an unrelated 32768 (see Defect 5).

3. Size the row cache from the same number: ngram_row_cache_slots = max_tokens * ngram_heads() (or make it an evicting LRU keyed by row id, which is the better fix — the cache exists to dedupe rows, so a full cache should evict, not error). 8193 × heads must be representable, and for a 32K context it must be bounded, so eviction is the correct semantics.

4. Make the error actionable (it already partly is): include chunk_size, seq_len_start, and the derived recommendation in the message so an operator can fix the right knob.

Regression test

Defect 4 — continuation prefill chunks (chunk 2+) fault CUDA_ERROR_MISALIGNED_ADDRESS at layer 0; chunk 0 never faults; process survives with a dead CUDA context

Symptom

CUDA_ERROR_MISALIGNED_ADDRESS (719) raised at layer 0 on prefill chunk ≥ 2. Chunk 0 is clean. The process does not die — the CUDA context is dead, so every subsequent request fails.

Code path

Root cause

CUDA_ERROR_MISALIGNED_ADDRESS on a chunk-2-only path means a kernel is handed a base pointer or pitch that is not a multiple of its vector width (these kernels do 16-byte vectorized loads). The code shape that produces exactly this asymmetry is the one documented at qsa.rs:100-103: for chunk 0 the attention/ingest path uses the contiguous cache-skip view (no device block table, implicit stride), while for chunk ≥ 2 it must use the paged view (device block table + kv_write_start offset). If the chunk-2 branch computes a byte offset from a token index with the wrong element size, or passes kv_write_start (tokens) where a byte offset is expected, or passes the chunk-local hidden pointer with the sequence-level pitch (or vice versa), the resulting address is misaligned by a non-multiple-of-16 amount and the very first kernel that touches it — layer 0 — faults.

The strongest specific suspects I can name from code I actually read, in order:

1. copy_d2d_2d_async at qsa.rs:231-239: dst_pitch = hd*2 and dst = raw_keys.offset((seq_start+off)*hd*2) are only mutually consistent when seq_start is a whole number of rows — fine — but the src pitch qkw*2 = (n_heads+1)*hd*2 is computed from self.n_heads/self.hd as u32 while off/seq_start are usize; a chunk-2 call where the caller passes a hidden pointer already offset into a larger slab (rather than the chunk base) would make hidden.offset(off*hidden*2) (qsa.rs:221) point mid-row, and the subsequent pitched copy would then read/write misaligned. This matches "chunk 0 never faults" because chunk 0's seq_start == 0 makes any double-offset a no-op.

2. prefill_table_dev (qsa.rs:169, sized max_tokens.div_ceil(8)*4): if the chunk-2 branch uploads the host block_table Vec without bounding it to ceil(seq_len/8) entries, the gather kernel reads past the allocation — a bad-address fault whose reported code depends on where the mapping breaks.

3. The generic chunked-prefill kv_write_start plumbing (trait_impl.rs:229) into the paged attention write kernel.

I could not confirm which of these is the actual faulting kernel within budget — I did not open qsa_select.rs (prefill_select) or the paged attention write path. Stated plainly: the class of the bug (chunk-2-only pointer/stride derivation) is confirmed by the qsa.rs:100-103 asymmetry comment and the seq_start == 0 no-op property; the exact line is not.

Separate, and independently important: the error-handling defect. A 719 is sticky — the context is unrecoverable — yet the process keeps serving. Every later request then fails with confusing secondary errors. The engine should treat CUDA_ERROR_MISALIGNED_ADDRESS / ILLEGAL_ADDRESS / ASSERT as fatal: log, mark unhealthy, and exit non-zero so the supervisor restarts. That is a small, high-value change regardless of the pointer bug.

Proposed fix

1. Reproduce with tooling first: run the failing shape under compute-sanitizer --tool memcheck --launch-timeout 120 with ATLAS_QSA_DEBUG=1-style tracing of every copy_d2d_2d_async/gather launch (base ptr, pitch, width, height) for chunks 0..3. The sanitizer names the kernel and the offending address; that pins which of the three suspects above it is.

2. Make offsets explicit and typed. Introduce RowOff(usize) (element-count newtype) for all DevicePtr::offset call sites in qsa.rs / ple/layer.rs / the paged attention write path, so (tokens × elem_bytes) vs bytes cannot be confused. Add debug_assert! on every pitched copy: src_pitch % 16 == 0 && dst_pitch % 16 == 0 && base % 16 == 0.

3. Normalize the chunk contract: require prefill callers to pass a chunk-local hidden pointer whose row 0 is the chunk's first token, and derive all sequence-level positions from seq_len_start only (never from pointer arithmetic into a parent slab). Assert hidden.0 % 256 == 0 at entry.

4. Bound the block-table upload in prefill_select to ceil((seq_len_start + num_tokens)/8) entries and assert <= max_tokens.div_ceil(8) (qsa.rs:169).

5. Fatal-on-sticky-CUDA-error: in the GpuBackend error mapping, classify 700/715/719/720 as GpuError::Fatal, propagate an error type the server treats as "shut down", and add a health endpoint that returns 503 once a fatal error is recorded.

Regression test

Defect 5 — ATLAS_QSA_MAX_TOKENS default 32768 aborts decode mid-stream with a 500

Symptom

A long generation dies mid-stream with HTTP 500; the underlying error is QSA: pos {pos} >= ATLAS_QSA_MAX_TOKENS (qsa.rs:297-300) or QSA: {} tokens exceeds ATLAS_QSA_MAX_TOKENS={} (qsa.rs:208-213).

Code path

Root cause

ATLAS_QSA_MAX_TOKENS is a buffer capacity that is enforced as a runtime request limit, with a default that is lower than the served context length, and no component is responsible for reconciling them. The correct relationship is: the indexer's per-sequence capacity must be max_model_len (prompt + generation), because the indexer ingests every token of the sequence (qsa.rs:242, :321) and the assert is a genuine out-of-bounds guard on raw_keys. Instead it is an independent 32768 constant, so any request whose prompt_len + generated > 32768 converts a capacity question into a 500 mid-stream. Compounding it: the check fires during decode, i.e. after tokens have already been streamed to the client, so the client gets a truncated stream plus an error — the worst possible failure shape.

Note the same non-derivation as Defect 3: two independently hand-set numbers (max_model_len, ATLAS_QSA_MAX_TOKENS) that must agree.

Proposed fix

1. Derive it. QsaIndexer::new should take max_seq_len: usize (the model's max_model_len, rounded up to block_size) as a parameter and use ATLAS_QSA_MAX_TOKENS only as an explicit override, logging the derived value at build. The allocator already scales with it (qsa.rs:163, :169), so this is a one-line default change plus threading max_model_len into the layer builder.

2. Admission-time, not mid-stream. Add a scheduler admission check: reject with HTTP 400 ("prompt + max_tokens exceeds the QSA indexer capacity (N)") when prompt_tokens + requested_max > indexer_capacity. The scheduler already has per-model limits available (model.verify_context_limit() is fetched at scheduler/mod.rs:726; add model.indexer_capacity() alongside it).

3. Clamp, don't kill, when the client didn't ask: if max_tokens is absent, clamp the effective generation budget to capacity - prompt_len and log once, rather than erroring at token capacity.

4. Keep the ensure!s in qsa.rs as the last-resort bounds guard (they are correct as OOB protection) but change their text to say "exceeded the indexer capacity derived from max_model_len" so the operator fixes the right knob.

Regression test

Fix plan

Ordered smallest-safe-first. P3 and P5 are the two-line default-derivation changes that unblock the long-prompt path; P2's error-routing item is the smallest correctness-of-observability change and should not wait for anything.

1. PR-A — Never launder an engine error into finish_reason=stop.

Files: crates/spark-server/src/scheduler/mtp_step.rs, crates/spark-server/src/scheduler/mod.rs (step-error handling around :754-830), the SSE/HTTP error mapping. Risk: low (error path only). No output changes on the happy path. Verified: inject a forced verify failure behind a test env flag; assert the client receives an error payload (non-streaming: HTTP 500) and never finish_reason=stop with completion_tokens < max_tokens. Prerequisite for trusting every other probe below.

2. PR-B — Fatal-on-sticky-CUDA-error + health gate.

Files: crates/runtime/.../gpu error mapping (GpuError::Fatal for 700/715/719/720), server shutdown + /health 503. Risk: low-medium (changes crash behaviour; needs supervisor/restart policy in place). Verified: ATLAS_TEST_MISALIGN=1 poison test exits non-zero within one step; /health flips to 503. Prerequisite for safely running the sanitizer work in PR-D.

3. PR-C — Derive ATLAS_QSA_MAX_TOKENS from max_model_len; admit-time 400 / clamp.

Files: crates/spark-model/src/layers/qsa.rs:131-134,163,169,208-213,297-300; layer builder threading max_model_len; Model::indexer_capacity(); scheduler admission (crates/spark-server/src/scheduler/). Risk: low (memory growth is bounded and computable; publish the per-seq number in the startup log). Verified: the three serve cases in Defect 5; nvidia-smi headroom at target concurrency. Prerequisite for 20K+ serving.

4. PR-D — Fix the prefill chunker off-by-one and derive ATLAS_PLE_MAX_TOKENS from the chunker; make NgramRowCache evicting / sized max_tokens × heads.

Files: crates/spark-model/src/model/trait_impl/prefill_b/ (chunk loop), crates/spark-model/src/layers/ple/layer.rs:267-273, the NgramRowCache impl, model build (single derived capacity shared by PLE + QSA + chunker). Risk: medium (touches the chunk boundary contract; needs the PR-G sanitizer run to prove no pointer regressions). Verified: chunker property test (max chunk ≤ chunk, sum == len); prompts 8191/8192/8193/8194/16384/16385 → all 200. Prerequisite for 20K+ serving.

5. PR-E — Pin MTP off once a sequence approaches the QSA inert bound.

Files: crates/spark-server/src/scheduler/mod.rs:754-780 (replace the instantaneous verify_ctx_limit predicate with a per-seq latch seq_len + num_drafts + 1 >= lim → a.disable_mtp = true), plus a pre-mutation VerifyUnsupportedWithActiveQsa bail in the verify entry (verify_a.rs / verify_fused.rs). Risk: low (only declines an optimization; output must be identical to serial decode). Verified: the seq_len ∈ {2048..2052} dispatch table test; the padded-prompt serve probe with token-for-token equality vs MTP-off (the only check that also catches a silently-dense verify).

6. PR-F — Watchdog rollback rewinds aux state (QSA + PLE), and declines when it cannot.

Files: crates/spark-model/src/layer/transformer_layer.rs (new rewind_aux_decode), crates/spark-model/src/layers/qsa.rs + qsa_aux.rs (rewind_decode: ingested -= dropped, pooled = ingested/ratio, table_len = 0), crates/spark-model/src/layers/ple/aux_state.rs, crates/spark-server/src/scheduler/rollback.rs (fold aux into the find_last_boundary_with_snapshot eligibility gate + new RollbackFallback::NoAuxSnapshot; carry aux blobs in SsmDecodeRing entries via the existing snapshot_aux/restore_aux, qwen3_attention/trait_impl.rs:149-188). Risk: medium (new state path on a hot code path; the decline fallback keeps it safe-by-default). Verified: the spark-server fake-model unit test; qsa_tests.rs rewind-then-decode parity; the non-streaming watchdog-triggering serve probe asserting 200 and zero indexer cache lost sync. Prerequisite for reliable long-form (watchdogs fire far more often at 20K+).

7. PR-G — Root-cause and fix the chunk-2 MISALIGNED_ADDRESS; typed row offsets; fatal classification test.

Files: crates/spark-model/src/layers/qsa.rs (prefill_ingest pitched copy :231-239, prefill_select in qsa_select.rs, prefill_table_dev bound :169), crates/spark-model/src/layers/qsa_select.rs, the paged attention write path consuming kv_write_start (layers/qwen3_attention/trait_impl.rs:229), plus a RowOff newtype and pitch % 16 == 0 debug asserts across the pitched copies. Risk: medium-high (pointer arithmetic; needs hardware). Start with a compute-sanitizer repro to identify the faulting kernel — the three candidate sites are listed under Defect 4 and I could not narrow it further from source alone. Verified: two-chunk prefill bitwise parity vs single-chunk (qsa_tests.rs) under compute-sanitizer --tool memcheck; 20000-token prompt (3 chunks) clean under sanitizer. Hard prerequisite for 20K+ serving — nothing above helps if chunk 2 still faults.

Serving 20K+ tokens reliably requires: PR-A, PR-B, PR-C, PR-D, PR-F, PR-G. PR-E is required for correct speculated long-form output (it is the defect that truncates long answers silently); PR-A and PR-B are the observability floor that makes the rest debuggable rather than lucky.

Open items I could not close from source in this budget (each needs one more look before the corresponding PR): the exact bail! inside the graphed k2 verify that refuses an active QSA selection (PR-E); the chunk-splitting arithmetic producing 8193 (PR-D); the NgramRowCache slot-count derivation (PR-D); the faulting kernel for chunk-2 (PR-G, needs sanitizer); and whether ssm_rollback_ring is enabled for qwen4_exp in this serve config, which determines whether Defect 1 also silently corrupts GDN state (PR-F).

</parameter> </function> </tool_call>