A model can run and still be wrong
Moving an open-weight model from one inference stack to another often looks like a deployment task: convert the weights, load the file, select a backend, send a prompt, and inspect the answer. If tokens arrive, the migration appears complete.
That conclusion is too weak. A runtime can load a model, allocate its tensors, and generate fluent text while computing a subtly different function from the intended model. The durable engineering discipline for catching that problem is inference correctness across runtimes.
This is timely because llama.cpp’s September 6 b10829 release corrected gated delta-net normalization for Qwen3-Next. The release says the prior path differed from the reference formula in where it placed epsilon, a small numerical-stability constant, and changed it to match the reference placement inside the square root. The release notes describe the correction. Hugging Face had previously merged an l2-normalization change intended to align Qwen3-Next with Flash Linear Attention’s implementation. That earlier fix is documented in the pull request.
The important lesson is smaller than the release note: a model artifact is not the whole model in operation. The artifact, converter, graph construction, kernels, state handling, tokenizer, and sampling setup together produce observable behavior.
What is inference correctness?
Inference correctness asks whether a model implementation, conversion path, precision choice, and serving backend preserve the intended computation closely enough for a stated use. “Closely enough” matters. The term is not fully standardized.
Some teams mean numerical acceptance: compare a runtime with a reference and require logits, hidden states, or selected intermediate tensors to agree within defined tolerances. Other teams mean behavioral acceptance: require output distributions or task-level results to remain sufficiently similar. These criteria are related, but they answer different questions. A behavioral test may show that users are unlikely to notice a discrepancy; it may not reveal where a mathematical divergence began. Numerical comparison can localize a bug, but it can also reject a valid optimized implementation because floating-point arithmetic is not perfectly identical across devices.
Plain English: “The model runs” means the software did not crash. “Inference correct” means it performs the model’s intended computation to a level you defined and can defend.
This is not a demand for bit-for-bit identity. Floating-point addition can produce different results when an implementation changes operation order. Quantization—representing weights or activations with lower precision—also intentionally changes values. Correctness is therefore an acceptance contract: which reference, which inputs, which outputs, which precision, and which tolerance are acceptable for this deployment?
Why portability is not enough
Model portability answers a useful but narrower question: can this model be loaded and executed in another environment? Inference correctness asks whether that environment computes the intended function closely enough.
That distinction becomes practical at every layer. An architecture must be translated into tensor operations, layouts, data types, kernels, and sometimes graph fusions. A kernel is a low-level implementation of an operation on a CPU, GPU, or other accelerator. A fused kernel combines operations to reduce memory traffic or overhead. Either can be fast and still be semantically wrong.
Then come the less visible parts of the conformance surface: tokenization, positional encoding, cache updates, recurrent state, conversion rules, and generation settings. The KV cache, for example, stores attention state from prior tokens so generation does not recompute the full prompt each step. A fault in cache update logic may only appear after a particular prompt length or across a multi-turn session. A converter can transpose a tensor incorrectly, attach an incorrect scale during quantization, or map a model component to the wrong runtime operation. The process remains operational; the output is simply no longer reliable evidence that the intended model is running.
llama.cpp’s Qwen3-Next implementation itself includes architecture-specific graph construction for gated normalization, recurrent attention, reshaping, and model-state handling. The implementation is visible in its source. That is why a general “the runtime works” suite cannot establish conformance for every architecture and conversion path.
How to test it, step by step
Start by making a reference explicit. A reference is not necessarily a metaphysical ground truth; it is the implementation and configuration against which you will compare a candidate. Record its model artifact, code or runtime version, converter version where relevant, tokenizer, device, dtype, and generation configuration. References can contain bugs too—the ecosystem fixes around Qwen3-Next are a useful reminder—so retain enough provenance to revise the baseline later.
Next, build a small architecture-specific test corpus. Generic prompts are not enough. Inputs should exercise the operations most likely to differ: short and long sequences, generation after cache growth, unusual shapes or batch sizes where supported, and components such as normalization, positional encoding, attention, or recurrent state. The goal is not broad product evaluation. It is to make a particular computation reveal itself.
Compare at three levels, moving from diagnosis to user impact:
- Compare intermediate tensors when you can instrument both implementations. This finds the first layer or operation where values diverge.
- Compare logits, the raw scores produced before the runtime selects the next token, or compare hidden states. These tests are useful for numerical conformance before sampling amplifies tiny differences.
- Compare generated outputs and task results under fixed settings. This answers whether a remaining difference changes behavior that matters to the deployment.
Plain English: When two answers differ, do not start by arguing about the answer. First find out whether the internal computation diverged before the model chose its next token.
For each comparison, declare a tolerance policy rather than choosing one after a failure. The appropriate threshold depends on dtype, quantization, operation type, sequence length, and intended use. Full-precision CPU and GPU paths may deserve tight numerical checks. A heavily quantized path may need distributional or task-level acceptance in addition to broader numerical bounds. Do not quietly reuse the same threshold for all architectures and backends.
Finally, turn the case into differential testing: run the same controlled input through independent implementations and investigate unexpected differences. Execute it whenever you change a converter, introduce a fused operation, add a backend, alter a graph rewrite, or upgrade a runtime. Store the result with the exact implementation context. “It looked fine on a prompt” is not a regression record.
What it is not
Inference correctness is adjacent to several valuable practices, but it does not replace them.
It is not ordinary model evaluation. Evaluation asks whether a model performs well on a dataset or task. A runtime defect may degrade a rare capability that the benchmark misses; a good score may therefore coexist with a wrong implementation. Conversely, a tiny numerical mismatch may not alter a relevant task score. Evaluation detects outcomes; conformance tests investigate whether the implementation changed the computation.
It is not merely quantization validation. Quantization is one source of expected numerical variation. Inference correctness also covers unquantized errors in normalization, layout, broadcasting, conversion, state management, and backend semantics.
It is not compiler or kernel correctness alone. Validating an individual operation matters, but a model is the composition of thousands of operations plus architecture-specific state and conversion logic. A correct kernel used with an incorrect graph is still an incorrect inference path.
And it is not a performance, security, memory-safety, or reliability guarantee. A conformant backend can still be slow, leak resources, mishandle credentials around its service, or fail under production load. Those require their own test plans.
A realistic engineering example
Consider a hypothetical team serving a Qwen3-Next model in a reference stack, then adding a self-hosted llama.cpp path for a hardware or cost experiment. The team converts the same weights, sees coherent completions, and prepares a rollout.
A stronger rollout begins with a versioned test package. It pins the source artifact, converter and runtime versions, tokenizer, dtype, backend, and fixed decoding configuration. It contains prompts that force both short generation and longer stateful generation. Before enabling sampling, the team compares logits at selected positions. If it has instrumentation, it adds checks around the architecture-specific normalization and recurrent-state boundaries.
Suppose the generated text differs only on long prompts. The team does not label that a harmless “model personality” change. It first bisects the comparison: tokenizer IDs, initial graph outputs, normalization outputs, recurrent state, cache behavior, then logits. If divergence begins near a normalization operation, the investigation has a concrete target. If the logits remain close but sampled output diverges, the team can assess whether the acceptance criterion is numerical, distributional, or task-level for that feature.
The point is not that every deployment needs full tensor tracing. It is that the team chooses observability proportional to risk. A research workstation can accept more uncertainty than a production path that feeds downstream automation. In both cases, the deployable unit is the model plus its implementation context, not a weight file with a familiar name.
Where this discipline breaks down
There is no universal tolerance that separates correct from incorrect. Even exact-looking reference outputs can be contingent on hardware, dtype, and operation order. Quantized and fused paths may be valid without matching full precision exactly. Define the requirement before reading the result.
Coverage is also expensive. You cannot exhaustively test every device, driver, prompt length, batch shape, cache state, model revision, and conversion route. Prioritize the paths you ship, the architecture features they exercise, and changes with the greatest semantic risk. That leaves residual risk; a coverage report should say so plainly.
A second trap is treating the reference as infallible. The Qwen3-Next history shows why independent implementations can expose an issue, but agreement among implementations is not proof by itself. Preserve test inputs and baselines so they can be rerun when a reference changes.
Plain English: Conformance testing reduces uncertainty; it does not turn a complex inference stack into a proof.
What a senior engineer can do this week
Pick one model architecture and one alternative runtime already under consideration. Write down a one-page acceptance contract: the reference, supported backends, model artifact, converter, tokenizer, precision, sampling configuration, and what “close enough” means for your use case.
Then add a narrow regression suite before adding a broad benchmark. Include deterministic or tightly controlled checks of logits or hidden states where available, plus a few fixed end-to-end generations. Make at least one case long enough to exercise state or cache behavior. Keep failures attributable by storing the inputs and all version metadata.
Separate release gates. A converter upgrade, a new fused kernel, and a new quantization recipe should each trigger the relevant differential tests. Do not let a passing generic smoke test stand in for architecture-specific conformance.
Finally, publish the residual uncertainty with the artifact. State which backend and precision paths were compared, at what level, and which cases remain untested. That is not bureaucracy. It is the operational difference between “the model loaded” and “we know what we are serving.”