What is backend-specific kernel fusion?

Backend-specific kernel fusion combines several GPU operations that would normally run separately into one specialized kernel for a particular backend. Instead of launching one small program, storing its result, launching the next, and repeating that pattern, the backend runs a larger section of the computation as one unit.

That sounds like a minor implementation detail until a model reaches decode time: the stage where it generates tokens one at a time. At that point, a workload can contain many small, ordered operations. The arithmetic in each one may be cheap, while scheduling work and moving intermediate values through memory becomes expensive.

A GPU kernel is a program executed in parallel on the GPU. A dispatch is the act of launching that program over a chosen amount of work. Fusion reduces the number of those launches and, where the operations allow it, keeps intermediate values in fast on-chip storage rather than writing them out to global GPU memory and reading them back.

Plain English: Fusion does not make the model smarter or change its mathematical job. It changes how much administrative work and memory traffic the GPU must do between steps.

The “backend-specific” part matters. A portable computation graph may describe matrix operations, reductions, reshapes, and elementwise transforms. But an efficient fused implementation has to make choices that depend on the actual execution environment: the GPU API, shader compiler, available synchronization primitives, subgroup layout, memory hierarchy, and supported workgroup shapes. A fast Vulkan kernel is not automatically a fast CUDA, Metal, or ROCm kernel.

This is related to operator fusion, the broader idea of combining graph operations. Here, the emphasis is on the final backend implementation: not merely recognizing that two operations are adjacent, but compiling or hand-writing a kernel that takes advantage of a specific platform.

Why it matters now

The immediate example is llama.cpp’s September 7 b10844 release. Its Vulkan backend added fused DeepSeek-V4 hyper-connection operations named DSV4_HC_COMB, DSV4_HC_PRE, and DSV4_HC_POST. The release notes say that, on the cited gfx1151 configuration, an unfused Sinkhorn combine chain took about 32% of decode-operation time and involved roughly 16,000 dispatches per token. One fused combine dispatch replaces approximately 137 strictly ordered node executions per site. Release b10844

Those figures are not a universal Vulkan benchmark. They do show something more durable: inference speed can be constrained by the way a backend lowers and schedules a graph, not just by parameter count, quantization, or headline GPU throughput.

Graph lowering is the translation from a framework’s abstract computation graph into backend operations, kernels, memory layouts, and synchronization steps. Two runtimes can execute nominally the same model while doing materially different amounts of dispatching, memory movement, and synchronization. That is why “this model is slow on my GPU” is often an incomplete diagnosis.

The research literature describes fusion as a way to avoid the costs of chaining separate GPU functions while retaining intermediates in on-chip memory. It commonly distinguishes vertical fusion, which joins sequential operations, from horizontal fusion, which joins independent or parallel work. The Fused Kernel Library

Plain English: A model graph is a recipe. The backend decides whether that recipe becomes hundreds of tiny GPU jobs or a few larger jobs designed for a particular machine.

How a fused path works

Start with an unfused path. A runtime sees a sequence of operations with dependencies: operation B needs the output of A; C needs B; perhaps a reduction or normalization is interleaved. The straightforward backend implementation gives each operation its own kernel. Each kernel may read inputs from global memory, perform a small amount of work, write outputs back, then hand control to the next dispatch.

A fusion pass first asks whether a region can safely become one unit. The operations must have compatible dependencies, shapes, layouts, and numerical requirements. It then creates a combined implementation. Intermediate tensors that would have been materialized in global memory can become local variables, shared-memory values, or register-resident state. Registers are the smallest and fastest storage directly associated with an executing GPU lane, but they are limited.

The llama.cpp Vulkan implementation is a concrete illustration. Its release notes describe performing the full 20-iteration Sinkhorn combine in registers and using subgroup shuffle operations with a defined lane layout for the 4×4 combination matrix. A subgroup is a hardware-supported group of lanes that can exchange data efficiently under a defined execution model. Release b10844

The backend must then make less visible trade-offs. A larger kernel can reduce global-memory round trips and dispatch overhead, but it can consume more registers or shared memory. That may reduce occupancy: the amount of concurrent work the GPU can keep resident to hide memory latency. It can also introduce awkward synchronization, poor lane utilization, or specialization pressure for unusual tensor shapes.

The useful unit of reasoning therefore changes. Do not ask only whether primitive operation X is fast. Ask whether the whole scheduled region—its launch overhead, memory traffic, synchronization, register demand, and layout conversions—is better than the unfused alternative on this backend and workload.

What fusion is not

Fusion is not quantization. Quantization changes the numerical representation of weights or activations, typically to lower memory bandwidth or arithmetic cost. Fusion changes the execution boundary and scheduling. You can use both together, but a quantized model can still spend too much time in tiny dispatches.

It is not batching, either. Batching increases the amount of independent work processed together. Fusion combines dependent or closely related work into one kernel. Batching is often strongest for throughput; fusion can matter even at batch size one, where token-by-token latency is the concern.

It is also more than recording several kernels in one command buffer. Command-buffer batching may reduce CPU-side submission overhead, but separate kernels still run separately and may still materialize intermediate data. Fusion removes or combines the kernel boundaries themselves.

Finally, fusion is not the same as autotuning. Autotuning searches implementation settings such as tile dimensions or workgroup size. Fusion changes the candidate program: it decides that several operations should become one implementation. An autotuner can then search parameters for either form.

A practical engineering example

Imagine a local coding assistant that must generate responses interactively on a mixed fleet of integrated and consumer GPUs. A regression appears after adopting a model architecture with a new decode-time micrograph. End-to-end tokens per second drop, but memory use and model quantization are unchanged.

The tempting reaction is to swap quantization formats or declare the backend unsuitable. A better investigation starts by separating the execution path into observable parts. Capture traces for representative prompts, then compare dispatch count, time spent in the longest kernels, synchronization points, and memory traffic during prefill and decode. If decode contains thousands of small ordered dispatches per token, that is a signal to inspect the graph boundary rather than the model weights.

Suppose profiling identifies a repeated sequence of elementwise transforms, reductions, and a fixed-iteration normalization. The team can test whether the sequence has stable shapes and layouts in its production path. If it does, a backend-specific fused kernel may be worthwhile. The implementation should preserve an unfused fallback: a known-correct path used for unsupported devices, unusual shapes, and debugging.

Correctness comes before the speed chart. Compare the fused and reference paths across realistic batch sizes, shape boundaries, and numerical tolerances. Test the exact device families you operate. The llama.cpp release includes both global and per-operation controls for disabling the fused operations, plus evaluation cases across production iteration counts, batch sizes, and subgroup/workgroup boundaries. That is the right operational pattern: make the optimization independently switchable so a production regression can be bisected. Release b10844

Plain English: Keep the old route. A fast kernel you cannot disable, compare, or isolate is difficult to trust in production.

Where it breaks

Fusion has real costs. A kernel tailored to one subgroup width, lane mapping, and memory arrangement may need another implementation—or a fallback—on another GPU. Dynamic graphs and irregular shapes can make specialization too expensive. Some neighboring operations have incompatible layouts, and converting layouts inside a fused kernel can erase the expected gain.

Numerical behavior is another concern. Reordering work, changing reduction structure, or retaining values at a different precision can alter results. “Same intent” is not sufficient for numerical kernels; define tolerances and test the boundary cases that matter to your product.

Maintenance is the less glamorous risk. The fused-kernel literature identifies manual implementation, a limited set of precompiled combinations, and low-level GPU complexity as practical constraints. Model architectures, compiler behavior, and device support all change. The Fused Kernel Library A specialized fast path can become a long-lived compatibility obligation.

And fewer dispatches are not automatically faster. If fusion increases register pressure enough to lower occupancy, or creates a poorly balanced workgroup, a supposedly optimized kernel can lose. Measure the actual target device and workload rather than treating a dispatch-count reduction as proof.

What you can do this week

First, add backend-level profiling to one representative inference workload. Split results between prefill and decode, and record dispatch counts alongside latency. That makes “slow” diagnosable.

Second, find one expensive, stable micrograph rather than attempting whole-model fusion. Good candidates are repeated decode-time chains with fixed iteration counts and clear intermediate-memory traffic. Confirm that the benefit survives your real batch sizes and prompt lengths.

Third, define the compatibility contract before writing the fast path: supported devices, subgroup assumptions, shapes, layouts, numerical tolerance, and the conditions that select the fallback. Treat those as product behavior, not comments in shader code.

Finally, make fused and reference paths continuously comparable. Run both in evaluation on shape boundaries and supported hardware. Expose a narrow kill switch. Backend-specific kernel fusion earns its complexity only when the team can prove it is faster where it matters and can safely retreat when it is not.