Qeravio
Canonical AI event

CUDA kernel fusion reduces kernel launches by 96 per batch on Qwen3.8-27B

Fusion of RMS_NORM and SCALE reduces kernel launches by 96 per microbatch on Qwen3.8-27B. Performance improves by up to 4.8% in draft-mtp speculative decoding.

26 Sept 20261 verified claims1 sources1 observations
What happened

The official source reports this update: b11177: CUDA: fuse RMS_NORM + SCALE into one kernel. CUDA: fuse RMS_NORM + SCALE into one kernel ( #29393 ) #28068 builds the GDN q/k l2norm as ggml_scale(ggml_rms_norm(x, eps/n), 1/sqrt(n)). This adds 2 SCALE nodes per GDN layer, 96 extra kernel launches per ubatch on Qwen3.8-27B (48 GDN layers). The extra kernels take no measurable GPU time, but each launch has a host/driver cost. It is small with plain batch processing and about 10x larger with draft-mtp speculative decoding. rms_norm_f32 gets a do_scale flag, the same pattern as do_multiply/do_add, so the fused path shares the kernel, the reduction and the launcher.

Why it matters

This official update documents a development concerning b11177: CUDA: fuse RMS_NORM + SCALE into one kernel. Its practical significance depends on the scope and evidence stated by the source.

What to watch next

Read the official source update and verify its stated scope, evidence and timing before acting on it.

Connected knowledge

Entities affected by this event

Continue this topic
Evidence trail

Sources behind the event