Qeravio
Canonical AI event

CUDA optimization speeds up prompt processing to 2600 t/s.

The optimization now supports n_expert_used = 10 and improves prompt processing to 2600 t/s on an RTX PRO 6000.

31 Aug 20261 verified claims1 sources1 observations
What happened

The official source reports this update: b10704: CUDA: use the fast mm_ids_helper path for any n_expert_used. CUDA: use the fast mm_ids_helper path for any n_expert_used ( #27978 ) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception padding 6 up to 8. Every other count fell back to the generic path, which walks the tokens one at a time with a warp reduction per token, for each of the n_expert blocks. The lane group only has to divide the warp, and the loop body already guards the padded lanes with iex generalizes to the next power of two.

Why it matters

This official update documents a development concerning b10704: CUDA: use the fast mm_ids_helper path for any n_expert_used. Its practical significance depends on the scope and evidence stated by the source.

What to watch next

Read the official source update and verify its stated scope, evidence and timing before acting on it.

Connected knowledge

Entities affected by this event

Evidence trail

Sources behind the event