Research & academia
Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
Dropless MoE keeps every routed token, but its uneven expert loads create ragged computation and communication bottlenecks. The article reports how JAX and Transformer Engine address them and the resulting DeepSeek-V3 performance.
Why it matters: For dropless MoE, avoiding dropped or padded tokens is not enough: kernels must handle variable expert loads efficiently. The reported implementation offers a concrete JAX path and metrics to track when validating and scaling a training run.