AWS Speeds Up Mixture-of-Experts Models with Kernel Tricks

AWS engineers have squeezed serious performance gains out of MoE models running on Amazon SageMaker AI by optimizing GPU kernels for H200 hardware. Their tweaks—including Swap-AB for better WGMMA usage, Triton backend routing for FP8 quantization, and ditching unnecessary JIT recompilation—deliver up to 10% higher throughput and 9% faster Time to First Token.