NVIDIA and Princeton's Dao AI Lab introduced ReplaySSM, which represents SSM state as a checkpoint plus a buffer of recent inputs, cutting state memory traffic roughly in half and speeding SSM decoding kernels 1.46x–1.80x at batch size 256 on Blackwell Ultra B300 GPUs. It also simplifies speculative decoding rollback (1.85x–2.10x higher speculative throughput on 120B-class models) and supports low-precision state reads with an FP32 or FP16 recurrence checkpoint, keeping three of four evaluated models within about 2 percentage points of FP32 accuracy on GPQA-Diamond. ReplaySSM is integrated in…
Read the full story on developer.nvidia.com
What to doIf you serve hybrid SSM-attention models, try ReplaySSM via the integrated TensorRT-LLM, vLLM, SGLang, or FlashInfer builds and the GitHub code for faster decoding and speculative throughput.
- Provider
- NVIDIA
- Source
- developer.nvidia.com