InfoReleasesHacker News · community
DeepSeek just open-sourced its inference engine
DeepSeek open-sourced DeepInfer, the inference engine behind its production services, under Apache 2.0. The release supports offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with features like optimized kernels, asynchronous scheduling, batch-invariant and deterministic sampling, and structured outputs. Developers can self-host the engine, but a KV cache system, prefill/decode disaggregation, and load balancing are not included.
Read the full story on github.com
What to doIf you want to self-host DeepSeek models, try DeepInfer from source (Python 3.12, CUDA 13 or CANN 9.2) and plan to supply your own KV cache and load balancing.
- Source
- github.com