HPC-Ops

Overview

HPC-Ops is a high-performance operator library open-sourced by Tencent Hunyuan, designed specifically for LLM inference scenarios and now integrated into the SGLang inference framework. It provides core operators such as Dynamic Attention, Router GEMM, and Fused MoE, which are deeply optimized for the three hot paths in MoE model serving, significantly reducing Time to First Token (TPOT) in real production environments, with a maximum reduction of 48.8%. HPC-Ops has been deployed in Tencent's large-scale production environment and is key infrastructure for Hunyuan online services.

Key Features

• High-Performance Attention Operator: Achieves load balancing for mixed-length decoding scenarios, supports real-time KV cache scheduling, and integrates attention prologue logic to reduce memory access overhead and improve inference efficiency for long sequences.
• Precision-Aware Router GEMM Optimization: Adopts a precision-aware BF16 computation scheme to balance routing accuracy and throughput in router matrix multiplication, avoiding expert selection bias caused by low precision while maintaining high computational efficiency.
• Fused MoE Low-Latency Pipeline: Designs a fusion around the redundant overhead of small expert GEMMs, building a latency-oriented MoE execution pipeline to reduce kernel launches and intermediate data movement, significantly lowering end-to-end inference latency.

Use Cases

• Large-scale MoE model online inference services
• LLM decoding scenarios with mixed-length requests
• Real-time AI applications sensitive to TPOT latency

Pros

• Reduces TPOT by up to 48.8%, significantly improving inference response speed
• Open-source and validated in Tencent's production environment, with high stability
• Seamless integration with SGLang for easy deployment
• Comprehensively optimized for the three hot paths of MoE, covering the complete inference chain

Pricing

HPC-Ops is an open-source project, free to use with no commercial licensing fees. Users can obtain the source code from the official repository and deploy it themselves.

Summary

HPC-Ops is an LLM inference operator library open-sourced by Tencent Hunyuan. Through high-performance Attention, Router GEMM, and Fused MoE operators, it achieves significant TPOT reduction under the SGLang framework, especially suitable for large-scale production deployment of MoE models. Its open-source, free, production-validated, and deeply integrated features make it a preferred solution for low-latency AI inference.