Moore Threads Megatron-MUSA-Titan

Overview

Megatron-MUSA-Titan is Moore Threads' open-source training infrastructure project, announced in October 2026. It has one focus: pretraining and reinforcement learning for trillion-parameter and larger MoE models on clusters of 10,000 or more Moore Threads GPUs.

It is not another training framework but a layer of adaptation and optimisation on top of Megatron-LM, following an upstream-first approach. General algorithms and bug fixes go upstream, and the repository keeps only two kinds of change: large-scale training optimisation tuned for Moore Threads GPUs and cluster topology, and MUSA-specific workarounds that cannot yet land upstream. Because MUSA C is compatible with CUDA C and torch_musa integrates with PyTorch, Moore Threads says a configured environment can clone the official Megatron-LM, slime and miles repositories and train without modifying upstream code.

The optimisation list is specific: blockwise FP8 training, DeepEP acceleration over an asynchronous copy engine, expert-parallel communication overlapped with compute, activation offloading with virtual pipeline parallelism, RDMA-backed pipeline overlap through Mooncake, and distributed holistic trace analysis.

Key Features

• One trillion parameters on 10,000 GPUs: Moore Threads reports completing 1T MoE pretraining on clusters of 10,000 or more of its GPUs with high model FLOPs utilisation and stable training. It requires MT-TransformerEngine for optimised FP8 support.
• One base for pretraining and RL: Pretraining runs on Megatron, reinforcement learning on slime and miles, and rollouts through sglang_musa. Optimisation work is shared across both rather than duplicated.
• Blockwise FP8 and communication overlap: Supports blockwise FP8 at scale, uses DeepEP over the asynchronous copy engine to accelerate MoE token dispatch and combine, overlaps expert-parallel communication with compute, and overlaps pipeline communication using Mooncake RDMA.
• Activation offloading for memory: Combines virtual pipeline parallelism with a one-forward-one-backward schedule to offload activations, allowing deeper pipelines and larger batches inside the same memory budget.
• Diagnostics at cluster scale: Distributed holistic trace analysis reads PyTorch traces from every GPU in real time and reports operators, communication, pipeline bubble rates and straggler nodes. Moore Threads says a 1T MoE analysis across 10,000 GPUs completes in hours and has informed real optimisation work.
• Three ready recipes: The examples directory includes a Kimi-K2 1T MoE recipe along with DeepSeek-V3 and Llama3 pretraining scripts. Moore Threads notes these are starting points for cluster configuration, not one-command reproductions of the 10,000 GPU benchmark.

Use Cases

• Teams pretraining trillion-parameter MoE models on domestic GPU clusters.
• Projects with existing Megatron-LM scripts that want to move the substrate to Moore Threads hardware.
• Reinforcement learning post-training on slime or miles with SGLang rollouts.
• Diagnosing efficiency on large clusters with distributed trace analysis for bubbles and stragglers.
• Memory-constrained runs that trade activation offloading and virtual pipeline depth for batch size.
• Building a training configuration from the Kimi-K2 and DeepSeek-V3 recipes.

Pros

• 1T MoE pretraining completed on clusters of 10,000 or more Moore Threads GPUs.
• Upstream-first, so official Megatron-LM, slime and miles repositories run unmodified in a configured environment.
• Pretraining and reinforcement learning share one optimisation base.
• Blockwise FP8, DeepEP over ACE and RDMA pipeline overlap are all documented.
• Trace analysis covers a 1T MoE run on 10,000 GPUs in hours.
• Ships Kimi-K2, DeepSeek-V3 and Llama3 recipes.

Pricing

Open source and free to use or modify. Self-hosting requires a Moore Threads driver, the MUSA toolkit, PyTorch with torch_musa and MT-TransformerEngine, with hardware cost scaling to cluster size. The reinforcement learning path additionally needs a MUSA-compatible SGLang environment. Licence terms follow the repository LICENSE.

Summary

The weight of this project is not in lines of code but in documenting a path to trillion-parameter training on domestic GPUs. Upstream-first keeps migration cost bounded, and the official Megatron repository runs once the environment is set up. Note that a recipe is not a reproduction; Moore Threads says the scripts are starting points, so validate environment and version combinations on a small cluster before going large.

Version History

• Megatron-MUSA-Titan open-sourced: trillion-parameter MoE training infrastructure (2026-10-28): Moore Threads open-sourced Megatron-MUSA-Titan for 1T+ parameter MoE pretraining and reinforcement learning on clusters of 10,000 or more of its GPUs. Built on Megatron-LM with MUSA adaptation, it reports completed 1T MoE pretraining at that scale with stable training. Optimisations include blockwise FP8, DeepEP over an asynchronous copy engine, expert-parallel communication overlap, activation offloading under virtual pipeline parallelism, Mooncake RDMA pipeline overlap and distributed holistic trace analysis. Reinforcement learning supports slime and miles with sglang_musa rollouts. Recipes cover Kimi-K2 1T, DeepSeek-V3 and Llama3.