Overview
BigMac is a new paradigm for multimodal large model training open-sourced by the dots infra team of Xiaohongshu, aiming to break through the Pareto frontier where computational efficiency and memory usage are difficult to balance in traditional training. It targets the heterogeneous architecture of multimodal large language models (MLLMs), consisting of modality encoders, LLM backbone, and modality generators, and proposes a dependency-safe nested pipeline technology. This technology orderly embeds encoder and generator computations without disrupting the LLM execution order, thereby efficiently achieving multimodal pipeline training. Compared to traditional pipeline parallelism solutions, BigMac decouples global scheduling from runtime execution in engineering, and provides rich interfaces and toolchains, significantly reducing the cost of model integration and system tuning.
Key Features
- Dependency-Safe Nested Pipeline: Using the mature LLM pipeline as the backbone, it orderly embeds encoder and generator computations without increasing LLM pipeline bubbles or keeping activation memory bounded, achieving efficient parallelism for multimodal training.
- Decoupled Global Scheduling and Runtime Execution: In engineering implementation, scheduling logic and execution logic are separated, enhancing system flexibility and maintainability, facilitating optimization for different models and hardware.
- Low-Cost Model Integration and Tuning Toolchain: A series of interfaces and tools are designed to lower the barrier for integrating new models into the BigMac framework and simplify the complexity of system tuning.
Use Cases
- Training and fine-tuning of multimodal large language models
- Development of AI systems that need to simultaneously process multiple data types such as text, images, and audio
- Model training with strict requirements on memory and computational efficiency in large-scale distributed training scenarios
Pros
- Training speed improved by 1.08 to 1.9 times compared to baseline
- Stable memory usage when global batch size increases
- Open-sourced and used as a core component for dots multimodal model training in production
- Effectively solves pipeline bubble and memory bottleneck issues caused by heterogeneous multimodal modules
Pricing
Open source and free, following the open-source license (please refer to the GitHub repository for the specific license)
Summary
BigMac is an efficient pipeline parallelism framework focused on multimodal large model training. Through dependency-safe nested pipeline design, it significantly improves training speed while keeping memory bounded, and provides an easy-to-use toolchain, suitable for training and deployment of large-scale multimodal AI models.