Overview
MolmoMotion is a **language-guided 3D motion prediction model** open-sourced by Allen AI (Allen Institute for Artificial Intelligence) on 2026-06-18, based on Allen AI's own **Molmo 2 visual backbone**. **Input = video frames + 3D point annotations + text instructions**, **output = 3D trajectory prediction for the next few seconds**. Two variants are provided: **autoregressive version** (generates step by step over time, slower but more accurate) + **flow matching version** (generates the complete trajectory at once, faster).
Accompanying release: **MolmoMotion-1M dataset** — **1.16 million real videos** with 3D point annotations and text instructions, the largest publicly available dataset for "language-guided 3D motion" to date. Models, datasets, and benchmarks are **fully open-sourced** — continuing Allen AI's consistent style of **pure academia + complete openness**.
Such models directly serve: **robot trajectory planning** (enabling robots to move according to natural language instructions), **autonomous driving prediction** (predicting future trajectories of other vehicles/pedestrians), **motion anchoring in AR/VR**, and **physical consistency in video content generation**.
Key Features
- Language-Guided 3D Motion Prediction: Video + 3D points + text instructions → 3D trajectory prediction for the next few seconds
- Molmo 2 Visual Backbone: Based on Allen AI's own Molmo 2 multimodal backbone, with top-tier video frame understanding
- Autoregressive + Flow Matching Dual Variants: Autoregressive version for high accuracy, flow matching version for fast speed, choose based on scenario
- MolmoMotion-1M Dataset: Accompanying open-source dataset of 1.16 million videos + 3D annotations + text instructions, the largest public dataset in the same field
- Fully Open-Source: Models/datasets/benchmarks all open-sourced, pure academia + openness
- From Allen AI: Non-profit research institute founded by Paul Allen under AI2, with top academic rigor
Use Cases
- Robot trajectory planning: natural language instructions → 3D motion
- Autonomous driving prediction: future trajectory estimation of other vehicles/pedestrians
- Object motion anchoring in AR/VR
- Physical consistency constraints for video generation (as discriminator or post-processing)
- Academic research: SOTA baseline for 3D motion prediction
- Animation/gaming: natural language-driven character motion generation
Pros
- MolmoMotion-1M with 1.16 million videos is the largest public dataset in the field
- Autoregressive + flow matching dual variants cover different scenarios
- Allen AI's full open-source is an academic conscience
- Molmo 2 backbone ensures strong visual understanding
- Directly beneficial for robotics/autonomous driving/AR three major directions
Pricing
**Fully open-source + free** (GitHub + HuggingFace). **Model inference/training compute** costs borne by user. **MolmoMotion-1M dataset** freely downloadable for academic and commercial research.
Summary
MolmoMotion is a key open-source achievement in the 3D motion prediction field as of 2026-06-18 — **language-guided + dual variants + 1.16 million dataset + fully open-source** makes it the SOTA baseline for robotics, autonomous driving, and AR/VR teams. If you work on physical world AI (especially scenarios involving predicting future trajectories), MolmoMotion is almost a no-brainer tool to evaluate; if you only work on 2D vision/text tasks, you can skip it; if you work on 3D generation/reconstruction (rather than motion prediction), tools like Meshy, ABot-Earth, Genie, etc. remain the main focus.