Overview
MolmoMotion is a language-guided 3D motion prediction model open-sourced by Allen AI (Allen Institute for Artificial Intelligence) on 2026-06-18, based on Allen AI's own Molmo 2 visual backbone. Input = video frames + 3D point annotations + text instructions, output = 3D trajectory prediction for the next few seconds. Two variants are provided: autoregressive version (generates step by step over time, slower but more accurate) + flow matching version (generates the complete trajectory at once, faster).
Accompanying release: MolmoMotion-1M dataset — 1.16 million real videos with 3D point annotations and text instructions, the largest publicly available dataset for "language-guided 3D motion" to date. Models, datasets, and benchmarks are fully open-sourced — continuing Allen AI's consistent style of pure academia + complete openness.
Such models directly serve: robot trajectory planning (enabling robots to move according to natural language instructions), autonomous driving prediction (predicting future trajectories of other vehicles/pedestrians), motion anchoring in AR/VR, and physical consistency in video content generation.
Key Features
- Language-Guided 3D Motion Prediction: Video + 3D points + text instructions → 3D trajectory prediction for the next few seconds
- Molmo 2 Visual Backbone: Based on Allen AI's own Molmo 2 multimodal backbone, with top-tier video frame understanding
- Autoregressive + Flow Matching Dual Variants: Autoregressive version for high accuracy, flow matching version for fast speed, choose based on scenario
- MolmoMotion-1M Dataset: Accompanying open-source dataset of 1.16 million videos + 3D annotations + text instructions, the largest public dataset in the same field
- Fully Open-Source: Models/datasets/benchmarks all open-sourced, pure academia + openness
- From Allen AI: Non-profit research institute founded by Paul Allen under AI2, with top academic rigor
Use Cases
- Robot trajectory planning: natural language instructions → 3D motion
- Autonomous driving prediction: future trajectory estimation of other vehicles/pedestrians
- Object motion anchoring in AR/VR
- Physical consistency constraints for video generation (as discriminator or post-processing)
- Academic research: SOTA baseline for 3D motion prediction
- Animation/gaming: natural language-driven character motion generation
Pros
- MolmoMotion-1M with 1.16 million videos is the largest public dataset in the field
- Autoregressive + flow matching dual variants cover different scenarios
- Allen AI's full open-source is an academic conscience
- Molmo 2 backbone ensures strong visual understanding
- Directly beneficial for robotics/autonomous driving/AR three major directions
Pricing
Fully open-source + free (GitHub + HuggingFace). Model inference/training compute costs borne by user. MolmoMotion-1M dataset freely downloadable for academic and commercial research.
Summary
MolmoMotion is a key open-source achievement in the 3D motion prediction field as of 2026-06-18 — language-guided + dual variants + 1.16 million dataset + fully open-source makes it the SOTA baseline for robotics, autonomous driving, and AR/VR teams. If you work on physical world AI (especially scenarios involving predicting future trajectories), MolmoMotion is almost a no-brainer tool to evaluate; if you only work on 2D vision/text tasks, you can skip it; if you work on 3D generation/reconstruction (rather than motion prediction), tools like Meshy, ABot-Earth, Genie, etc. remain the main focus.