Overview
Real-time audio and video conversation used to mean wiring three models together: speech recognition, vision understanding and speech synthesis. Every hop loses some information and adds some latency. The harder part is deciding when to speak, which the cascaded setup hands to a small module that only listens for silence. Guess early and it talks over you; guess late and it sits there after you have finished.
SeedRealtime 1.0 takes a different route. Volcano Engine put the Doubao real-time audio-video call model into general API availability on Nov 19, with sound, picture, timing and expression handled inside a single end-to-end model. Vision now takes part in deciding when to speak, not only in deciding what to say. That sounds like a small shift and it changes the experience completely.
Two demos from the launch show the difference. In an airport, with strangers chatting in the background, the model separates the speech addressed to it, pulls the departure board it saw a moment earlier when asked, then reads a sign and walks the traveller to the ride-hailing pickup point. At a group dinner it works out who is who around the table, remembers each person's identity and tailors travel plans to what each one wants. A cascaded pipeline can attempt these tasks, but once a reference points into the picture, such as this or over there, the visual context has already been dropped somewhere in the chain.
The second difference is who starts. The model keeps watching instead of waiting to be asked, and speaks up when a target appears, the scene changes or a task step arrives. In the coffee demo, the user pours whole beans into the portafilter and the model points out the beans need grinding first; after extraction it reads the crema and yield and suggests a shorter shot. In the study demo it follows the page turns, chimes in when the requested chapter appears and recites the learning rate, momentum and weight decay settings alongside.
Noise tolerance decides whether any of this survives in the real world. The official example is a home scene where a mother is busy elsewhere and asks the model to teach her daughter animal names in English, while the father is on a call in the background; the model keeps following the girl's finger, correcting pronunciation and making up example sentences. Volcano Engine's claim is that rhythm problems in audio-video conversation drop by half compared with a pipeline of ASR, VLM and TTS, and that a single call completes smoothly far more often. That half is an internal comparison without a published test set or reproduction setup.
Pricing runs by the minute in two tiers. Audio only costs 0.2 yuan per minute; sending video frames or images costs 0.4 yuan per minute, which already covers the audio. Above 0.5 fps of video frames the call duration is the video billing duration; below that, each image counts as two seconds. Packages lower the rate: 19.6 yuan for 100 minutes, 9,600 yuan for 50,000 minutes, 94,000 yuan for 500,000 minutes and 360,000 yuan for 2 million minutes, working out to 0.18 to 0.196 yuan per minute. Package deduction differs by mode as well, 1:1 for voice and 1:2 for video. The default ceiling is five concurrent calls, with more available by arrangement.
On the integration side there are Android and iOS SDK documents, and the model already ships inside the Doubao app, so an ordinary user updates, taps the call button and turns on the camera. That route lets a team test the feel with consumers before deciding whether the API is worth paying for.
Three numbers are missing from the launch: parameter count, context or maximum session length, and end-to-end latency in milliseconds. Those are the first three things anyone should check when evaluating a real-time voice product, especially the last, where the only figure offered is the halved rhythm problem rather than an absolute value. The product is also positioned for calls, not general visual reasoning, so long-video understanding or complex chart questions are a mismatch.
Three kinds of team fit. Hardware and companion-device makers that need conversation to hold up in noisy rooms; live support and remote guide products that need the model to watch and prompt on its own; and anyone already running a three-stage pipeline who wants one rebuild to replace the stitching.
Key Features
- One end-to-end model: Sound, picture, timing and expression are aligned inside a single model, removing the latency stacking and information loss of chaining ASR, VLM and TTS.
- Vision informs turn taking: Deciding whether the user has finished no longer depends on silence alone, so the model interrupts less and responds sooner.
- Continuous sensing and unprompted speech: It speaks up when a target appears, the scene changes or a task step arrives, so users stop having to ask at every step.
- Noise tolerance: It separates nearby chatter and background noise from speech addressed to it, holding focus in airports and busy homes without false triggers.
- Speaker identity in groups: Vision and voice together distinguish who is talking, remember each person's identity and answer each request separately.
- SDKs and a consumer app: Android and iOS SDK documents ship alongside a call feature already live in the Doubao app, so the experience can be tried without writing code.
Use Cases
- Real-time video calls on hardware and companion devices
- Remote support and guided tours that need the model to watch and prompt
- Step-by-step equipment operation with corrections drawn from what the camera sees
- Children's education and language practice in noisy rooms
- Instant questions in information-dense places such as airports and museums
Pros
- Vision feeds the decision of when to speak, a layer where cascaded systems can only guess from silence
- Unprompted reminders turn the interaction from question and answer into ongoing collaboration
- Per-minute billing with package discounts keeps costs predictable
- Already inside the Doubao app, so the feel can be checked before writing any code
Pricing
Postpaid by the minute: 0.2 yuan per minute for audio only, 0.4 yuan per minute when video frames or images are sent, with the audio included in the video tier. Above 0.5 fps of frames the call duration is the video billing duration; below that, each image counts as two seconds. Packages are 19.6 yuan for 100 minutes (0.196 per minute), 9,600 yuan for 50,000 minutes (0.192), 94,000 yuan for 500,000 minutes (0.188) and 360,000 yuan for 2 million minutes (0.18); deduction runs 1:1 for voice and 1:2 for video. The default ceiling is five concurrent calls, with expansion quoted separately.
Summary
It folds a three-stage pipeline into one model, and the practical change is that vision now shapes when the model speaks. Billing runs by the minute, with video at 0.4 yuan, which is above pure speech APIs and buys back the engineering and latency of stitching three services. Neither end-to-end latency nor parameter count nor context limits were published, so plan a load test before committing.
Version History
- General API availability (2026-11-19): Volcano Engine made the SeedRealtime 1.0 API generally available, with sound, picture, timing and expression unified in one end-to-end model for real-time seeing, listening and speaking that replaces a chain of ASR, VLM and TTS. The company reports half as many rhythm problems in audio-video conversation compared with cascaded systems, plus continuous environmental awareness and unprompted speech, aimed at group conversation, equipment operation, study assistance and live support. Billing is postpaid by the minute at 0.2 yuan for audio and 0.4 yuan with video, five concurrent calls by default, with Android and iOS SDKs.