Multimodal Models · Speech Synthesis · Speech Recognition
SwanSphere vs Audio Interaction Model vs THUD: Streaming Audio, Compared
Three 2026 papers define real-time audio AI: SwanSphere streams spatial audio with a 0.21s first chunk, the Audio Interaction Model learns when to speak, and THUD catches models faking it.
The three jobs hidden inside “real-time audio”
“Streaming audio model” is used for at least three different things in 2026, and the papers behind each barely cite each other. SwanSphere is a generator: it produces four-channel first-order ambisonic audio (FOA) synced to a video or a text prompt, emitting its first chunk in 0.21 seconds instead of waiting for the whole clip. The Audio Interaction Model is a conversationalist: a large audio language model that listens continuously and makes a semantic decision about whether and when to respond, rather than waiting for a silence timer. When Vision Speaks for Sound is neither; it is a lie detector, showing that video models marketed on audio understanding often just read the picture, and supplying both the probe and a cheap fix.
The common thread is latency in the broad sense: not just milliseconds, but the gap between when information exists and when the model acts on it. A generator that needs the full clip, a listener that needs a clean turn boundary, and a benchmark that can be passed without touching the audio stream are all the same failure wearing different clothes.
Key numbers
| Dimension | SwanSphere | Audio Interaction Model | THUD (When Vision Speaks for Sound) |
|---|---|---|---|
| Job | Streaming spatial audio generation | Streaming audio LLM (perceive–decide–respond) | Audio-grounding probe + training fix |
| First-response latency | 0.21s first chunk (9.13s full clip) | Not published | n/a |
| Model scale | 1.09B parameters | Not disclosed | Qwen3-Omni-30B backbone for the fix |
| Training data | ~165K video-audio pairs (~458h) + ~1M pseudo-FOA curriculum samples | StreamAudio-2M: ~2.6M items, 7 abilities, 28 sub-tasks | ~10K samples for the two-stage fix |
| Headline quality metric | FD 120.28 vs OmniAudio 157.67 (video-to-spatial) | “Competitive” across 8 benchmarks, no per-benchmark scores | Audio hallucination on muted clips over 63% before the fix |
| Localization metric | SELD weighted cosine 0.63 vs OmniAudio 0.41; angular error 1.03° vs 1.27° | Proactive-Sound-Bench (new, same authors) | Shift 34.3% → 83.1%; VGGSync 36.8% → 56.4% |
| Subjective check | MOS-SQ 4.32 / MOS-AF 4.44 vs ground truth 4.60 / 4.58 | n/a | n/a |
| What the fix costs general skill | Single-source captions limit multi-source scenes | Proactivity claims lack external validation | Video-MME 70.1%, LVBench 52.1%, WorldSense 50.3% held steady |
Read the table for its gaps as much as its numbers. SwanSphere is the only one of the three with published latency and per-metric scores; the Audio Interaction Model deliberately reports breadth (8 benchmarks, 2.6M items) instead of depth, and its own authors call it work in progress; THUD’s numbers are about other models’ failures, not a product you can download. These are three different maturity levels, which is exactly why they answer different buying questions.
SwanSphere: generation with an interactive-grade first chunk
SwanSphere’s contribution is architectural and culinary. Architecturally it splits the pipeline into a Spatial Language Model that autoregressively predicts semantic embeddings from video features or text (the causal part that makes streaming possible), and a Local Diffusion Transformer that decodes those embeddings into continuous FOA waveforms via flow matching, in the latent space of a purpose-built 4-channel FOA-VAE. The training ideas do the flavor work: SVAC contrastive learning aligns visual motion with audio direction so a car panning left produces sound arriving from the left, and a multi-objective online DPO stage optimizes fidelity, spatial accuracy, and sync at once rather than a single reward.
The numbers that matter for a builder are the latency and the localization pair. 0.21 seconds to first chunk is genuinely interactive-grade (live avatars, AR, and game audio all become feasible), and it is achieved without quality collapse: FD 120.28 against OmniAudio’s 157.67 on video-to-spatial, KL 1.36 vs 1.93, angular error 1.03° vs 1.27°, and the most direct probe of “is the sound in the right place,” SELD weighted cosine similarity, at 0.63 vs 0.41. The text-conditioned variant holds the same gap (FD 142.80 vs 174.13), so the advantage is not an artifact of video conditioning. Human raters put it near ground truth (MOS-SQ 4.32 and MOS-AF 4.44 against 4.60 and 4.58).
The honest caveats from the paper itself: spatial captions mostly describe a single dominant sound source, so multi-source scenes (an orchestra with instruments in different directions) are under-modeled; much of the training signal comes from pseudo-FOA conversion of ordinary audio, not measured ambisonics; and the 5% held-out test set is same-distribution, so out-of-domain generalization is untested.
Audio Interaction Model: turn-taking as a learned decision
The Audio Interaction Model attacks a different latency: the social one. Most audio LLMs run a clean offline transaction (receive a finished clip, transcribe, reply), which forces them to wait for the end of the audio and gives them no concept of initiative. This paper inserts an explicit decision stage into the loop: as sound streams in, the model perceives the acoustic scene and any instruction, makes a semantic judgment about whether there is enough information to act and whether now is the moment, and only then commits to generating. The decision is driven by content, not a fixed silence threshold, which is what lets the same model stay silent, answer when addressed, or intervene proactively when the environment warrants it.
The engineering around the loop is half the contribution. SoundFlow covers streaming-native data construction (train on inputs shaped like live streams), comprehension-aware training, and asynchronous inference so the model can keep listening while it speaks. The data foundation, StreamAudio-2M, is ~2.6M items spanning 7 core abilities and 28 sub-tasks (dialogue, real-time ASR, instruction following, environmental reaction), which is what makes a single unified model plausible instead of a pile of per-task systems. The authors also introduce Proactive-Sound-Bench to measure proactive audio intervention, the thing turn-based suites do not test.
Now the honest read, which the paper itself supports: the abstract claims “competitive” performance across 8 benchmarks, not state of the art, and publishes no latency figures or per-benchmark scores, so you cannot compute what the streaming design costs or buys versus an offline model. Proactive intervention is double-edged (a model that can jump in unprompted can also interrupt at the wrong moment), and Proactive-Sound-Bench comes from the same authors, so the proactivity claims await external validation. Treat this as the recipe and the corpus mattering more than any single number, and wait for released weights before building on it.
THUD: the mute test for “we understand audio”
The third paper explains why you should be skeptical of both of the above, and of every demo where a video model “uses” audio. Its finding is an audio-visual Clever Hans effect: mute the soundtrack and top models still report hearing sound more than 63% of the time, because they are reading the scene. The shortcut shows up in open models (Qwen3-Omni, MiniCPM-o, Ming-flash-omni) and in proprietary Gemini- and GPT-class systems alike, which means standard audio-video benchmarks have been grading lip-reading.
THUD’s fix is methodological, and cheap to copy. It holds the video fixed and edits only the audio with three counterfactuals: Shift (displace the track in time, testing sync detection), Mute (silence, testing absence detection), and Swap (mismatched track, testing cross-modal consistency). A model leaning on the picture scores at chance on the audio question; that is exactly what the paper observes. The repair recipe runs SFT on intervention-derived data, then DPO mixing counterfactual temporal preference pairs with general video preferences, about 10K samples on a Qwen3-Omni-30B backbone in total. The result: +28 percentage points averaged across the three THUD dimensions, with the largest jump on Shift (34.3% → 83.1%), out-of-distribution gains on VGGSync (36.8% → 56.4%), and general video scores held steady (Video-MME 70.1%, LVBench 52.1%, WorldSense 50.3%, DailyOmni 67.9%).
Two of the paper’s own caveats deserve to travel with the number. The 28-point average folds in all three dimensions, and the profile is uneven: temporal sync is the easiest behavior to teach with paired data, while Mute and Swap, the harder presence and consistency checks, improve less. And the fix is validated on one open backbone only; the proprietary models it diagnoses cannot be repaired from outside.
When to use which
- You are building live avatar, AR, or game audio and need sound now: SwanSphere is the only one of the three with a measured, interactive-grade latency number and per-metric quality scores. Its constraints (single dominant source, pseudo-FOA training data) define where it will not shine.
- You are building a voice assistant and the hard part is interaction feel, not accuracy: the perceive–decide–respond loop is the right abstraction, and StreamAudio-2M is the largest public bet on streaming-native audio training data. Budget for unvalidated claims: no latency or per-benchmark numbers are public yet.
- You are evaluating or buying any model that advertises audio understanding: run THUD first. Muting a clip and re-asking is a five-minute sanity check that has caught flagship models; the 10K-sample fix shows gaps are closable, so a failed mute test should trigger a data conversation, not a write-off.
The deeper point is that these works compose. A system that generates spatial audio in 0.21s, decides semantically when to speak, and is verified with counterfactual audio probes is a very different product from one that demos well on a scripted clip, and the 2026 numbers above are currently the best public evidence for each leg.
Limits and open questions
All three papers share one blind spot: none of them measures the composition. SwanSphere’s latency was not tested in a live conversational loop; the Audio Interaction Model’s turn-taking was not tested with generated spatial audio; THUD’s probes target video understanding models, not audio generators or audio LLMs, so we do not know whether a streaming listener has its own Clever Hans problem. Cross-testing (THUD-style counterfactuals applied to audio generators and conversational listeners) is the obvious open experiment. The other open question is data realism: SwanSphere leans on pseudo-FOA conversion, the Audio Interaction Model’s corpus is broad but new, and THUD’s counterfactuals are deliberately synthetic. Real acoustic environments are messier than all three.
FAQ
What is a streaming audio model?
In 2026 usage it means one of three things: a generator that emits audio chunk-by-chunk instead of rendering whole clips (SwanSphere, 0.21s to first chunk), a listener that processes sound continuously and decides when to respond (the Audio Interaction Model’s perceive–decide–respond loop), or a verification method that checks whether audio is actually being used (THUD’s counterfactual probes). The page above compares all three.
How fast can a streaming audio model respond?
The only published first-chunk latency among these papers is SwanSphere’s 0.21 seconds, with full-clip generation taking 9.13 seconds. The Audio Interaction Model paper does not publish latency figures, which its own authors flag as an open gap; its contribution is deciding whether and when to respond rather than raw speed.
Which streaming audio model has the best quality?
For generation, SwanSphere beats its OmniAudio baseline on every reported metric: FD 120.28 vs 157.67 (video-to-spatial), KL 1.36 vs 1.93, angular error 1.03° vs 1.27°, and SELD cosine 0.63 vs 0.41, with human MOS ratings near ground truth. For listening, the Audio Interaction Model reports “competitive” results across 8 benchmarks but releases no per-benchmark scores, so a ranking is not currently possible from public numbers.
How do I test whether a video model really uses audio?
Mute the clip and re-ask. In the THUD study, models reported hearing sound on over 63% of muted clips, exposing visual shortcutting. The full THUD protocol adds Shift (time-displace the audio) and Swap (substitute a mismatched track); the 10K-sample fix lifted average probe scores by 28 points, so a failed mute test points to a training-data gap, not a dead end.
Can the Audio Interaction Model decide when to speak on its own?
Yes; that is its core design. A semantic decision stage between perception and generation lets it stay silent, answer when addressed, or intervene proactively when a sound event warrants it, replacing the fixed silence timer used by conventional voice interfaces. The caution: Proactive-Sound-Bench, the benchmark measuring this, is from the same authors, so independent validation is still pending.