What happened this week
A full-duplex model listens and speaks at the same time, so it produces output every few tens of milliseconds even while the user is talking. Its memory of the conversation keeps filling during silences too, and a new benchmark shows what happens after a few minutes: the models start repeating themselves, go quiet, or produce gibberish. The same week, ElevenLabs and Mistral both shipped voice agent products in which one model name stands for the whole stack.
Long conversations
FD-LHB, the Full-Duplex Long-Horizon Benchmark from KAIST and Kyung Hee University, runs five open full-duplex models through long conversations: 40 replayed CANDOR conversations up to 60 minutes and 40 live sessions of up to 20 minutes with a large language model (LLM) as the other speaker. Every model eventually degrades or runs out of context. Moshi and PersonaPlex keep a sliding window of 3,000 frames at 12.5 Hz, which fills after four minutes. Raon-SpeechChat, BayLing-Duplex and Lychee-FD keep the full history and run out after about 5, 17.5 and 10.9 minutes.
The authors trace the sliding-window failures to the window dropping its very first entry in the key-value (KV) cache, the model's running memory of the conversation, which they call Initialization-Anchor Eviction. Two fixes that need no retraining, pinning that first entry and a repetition penalty that also guards against long silence, cut gibberish from 100% to 0%, and repetition from 65% to 0% for Moshi and from 90% to 2.5% for PersonaPlex. Prolonged silence falls from 100% to 7.5% for Moshi but only from 85% to 47.5% for PersonaPlex. The fixes run in real time on replays of up to an hour. For a team running an open full-duplex model in production, a call that goes past four minutes is the case to test first. Code is promised on publication; the CANDOR audio itself will not be redistributed.
Voice agents and tools
Heartbeat Thinking, from Ant International, starts reasoning and tool calls while the user is still speaking and lets them finish even as the transcript keeps changing, instead of cancelling and restarting at each update. On FDBench v3, speculative tool coverage rises from 34.67% to 50.67% and mean final-response latency falls from 6.90 s to 4.39 s, 36.4% lower. On SpeechFCEval, final accuracy is 66.85 against 67.95 for the windowed baseline, so the authors call the effect on task quality benchmark-dependent. In practice, a caller asking for a booking hears an answer about two and a half seconds sooner.
Training data
Training a full-duplex model needs conversations with each speaker on a separate, consistent track, and most recordings mix both voices on one channel. Beyond Two Tracks, from CUHK-Shenzhen, the University of Melbourne and the company Lulula, adds a 1.84M-parameter router that assigns separated voices to fixed speaker identities before a frozen generator rebuilds the audio. On 25-minute remote conversations, time spent on the wrong track falls from 43.9% to 4.1%, against 6.2% for waveform re-identification, and fixed-track activity F1 rises from 0.46 to 0.84. The router transfers to face-to-face and telephone recordings, where a verification step brings wrong-track time to 0.9 to 1.7%. Only audio examples are public.
Models and platforms
ElevenLabs launched Eleven Realtime on October 20: two models, v1 Mini and v1 Max, each a complete voice agent where ElevenLabs picks the speech recognition, LLM and text-to-speech (TTS) settings. The API is compatible with OpenAI's Realtime API, so an existing client changes only the endpoint and model name. In ElevenLabs-reported results from Artificial Analysis, v1 Max answers 98.8% of Big Bench Audio questions and starts speaking in 0.57 s on average, against 75.9% and 0.62 s for Mini. Mini scores 95.8% on Full-Duplex-Bench and Max 94.1%; on tool-use tasks Max completes 87.1% and Mini 60.3%.
Mistral released Voxtral 3 on October 22: batch and realtime transcription models, a TTS model, and a voice agent API built on LiveKit at $0.04 per minute with Mistral Small 4 or $0.08 with a larger model. The realtime transcription model costs $0.008 per minute, reports under 500 ms latency and predicts end of turn from acoustic and semantic cues. Mistral reports a 4.0 to 4.1% macro word error rate (WER) across 45 FLEURS languages for Voxtral Small Transcribe 3. No open weights are mentioned.
On October 23, going by the team lead's post on X (the page itself is undated), Block's small AI lab announced Dessa, a full-duplex speech-to-speech model it says was trained from scratch on human conversation only, with no text and no pretrained models, on a single NVIDIA GB200 NVL72. Block says it backchannels, times its own interruptions and runs faster than real time on a MacBook Pro M4 Max or better. The page gives no parameter count, data size or benchmark. Access is a waitlist, plus early access for AI safety researchers.
Gradium released Phonon, new to Hugging Face: on-device streaming TTS at 31M parameters for English under Apache-2.0, and 40M-parameter models for English, French, German, Spanish and Portuguese under CC BY-NC 4.0. It runs on a Rust runtime with no PyTorch, including in the browser and on Snapdragon NPUs, and builds on Kyutai's Pocket TTS and Mimi codec. In Gradium's own listening test it scores an Elo of 2098, against 2043 for Kokoro. Kyutai's own Pocket TTS v4.0.0 added English models of 18M, 7M and 2M parameters the same week.
Also this week: Deepgram's Nova-3 Multilingual went from 10 to 55 languages with a new language_hint parameter (October 23). LiveKit Agents 1.8.9 evaluates named decisions through LiveKit Inference and adds a ThunderPhone speech-to-speech plugin; Pipecat 1.14.0 can use NVIDIA Nemotron Omni in place of a separate speech-to-text (STT) step.
On Hugging Face, the week was small on-device TTS (Phonon, Pocket TTS, 33.6M-parameter Silencio) and compressed Parakeet ASR (automatic speech recognition), with Turkish appearing twice. Licences split: Apache-2.0 for the smallest models, non-commercial or custom terms for larger ones.