Full-Duplex Speech-to-Speech Models vs Cascaded Voice AI
How full-duplex speech-to-speech models like Haru-4B listen and talk at once, and how they differ from cascaded voice assistants with 2 to 3 second delays.
For Customer Support, Marketing and Design

What is a full-duplex speech-to-speech model?
A full-duplex speech-to-speech model is a single AI model that takes in audio and produces audio, and can do both at the same moment. "Full-duplex" is the phone-call kind of conversation, where both sides can talk and listen at once, as opposed to a walkie-talkie where you take strict turns. Instead of converting your voice to text, thinking, and converting text back to voice, the model works on the sound directly, in tiny slices, so it can respond quickly, overlap with you, or interrupt.
Haru-4B, a recently released open model, is a good example to study because its makers published the architecture and the claimed numbers. The numbers and the real-world behavior do not yet match, which is also worth understanding.
TL;DR
- Most voice assistants are a relay race. Speech recognition, a language model and a text-to-speech engine run one after another, and each waits for the one before it.
- A full-duplex model collapses the relay into one model. Haru-4B chops audio into slices of about 133 milliseconds and handles listening and speaking in the same step.
- The claimed latency gap is large. Its makers illustrate roughly 2 to 3 seconds for a cascaded system versus about 0.32 seconds for Haru.
- Skipping text keeps tone and emotion. In a cascaded system everything becomes plain text first, so excitement or hesitation in your voice gets lost.
- The architecture is ahead of the results. One reviewer who ran Haru-4B locally found irrelevant answers and repetition, and the company calls it a pre-trained base model for research.
- For now, treat it as a research direction, not a product to deploy. The idea matters more than this particular model.
How does a cascaded voice assistant work?
Most voice assistants you have used are "cascaded." That means a chain of three separate systems:
- Speech recognition (ASR) listens to your audio and writes it out as text.
- A language model reads that text and writes a text reply.
- A text-to-speech (TTS) engine reads the reply aloud.
This design is popular because each piece can be swapped and improved on its own. It has two well-known weaknesses.
The first is delay. Each stage typically waits for the one before it to finish, so the delays add up. Haru's makers illustrate about 2 to 3 seconds for the old approach. Anyone who has talked to a voice bot knows the awkward pause that creates.
The second is lost information. Once your voice is turned into text, the system no longer knows whether you sounded excited, annoyed or unsure. It only sees the words. It also struggles with natural overlap, like a quick "mm-hm" while you are still talking, because the pipeline is built around clean turns: you finish, then it starts.
How does Haru-4B work?
According to the model's published diagram, as described by a reviewer who walked through it, the process runs like this.
Step 1: Slice the audio. Incoming sound is cut into slices of about 133 milliseconds each. That is roughly an eighth of a second.
Step 2: Combine three inputs per slice. For every slice, the model takes your voice, its own previous voice output and its own previous words. These are added together into one input.
Step 3: Feed one language model. That combined input goes into a single language model. Haru-4B is built on Qwen 3.5 with 4 billion parameters.
Step 4: Predict two things at once. The model predicts the next word and the next slice of its own voice in the same step. Because it is listening and speaking in the same step, it can in theory laugh, say "uh," or interrupt while you are still talking.
The loop then repeats, slice after slice, for the length of the conversation. There is no "wait until the user finishes" gate, because the model is continuously updating as sound arrives.
You set up a character with a text card (for example, a persona description) and give it a short audio clip to define the voice. The company says it supports nine languages, though the reviewer found it only really worked in English and Japanese.
What are "continuous audio latents" and why do they matter?
This is the most technical idea in the design, and it is worth understanding in plain terms.
Many speech models turn audio into "codes," a bit like converting a smooth curve into a list of chunky steps picked from a fixed menu. That works, but it throws away some detail and often needs several passes to produce one slice of sound.
Haru, according to its makers, does something different. The voice is not turned into chunky codes. It is a smooth stream of numbers (a "continuous latent") produced in a single quick step. A latent is just the model's internal numeric description of a sound. Keeping it continuous means less rounding, and producing it in one step helps with speed.
The reviewer called that single-step design a very clever design on paper and quite impressive. It is a claim from the company's own diagram, not an independent benchmark.
Is a single model really faster than a pipeline?
In principle, yes, for a simple reason: there is less waiting. In a pipeline, the language model cannot start until recognition has produced text, and the voice cannot start until the language model has produced words. In a single model, the next slice of voice is predicted as part of the same step that handles the incoming audio.
The comparison its makers draw:
| Cascaded (ASR, LLM, TTS) | Haru-4B (single model) | |
|---|---|---|
| Structure | Three systems in sequence | One model, listening and speaking together |
| Illustrated delay | About 2 to 3 seconds | About 0.32 seconds |
| Tone and emotion | Lost when speech becomes text | Model hears the audio directly |
| Overlap and interruption | Awkward, turn-based | Possible in theory |
Be careful with those delay numbers. They come from the company's own illustration. Real-world delay depends on your hardware, your network and how the test is measured. A well-tuned cascaded system can also be faster than the 2 to 3 seconds shown.
Does Haru-4B actually work?
Not yet, based on the one hands-on test we have. A reviewer ran the model's own demo locally, using about 12 GB of VRAM (the memory on a graphics card) on an H100, though he noted it could fit on a consumer GPU. Two things stood out:
- In the company's curated demo, the model was asked something and replied with an answer that was irrelevant to the question, repeating a self-introduction.
- In a custom test using a "home baker" persona and a recorded question about burned baking, the model sounded responsive but repeated itself and gave answers that did not connect to what was said.
Even the demo on the company's technical blog, a conversation about investing with a financial adviser, was judged unnatural and flaky, though better than the live local test.
To be fair, this is one person's testing, and the company describes Haru-4B as a pre-trained base model for research, with training pipelines and a GitHub repo shared so others can fine-tune it. It is open weights under an Apache 2 license, so researchers can build on it. A base model is the raw starting point, not a polished assistant. A fair reading: the design is interesting, the current checkpoint is early.
What does this mean for you?
If you use voice AI at work, such as call-center bots, meeting assistants or phone menus, expect the experience to improve over the next few years as single-model approaches mature. The delay and the "bot talks over me" problems are exactly what this architecture targets. You do not need to act on Haru-4B itself.
If you are choosing a voice product today, you are almost certainly choosing a cascaded system. Judge it on what you can test:
- Measure the pause between you finishing a sentence and the first sound of the reply.
- Interrupt it mid-answer and see what happens.
- Speak with an accent, background noise or a pause, and check whether it holds up.
- Ask whether the system sees only text or also picks up tone.
If you are technical or curious, the model is open, so you can run it and form your own view. Expect to fine-tune it for any serious use.
If you manage a team, the practical lesson is about vendor claims. A diagram showing 0.32 seconds is a hypothesis. Ask for a live demo on your own use case, not a curated clip.
Could you build a voice tool for your own process?
You do not need a research model to get a useful voice tool. If your pain is something like a phone line that takes appointment requests, or an internal helpdesk people would rather call than type into, the app around the voice matters as much as the voice itself: a database, sign-in, and somewhere for requests to land. Remy builds full-stack business apps from a plain-English description, and one backend can power a web app and also a voice agent that works over the phone or in the browser. You could describe a request-intake line and a simple dashboard for the people who follow up, instead of stitching together another subscription.
Frequently Asked Questions
What does full-duplex mean in AI voice models?
It means the model can listen and speak at the same time, like a normal phone call. Half-duplex systems take strict turns. A full-duplex model can in theory respond to a mid-sentence "wait, no" or add a quick acknowledgment while you are still talking.
What is the difference between speech-to-speech and cascaded voice AI?
A cascaded system chains three parts: speech recognition, a language model and text-to-speech. A speech-to-speech model handles audio in and audio out in one model, without forcing everything through text. That can cut delay and keep tone and emotion.
Is Haru-4B ready to use in a product?
Based on one reviewer's local tests, no. It gave irrelevant and repetitive answers, and the company itself describes it as a pre-trained base model for research that you can fine-tune. It is open weights under Apache 2, so it is useful for experimentation.
Does 0.32 seconds of latency mean it will feel instant?
Not necessarily. That figure is the company's own illustration, compared with about 2 to 3 seconds for a cascaded system. Actual delay depends on hardware and how it is measured, and the model has to give good answers for low delay to matter.
How we source, write, and correct our articles. Editorial standards
Related articles

How Does ReShoot 4D Change a Video's Camera Angle With AI?

Tencent's Open World Model: Can an RTX 4090 Run It?

How Does Zonos 3 Mini Work? MoE Text-to-Speech Explained
