Blog
Zonos 3 Mini architecturemixture of experts TTSaudio codec codebooks

How Does Zonos 3 Mini Work? MoE Text-to-Speech Explained

Zonos 3 Mini explained in plain English: an 800M-parameter mixture of experts with about 100M active, a 15 fps audio codec, speaker cloning and 44.1 kHz output.

Portrait of Luis Chavez-Mattos
Edited by Luis Chavez-Mattos, Director, Product··RSS

For Engineering, Marketing and Design

A flat wall of many small identical expert modules pinned to a backdrop by one open palm, with only two of them glowing.

How does Zonos 3 Mini turn text into speech?

Zonos 3 Mini, from Zyphra, turns text into speech in four steps. It cleans and tokenizes your text. It optionally turns a short reference clip into a voice signature. A mixture of experts model then predicts compressed audio "frames" one at a time, 15 per second. Finally, a decoder converts those frames into 44.1 kHz audio as they are generated. The model has 800 million parameters in total, but only about 100 million work on each token, because it routes each step through a small group of its 64 experts.

That is the whole pipeline. The rest of this article explains each piece so you can judge what the model is good for.

TL;DR

  • Zonos 3 Mini is a text-to-speech model from Zyphra that supports 43 languages and can copy a voice from a short audio clip. It is released under an Apache 2 license, which allows commercial use.
  • It has 800 million parameters in total, but only about 100 million are active on each token, because it uses a mixture of experts design with 64 experts.
  • The model does not generate sound waves directly. It predicts codec frames at 15 per second, and a decoder turns them into 44.1 kHz audio.
  • Each frame holds 10 codes: one semantic code (what is being said) and nine detail codes (how it sounds).
  • The voice comes from a reference clip, either as raw audio codes or as a single speaker embedding. Loudness, bandwidth and speaking rate are also conditioning inputs.
  • In one hands-on test, the model used close to 25 GB of VRAM, so a 24 GB GPU is a plausible target. Quality was good in many languages but uneven in voice cloning.

What is a mixture of experts model?

A mixture of experts (MoE) model is a neural network split into many smaller specialist sub-networks, called experts. A small routing component looks at each piece of input and picks which experts should handle it. The rest stay idle for that step.

Think of a hospital front desk. Every patient checks in at the same place, but the receptionist sends each one to a few relevant specialists instead of making all 64 doctors examine everyone. The hospital has a lot of total expertise, but each patient only uses a slice of it.

In Zonos 3 Mini, there are 64 experts, and the model picks a couple of them at each step. That is how a model with 800 million parameters can run with only about 100 million active on any given token.

Why does "100M active" matter?

Total parameters tell you how much the model can store. Active parameters tell you how much computing each step costs. Those are different numbers in an MoE model.

This has practical effects:

  • Speed per step. Each step does the work of a roughly 100M-parameter model, which helps with the real-time feel. In the hands-on demo, audio rendered quickly enough to play back in real time.
  • Capacity. The full 800M parameters still give the model room to learn many voices, accents and 43 languages.
  • Memory. All 800M parameters still need to be loaded, so the memory footprint reflects the full model, not just the active slice. The creator who tested it saw close to 25 GB of VRAM in use on a 48 GB card. That figure is from one setup, so treat it as a rough guide, not a requirement.

The tradeoff is simple. An MoE gives you a bigger model's knowledge at closer to a smaller model's per-step cost, but you pay for it in memory.

What is an audio codec, and why does the model predict codes?

Raw audio at 44.1 kHz has 44,100 samples every second. Predicting that directly, one sample at a time, is slow and hard. So modern speech models use a codec: a pair of tools that compress sound into a short list of numbers and expand it back.

Zonos 3 Mini works like this:

  1. Encoder (training time). Audio is squeezed down step by step to just 15 frames per second.
  2. The model. It predicts those compact frames, one after another, from the text and voice inputs.
  3. Decoder. It converts the predicted frames back into a 44.1 kHz audio stream as they are generated.

Predicting 15 frames a second is far less work than predicting tens of thousands of samples a second. Each frame stands in for a slice of roughly one fifteenth of a second of sound. The decoder reconstructs the fine detail from the codes.

What are the semantic code and the nine codebooks?

Each frame is described by 10 codes, and they split into two jobs.

PartCodes per frameWhat it captures
Semantic branch1What is being said (the linguistic content)
Detail codebooks9Whatever is left over: the fine detail of the sound

The semantic branch is built on a frozen speech model called W2v-BERT. "Frozen" means it was already trained and is not changed during codec training. It listens to the audio and stores the meaning of the speech in one codebook. A codebook is a lookup table of possible values, and each code is an entry from that table.

The remaining nine codebooks store everything the semantic code misses: timbre, texture, breathiness and other acoustic detail. On the way out, the decoder adds the two parts back together and produces the voice.

Splitting content from detail is a sensible design. The first code anchors the model on the right words and pronunciation. The other nine make the result sound like a human voice instead of a flat approximation.

By simple arithmetic, 10 codes per frame at 15 frames per second means the model works with about 150 codes for every second of speech.

How does Zonos 3 Mini know whose voice to use?

The model takes two input pipelines.

The text pipeline cleans up your text, splits it into tokens and embeds them. The demo text included dates, currency, units, an email address and tongue-twisters, which is a good stress test for this stage. The creator called the model's reading of the numbers and dates in the English voices tested "not a bad rendering."

The speaker pipeline takes a reference audio clip and turns it into one of two things:

  • raw audio codec codes, or
  • a single speaker embedding (a compact numeric summary of a voice).

The model uses one of those two paths to decide whose voice to copy. Everything is then lined up as conditioning for the main model: the voice, the loudness and bandwidth, the speaking rate and the text tokens.

Loudness, bandwidth and speaking rate being separate controls is useful. It means pace and audio character are inputs, not side effects.

How good is Zonos 3 Mini in practice?

One creator who ran it locally tested several languages, including English voices, Hindi, French, Italian, Korean, Telugu, Chinese, Spanish and Portuguese. His verdict on the built-in voices was generally positive, with the English number and date reading called "not a bad rendering."

Voice cloning was more mixed. He uploaded his own voice and had it speak Telugu, and found it acceptable. In English the clone did not sound that good to him. In Chinese he found the cloning quality "not bad" but the delivery still a bit "plasticky." His overall take was that it is a solid effort for a text-to-speech model, but not perfect.

That is one person's listening test, not a benchmark. Test with your own text and your own voices before you plan anything around it.

What does this mean for you?

If you work in marketing, training or communications. An open model with a commercial license means narration for training videos, internal explainers or multilingual content no longer has to run through a paid voice service. The 43-language support is the relevant feature. Always have a native speaker review non-English output before it goes to customers.

If you manage IT or a data team. The model is small enough to host yourself. The tested setup used a git clone and the UV package manager, and ran on a GPU with 48 GB of VRAM while using about 25 GB. That points to a 24 GB card as a workable target, but verify on your own hardware. Self-hosting keeps scripts and voice samples on your own machines.

If you want to clone voices. Get consent from the speaker, in writing, and check your company's policy first. Cloning a colleague's or executive's voice raises legal and trust issues that a license does not solve.

What to do this week:

  1. Write a 150-word script from your real work, with numbers, names and any jargon.
  2. Run it through several voices and at least one language besides English.
  3. Listen for mispronounced numbers, odd pauses and flat emphasis.
  4. Decide whether the output is good enough for internal use, customer-facing use, or neither.

Could I build a small tool around a voice model like this?

Yes, and the useful part is usually not the voice itself but the workflow around it. A team that needs narration regularly could use a request form where people submit a script, choose a language and voice, and get back an audio file with an approval status. Remy builds this kind of internal app from a plain-English description, including the database, sign-in and deployment. Its catalog of nearly 400 AI models includes voice and audio models, so you could build the tracker without managing API keys. Remy's pricing starts at $99 per month, plus AI tokens billed at cost.

Frequently Asked Questions

5 questions

Is Zonos 3 Mini free for commercial use?

It is released under an Apache 2 license, which permits commercial use. Read the license text and the model's repository terms for conditions on attribution and redistribution before you ship anything.

What does "100M active parameters" actually mean?

The model has 800 million parameters in total, split across 64 experts. At each step, a router picks a couple of experts, so only about 100 million parameters do work on that token. The rest sit idle until a later step needs them.

Why does the model run at 15 frames per second?

The codec compresses audio so that each second becomes 15 frames. Each frame carries 10 codes. Predicting a short sequence of frames is much easier than predicting every audio sample, and the decoder rebuilds 44.1 kHz audio from them.

What hardware do I need to run it?

In one hands-on test, the model used close to 25 GB of VRAM on a 48 GB card, and the tester suggested a 24 GB GPU should work. That is a single observation, not an official requirement, so check the project's documentation for current guidance.

Can it clone my voice from a short clip?

Yes, it can copy a voice from a short reference clip, either as raw codec codes or as a single speaker embedding. Quality varied in testing, with some languages sounding more natural than others. Get the speaker's consent before cloning anyone's voice.

How we source, write, and correct our articles. Editorial standards

Related articles

Straight to your inbox.
New pieces as they publish. No noise.