Voice Cloning Is a Prompt, Not Training — Modern TTS Is an LLM Over Audio Tokens
What the Fish Audio API fields give away: you pass a voice as a reference clip, not a model id
When people hear "voice cloning" they tend to picture hours of training an AI on their voice. I did too. Then I built a Fish Audio tester in app10 and read the API docs. The field where you pick a voice doesn't take a model id. It takes a reference clip.
After reading this you'll know why modern TTS (text-to-speech) is built like an LLM, and what you actually need to prepare when you add cloning.
Turning sound into words
It starts with a neural codec. The codec slices the waveform into short frames and replaces each frame with a number from a fixed codebook. Sound becomes something like words in a dictionary. Those are audio tokens.
Once sound is tokens, TTS is the same problem as a text LLM: look at the previous tokens, predict the next one.
Cloning falls out of this for free. Encode a reference clip into tokens, put it in front of the prompt, and the model continues "in this voice". Same idea as giving ChatGPT a few sample sentences and asking it to match the style.
The API shape gives it away
Fish Audio's /v1/tts request is pretty blunt about it. I re-checked the official docs on 2026-10-19.
| Field | What it does | LLM equivalent |
|---|---|---|
reference_id |
A voice you uploaded earlier | A saved system prompt |
references |
Attach a reference clip plus its transcript to the request | Few-shot examples |
temperature / top_p |
Default 0.7. Higher means more varied intonation | Sampling temperature |
repetition_penalty / max_new_tokens |
Repetition control, cap on audio tokens per chunk | Same names |
chunk_length / condition_on_previous_chunks |
Split long text, pass the previous chunk's audio as context | Context window |
The docs describe max_new_tokens as the maximum number of audio tokens. That line settled it for me.
One thing tripped me up. When you attach audio inline with references, the docs say the request has to be MessagePack, not JSON, because it carries raw audio bytes. If your tester only speaks JSON, you hit this wall once.
How it differs from older TTS
| Older TTS (concatenative, parametric) | LLM-style TTS |
|---|---|
| Phoneme dictionaries, pronunciation rules | Text goes in as tokens |
| Train a model per speaker | Reference clip as the prompt |
| Flat intonation | temperature sets the range |
| Same input, same output | Sampled, so every run differs a bit |
That last row matters more than it looks. Regenerate the same sentence and the intonation can change. For announcements that must sound identical every time, generate once and keep the file.
So when do you fine-tune?
Fine-tuning a model on your own data, like GPT-SoVITS does, is still an option. The rule of thumb is simple.
You just need it to read in your voice -> start with a reference clip. All you need is one short recording
The reference clip can't get the tone you want, or it has to run offline -> then look at fine-tuning
To be honest, I haven't generated audio and compared quality yet. The intonation gap between temperature 0.1 and 0.9, and how clip length changes the result, I'll write up after listening myself. This post is about the structure you can read off the API.
Don't clone someone else's voice without their consent. The easier it gets, the clearer that line needs to be.
If you're adding TTS to a product and can't decide between prompt-style cloning and fine-tuning, feel free to ask negabaro (contact).
Key Concepts
A neural codec maps each short slice of audio to one code: sound becomes tokens
A transformer reads the text and previous audio tokens and predicts the next audio token
Put reference-voice tokens before the prompt and it continues in that voice: that is cloning
It is sampled, so the same sentence varies; save the file when you need it fixed