A Talking AI Avatar Is Not One Model — It Is STT, LLM and TTS With a Face on the End, and Latency Adds Up

The first time you see a talking AI avatar, it looks like one giant model that listens, thinks and moves a face. Take it apart and it isn't. It's an ordinary voice agent with a face generator attached to the end.

So what decides how good it feels is not how real the face looks but the sum of four latencies. After reading this you'll know why avatars feel slow and in what order to fix it.

Four stages

(1) STT, speech to text. (2) LLM, writing the reply. (3) TTS, reply to speech. (4) Face, speech to mouth and expression frames. VAD (voice activity detection) decides when the user has finished talking, and the video streams both ways over WebRTC.

The first three are a plain voice assistant. Only (4) is new. If you've built a voice agent, think of it as adding one stage. If you've only built text chatbots, the places where latency can hide go from one (the LLM) to four.

Waiting in sequence is slow

If each stage finishes before handing off, the user waits for all four added together. So real products overlap them. Start TTS as soon as the LLM emits its first sentence. Start the face as soon as a bit of audio exists.

The lower rows show the overlap. What people notice is time to first sound, and that moves way earlier. Bar lengths are for illustration, not measurements.

Interruptions are a pipeline problem too

When the user cuts in mid-reply (barge-in), stages (2), (3) and (4) all have to stop. Stop only the face and let TTS keep going, and you get a closed mouth that's still talking.

The Anam SDK has a dedicated TALK_STREAM_INTERRUPTED event and an interruptPersona() call for this. The Anam API docs also list a session analytics endpoint that returns p50 / p90 / p99 per stage, from transcription through LLM and TTS to first audio. The vendor is telling you to look at latency stage by stage. I haven't called that endpoint yet.

What I built to take it apart

In app10 I made a page (/avatar_pipeline) that runs STT, LLM, TTS and face as separate steps and shows milliseconds for each. Anam lets you unbundle stages, which is what makes this possible. Set llmId to CUSTOMER_CLIENT_V1 and you own (2). Turn on enableAudioPassthrough and you own (2) and (3), and Anam only does (4).

I don't have numbers from a real API key yet, and I left the table empty rather than guess. I already know my pipeline will look worse when I do measure: every stage round-trips through my server and nothing is streamed. How much worse is what I'll add once it's measured.

Where to start cutting

Swapping in a better face model comes last. First measure each stage. Overlap what you can with streaming. Add interruption handling. If it's still slow, replace the longest stage.

If you're wiring an avatar or voice agent into a product and latency is where you're stuck, you can ask negabaro (contact).