Nova Gets a Voice

Part of the free Generative AI course on LogicWiz, module: Nova Finds Her Voice.

Episode 28: Nova Gets a Voice

"Priya's mom will never open a chat window, but she'll happily talk to a phone. The moment Nova can listen and speak, she reaches millions of people who'd never type a word. And the surprise is how little of what you already know has to change."

Nova Learns to Listen

Nova can read, see, and generate. Now she gets the sense that changes who can use her more than any other: voice. Voice agents are everywhere — customer support lines, in-car assistants, healthcare triage, banking fraud calls, language tutors. Anywhere hands or screens are awkward, voice wins.

And here's the reassuring part: you already know most of it. You probably assume a talking AI needs some special, exotic "voice model." The most common way to build one doesn't: a converter turns your speech into text, your ordinary text LLM thinks, and a second converter turns its words back into sound. The hard-looking part is just two off-the-shelf converters bolted onto the LLM you already know.

Three Jobs: Listen, Think, Speak

Strip any voice agent down and it always does exactly three things:

Listen — capture the audio and turn it into text. This is Speech-to-Text (STT).
Think — read that text, reason, and plan a reply. This is just the LLM you already use.
Speak — turn the reply text back into audio. This is Text-to-Speech (TTS).

Sound in → text → think → text → sound out. The "Think" in the middle is completely ordinary. Let's prove it.

Listen for Real, Then Think

Let's run the whole loop for real. We'll take an actual audio clip, use Whisper to hear it (Listen → text), hand that to the ordinary chat LLM to think up a reply, then use TTS to speak it back — Nova reads it aloud. Watch how little of this is "voice-specific":

Look what actually happened: a real speech-to-text call turned audio into words, an ordinary LLM call thought up a reply, and a TTS call spoke it back. That's the whole point of this episode: the "Think" step is completely ordinary — the only genuinely audio parts are the Listen converter (Whisper) at the front and the Speak converter (TTS) at the back. The one voice-specific habit is the system prompt asking for a short, spoken reply — because a good voice assistant talks like a person, not like a wall of text. So the full Listen → Think → Speak loop is really just your familiar LLM with a converter bolted on each end. (Runs live in a real browser with your key and your speakers on. Some browsers block auto-playing sound; if you hear nothing, Nova's reply still printed above.)

Two Ways to Wire It

There are two very different ways to assemble Listen → Think → Speak, and the choice drives cost, speed, and quality:

Cascaded — three separate specialist models chained together: an STT model, then your LLM, then a TTS model. Like Lego: you can swap in the best (or cheapest) model for each stage. It's flexible and cost-effective, but it adds a little latency at every hop — often a few hundred milliseconds to a second of extra round-trip delay (STT → LLM → TTS), which is why native end-to-end models feel snappier in live conversation — and there are three pieces to maintain.
Native (end-to-end) — a single model that takes audio in and gives audio out, with no separate STT/TTS. It's fast and natural, and because it hears the raw audio it can pick up tone — sarcasm, anger, sadness — which pure text throws away. The trade-offs: it's more expensive and locks you into one provider.

💡 The killer advantage of native models is emotional intelligence. In customer service, hearing an angry tone lets Nova respond very differently than she would to the same words typed flatly. Voice carries feeling that text simply loses.

The Models Behind the Voice

You don't build STT and TTS from scratch — you plug in existing models:

Speech-to-Text (Listen): Whisper (OpenAI, the long-time standard), Parakeet (Nvidia), and hosted transcription services like Deepgram and AssemblyAI. Whisper is cheap — roughly 0.6 cents per minute of audio.
Text-to-Speech (Speak): OpenAI TTS (e.g. tts-1, about $15 per 1M characters), ElevenLabs, and others.
Native (all-in-one): GPT-4o realtime, Gemini 2.5 Flash, Qwen3-Omni. These use token-based pricing across audio and text — and audio tokens are expensive, so a native realtime model can cost far more than a cascaded pipeline of small specialists.

That cost gap is exactly why the cascaded approach usually wins on price: three small, efficient models sip resources, while one giant native model guzzles them. So the decision comes down to three pillars: quality (emotion, latency), cost, and complexity (how much you want to maintain).

Where Voice Wins

Voice isn't a gimmick — it unlocks whole categories of use:

Customer support — handle queries, take issues, escalate to a human when needed.
Hands-free — driving, accessibility, factory floors.
Healthcare triage — collect symptoms, route patients.
Banking & fraud — Nova can call you to verify a suspicious transaction, and hand off to a human if something's off.
Language learning — real-time conversation practice with corrections.

The pattern is the same everywhere: Listen → Think → Speak, wired as cascaded or native depending on what you're optimizing for.

What Nova Learns Next

Nova can now listen and speak. But great conversation is more than turning sound into text and back — it's about actually following what you asked, remembering the thread, and knowing when to ask you a question. Next episode is the human touch: talking to Nova naturally — instruction drift, follow-ups, and keeping her on track.