Nova Learns to See and Hear

Part of the free Generative AI course on LogicWiz, module: Nova Finds Her Voice.

Episode 27: Nova Learns to See and Hear

"Someone plays Priya two songs and asks which one a human made. She picks confidently — and gets it wrong. Both were generated by AI. The same machinery that lets Nova write can also let her see a photo, compose a tune, and paint a picture. This is where 'language model' stops being about language."

Nova Opens Her Eyes

So far Nova has lived in text: words in, words out. This episode she gains senses. She'll look at an image and tell you what's in it, and we'll see how the very same ideas let models generate music, audio, and pictures.

The word for this is multimodal — a "modality" is just a type of data (text, image, audio, video). A multimodal model handles more than one. And the surprising part is that it mostly reuses machinery you already understand.

A Token Is a Token

Remember the two big ideas from this chapter: everything is tokens, and models predict the next token in a loop. Here's the leap: a token doesn't have to be text.

For text, a token is a word or word-piece.
For an image, a token can be a small patch of the picture.
For audio, a token can be a tiny slice of sound.

Chop any kind of data into a sequence of tokens, and "predict the next token" suddenly applies to it. That's the whole secret to why one family of models can write, draw, and sing — it's the same skill, pointed at different tokens.

Nova Looks at a Picture

The easiest sense to add is sight — letting Nova take an image as input and answer questions about it. Amazingly, it's the same chat call you already know, with one twist: the message's content becomes a list that mixes text and an image. Walk through it, then run it:

That's genuine multimodal input: pixels go in, an answer comes out. The model isn't reading a caption — it's actually looking. (This runs live in a real browser tab with your OpenAI key; a vision-capable model like gpt-4.1-mini handles the image.)

Nova Makes Music

Sight is input. Now the other direction: generating audio. Type "upbeat lo-fi with a soft piano" into a tool like Google's Lyria (available in Gemini) and it produces a track — no instruments, no recording.

How? The autoregressive loop from Episode 25, applied to sound. The model generates a short chunk of audio (say two seconds), appends it, feeds the whole thing back, and generates the next chunk — over and over until the song is done:

This is exactly how autoregressive audio models like Meta's MusicGen work: a text prompt goes in, a little audio comes out, and the audio so far is fed back in — the same predict-append-repeat you saw for words. (Lyria above is the tool you'd actually use; MusicGen is a good example of the mechanism, since its inner method is public. And remember from last episode — audio tokens are pricey, which is why generating media costs more than chatting.)

Drawing: A Different Trick

Not everything is made token-by-token, though. Images and video are mostly built a completely different way, and it's worth knowing the contrast:

Autoregressive image generation places one patch after another — the token loop again.
Diffusion (the method behind Stable Diffusion and most modern image/video tools) starts from pure random noise and sharpens the whole picture at once, a little each step, until it matches your prompt. Not piece-by-piece — the entire image emerges together.

So the two great generation styles are: sequence-of-tokens (text, audio) versus denoise-the-whole-thing (images, video). Same goal, opposite strategies.

Whose Art Is It? Watermarks and Copyright

When AI can produce music and images this good, two hard questions follow.

"Can you even tell it's AI?" Increasingly, not by ear or eye. So providers embed an invisible watermark — a signal woven into the pixels or audio that humans can't perceive but software can detect. Google's version is called SynthID; feed a SynthID-marked file back into Gemini and it can flag "this was AI-generated." Other provenance efforts include Meta's Stable Signature and the industry-backed C2PA "content credentials" standard (backed by Adobe, Microsoft, and OpenAI), all aiming to establish where a piece of media came from.

"Who owns it — and was it trained legally?" This is genuinely unresolved. There's intense legal debate: authors and news outlets (like The New York Times) have sued major AI companies, arguing their books and articles were used to train models without permission. Whether AI outputs can be copyrighted, and whether training on copyrighted work is fair, varies by country and has no global answer yet.

⚠️ Treat AI-generated media as legally gray. If you ship it in a product, check provenance and your provider's terms — the rules are still being written.

What Nova Learns Next

Nova can now see and (in principle) sing. But there's one sense that changes how people talk to her more than any other: voice. Next episode, Nova learns to listen and speak — and you'll see the two very different ways to build a voice assistant, and when to pick each.