Multimodal AI:
When Models Learn to See, Hear, and Read

The human brain does not process the world through separate, siloed systems. When you watch someone speak, you hear their words and read their lips simultaneously, and your brain reconciles the two streams into a single percept. When you read a recipe, you may mentally taste and smell. Perception is inherently multimodal. The question AI researchers spent decades avoiding — because it was too hard — was how to build systems with the same property.

That question now has partial, impressive, and still incomplete answers.

The architecture breakthrough: CLIP

The paper that changed the trajectory of multimodal AI was CLIP (Contrastive Language–Image Pre-Training), published by OpenAI in 2021.^[1] The insight was elegant: rather than training a vision model to classify images into a fixed set of categories, train it alongside a language model to associate images with their natural-language descriptions. Using 400 million image-text pairs scraped from the internet, CLIP learned a shared embedding space in which semantically similar images and text were positioned near each other.

The consequences were substantial. CLIP could perform zero-shot image classification — describe a category in plain English and it could identify images of that category without having been trained on examples. More significantly, it demonstrated that the representational gap between vision and language was bridgeable: both could be projected into the same latent space and reasoned about jointly.

CLIP was not a conversational system. It could not answer questions about images or generate descriptions. What it provided was the representational foundation on which conversational multimodal systems would be built.

Vision language models

Flamingo, from DeepMind in 2022, was among the first systems to demonstrate fluid interleaving of image and text in a conversational context.^[2] Given a sequence containing images and text, Flamingo could answer questions about the images, continue stories, describe scenes, and perform visual reasoning tasks — all in natural language. The architecture connected a pretrained vision encoder (similar to CLIP) to a large language model via cross-attention layers, allowing visual tokens to condition language generation.

GPT-4V, released by OpenAI in 2023, brought this capability to a frontier-scale model and a mass audience.^[3] For the first time, a widely accessible system could read handwriting in photographs, describe the content of complex charts, interpret UI screenshots, and reason about spatial relationships in images. The model's performance on vision-language benchmarks was significantly above the previous state of the art — and its qualitative capabilities, to users accustomed to text-only models, felt qualitatively transformative.

Native multimodality: Gemini

The systems above were multimodal by composition: a vision encoder grafted onto a language model. Google DeepMind's Gemini, announced in 2023 and expanded substantially through 2024–25, represented a different design philosophy — multimodality built in from the start.^[4]

Gemini was trained natively on text, images, audio, and video simultaneously, rather than training separate models and connecting them. The claim — and the evidence from published benchmarks supports it — is that native multimodal training produces richer cross-modal representations: the model does not merely translate between modalities, it reasons in a shared representational space that reflects genuine understanding of how modalities relate.

"Gemini is natively multimodal, pre-trained from the start on a dataset that is inherently multimodal. This is a different and more powerful approach than retrofitting multimodal capabilities onto an existing language model."

— Gemini Team, Google DeepMind, 2023^[4]

The practical upshot was performance on video understanding tasks that text-centric models could not approach — the ability to reason about events unfolding across time, not just static images.

Audio and speech

The audio modality has followed a parallel trajectory. OpenAI's Whisper (2022) demonstrated near-human transcription accuracy across 99 languages using a transformer architecture trained on 680,000 hours of multilingual audio.^[5] Whisper is not a multimodal reasoning model — it transcribes speech to text — but it resolved the input bottleneck that had made audio a second-class modality in multimodal systems.

The more striking development is real-time audio-to-audio reasoning: systems that can listen to a spoken question, reason about it, and respond in natural speech without converting to text as an intermediate step. GPT-4o's Advanced Voice Mode, demonstrated in 2024, showed a model that could detect emotion in a speaker's voice, respond to laughter, sing, and adjust its speech prosody in real time — capabilities that require reasoning at the audio level, not just the semantic level.

What multimodal models get wrong

Multimodal capability does not imply multimodal reliability. The failure modes of vision-language models are well-documented and in some cases surprising.

Current models demonstrate strong performance on natural images but degrade significantly on images that deviate from training distribution — medical images, scientific diagrams, architectural drawings, non-Western writing systems.^[6] They exhibit spatial reasoning failures that would not occur in human perception: confusing left and right, misidentifying object counts, failing to track identity across video frames.

Hallucination — the generation of confidently stated false information — extends into the visual domain. A model presented with an image may describe objects that are not present, misidentify depicted text, or construct plausible but incorrect captions. The LMMS-Eval benchmark^[7] has systematised evaluation across these failure modes, providing a clearer picture of where current multimodal models stand and where they remain unreliable.

The applications horizon

Reliable multimodal AI changes the access surface for a wide range of tasks previously requiring human specialists. Radiological image interpretation, accessibility tools for visually impaired users, real-time translation of visual text in foreign environments, code generation from UI mockups, scientific figure analysis — all of these are meaningful applications that are moving from research demonstration to deployed capability.

The constraints are real: performance on out-of-distribution inputs remains a legitimate concern, the computational cost of processing high-resolution images and video at inference time is substantial, and the evaluation frameworks for multimodal systems are less mature than those for language-only models. But the trajectory is unambiguous. The era of single-modality AI systems as the frontier is over. What replaces it is still being built.