What Is Multimodal AI? Input Support, Token Costs and How to Choose

Multimodal AI is AI that takes images, PDFs, audio and video as input, not just text, and answers about them. But a "multimodal" label says little about what you can actually send: it varies a lot by model. Checking each vendor's official documentation on November 12, 2026, the APIs of Claude and OpenAI's general-purpose models stop at text, images and PDFs, while Google's Gemini 4 and Qwen's Omni models accept audio and video directly. The three big vendors also count image tokens differently, so the same phone photo can cost more than ten times as much input on one model as on another.

This article covers how multimodal AI works (stitched vs native), a table of what the major models accept, the token formulas for images, video and PDFs with worked examples, how to read the benchmarks, what to check for each use case, and the limits the official documentation itself lists.

1. What is multimodal AI — AI that takes more than text

The definition: "an AI model that can also handle non-text input (images, audio, video and so on)." How far that goes depends on the model — from text plus images only to audio and video in a single model.

Earlier AI was built for one modality. GPT-3 handled text only, Whisper only turned speech into text, and Stable Diffusion only turned text into images. Combining them meant building a "pipeline" that feeds one model's output into another, and information was lost along the way. How the text-generating part works is explained in What is an LLM.

In multimodal AI, one model looks at several inputs at once and answers. For example, "look at the error message in this screenshot (image) together with my question (text) and tell me the cause" takes a single API call.

2. Stitched vs native — two ways to build it

There are two basic designs. It used to be standard to chain separate models: one transcribes speech, one thinks about the text, one reads the reply aloud (OpenAI describes ChatGPT's voice mode before GPT-4o as exactly this three-model chain). Today, designs where one model takes several kinds of input directly are spreading.

A concrete example: the task "watch this cooking video and write up the recipe." A stitched setup needs several stages — "transcribe the audio with a speech model, then summarize with a text model" plus "extract frames from the video and analyze them separately." A model that accepts video directly can take the whole video in one API call and match the spoken explanation to the on-screen actions inside the model.

But not every vendor has moved to a single native model. OpenAI splits the work: its general-purpose models (such as GPT-6 Astra) take text and images, voice conversation goes to voice models such as GPT-Live-1 and GPT-Realtime, and transcription goes to models such as GPT-Transcribe (OpenAI's model list, checked November 12, 2026). You pick a dedicated model per job, and the voice models themselves take audio directly and answer in audio. Full-duplex voice conversation — listening while speaking — is covered in our GPT-Live explainer.

3. What the major models accept — a table from official docs

The table lists the input types that each vendor's API documentation says it accepts (checked November 12, 2026). Features that route through another model or a transcription step behind the scenes, such as voice mode in an app, are not included.

Sources: Anthropic's models overview ("All current models support text and image input, text output") and PDF support; OpenAI's GPT-6 Astra page (input: text, image; output: text) and file inputs guide; the Gemini 4 Carbon model card and Gemini API model list; Alibaba Cloud's Qwen-Omni docs; DeepSeek's API changelog (V4.1 Flash has "native multimodal visual understanding"). For the current model lineup, see our model and cutoff list.

Three things stand out.

• All three big vendors read PDFs as "page images plus extracted text." Claude turns each page into an image and passes the page's extracted text alongside it. OpenAI also sends both text and page images on vision-capable models. That is why tables and charts can be read — and why PDFs use more tokens (§4).
• Audio and video split vendors into "pick another model" and "put it into one model." Gemini 4 and Qwen's Omni models take them in one model. OpenAI routes audio to voice and transcription models, and the Claude API accepts neither audio nor video. To handle video with Claude or OpenAI's general-purpose models, you extract frames and send them as images, and transcribe the audio separately.
• Output is almost always text. Images come from image-generation models (our image generator comparison) and spoken replies from voice models — "understanding" and "generating" are often handled by different models.

For Qwen, Alibaba Cloud's docs list Qwen3.8-Omni-Flash (input: text, images, audio, video; output: text; 1M-token context) and Qwen3.5-Omni (output: text and audio; up to 3 hours of audio and 1 hour of video). Google's model list likewise carries separate models for low-latency voice dialogue (Gemini 3.8 Live) and transcription (Gemini 3.5 Transcribe).

4. How many tokens are images, video and PDFs — formulas and examples

Multimodal input is converted to tokens and billed like text — but each of the three vendors converts differently. These are the formulas in each vendor's docs (checked November 12, 2026).

Worked example: how many tokens is the same image?

We applied the formulas above to two common images. Costs cover input only; the model's thinking and output are billed separately.

Price sources: Anthropic, OpenAI, Gemini API pricing (all checked November 12, 2026). Claude's full-HD figure matches the table in Anthropic's docs. Gemini's figures are caps, so real counts are at or below them.

The takeaway: the token gap comes almost entirely from whether the image gets downscaled. GPT-6 Astra keeps the original resolution by default, which helps with fine print but makes large photos expensive. If you send many phone photos, setting detail to high or resizing before upload cuts input to roughly a fifth. Conversely, Gemini has settings to raise the resolution when you need to read small text. Whatever the formula, cropping away what you don't need before sending helps most.

Video and audio: how many tokens is an hour?

• Gemini (video as is): by default it samples one frame per second. Google's video docs say "approximately 100 tokens per second of video at default resolution, about 300 at high resolution." An hour is about 360,000 tokens, so a 1M-token model fits about 3 hours at default and about 1 hour at high resolution (video understanding docs). Gemini 4 and some other models also offer opt-in "agentic video understanding," which looks only at the parts it needs; Google says it uses up to 88% fewer tokens on long videos, but also notes that on short clips (under 5 minutes) or broad questions that require watching most of the video, it can use more tokens and take longer.
• Gemini (audio only): at 25 tokens per second, an hour of audio is 90,000 tokens.
• Claude and OpenAI (frames as images): a 1280×720 frame is 46×26 = 1,196 tokens on Claude and 40×23×1.2 = 1,104 tokens on GPT-6 Astra. One frame every 10 seconds for an hour (360 frames) comes to roughly 400,000–430,000 tokens, with no audio included. On GPT-6 Astra, a request with more than 272K input tokens is priced at 2x input and 1.5x output for the whole request, so long videos are cheaper sent in parts. Claude accepts up to 600 images per request, but above 20 images a stricter per-image size limit applies (keeping the long edge at 3,000 pixels or less stays within it on every platform).

PDFs: cost grows with page count

Anthropic's docs say each PDF page typically uses 1,500–3,000 tokens for the text alone, plus the page image counted with the formula above (up to 600 pages per request, or 100 when the context window is under 1M tokens). OpenAI also adds page images, and since PDF detail defaults to high on GPT-5.6 and later, the image part is large (files up to 50 MB combined per request). Gemini does not charge for the native text embedded in a PDF and counts pages as images (up to 1,000 pages or 50 MB; document processing docs). For documents over 100 pages, sending only the sections you need is both cheaper and more reliable. General token-saving tips are in how to cut AI token costs.

5. What decides strength in each modality

"Which model is best" changes with every generation. What stays useful is knowing what technically determines strength in each modality. With these four axes, you can judge new models yourself.

Through this lens, video gaps open up with length. Video-MME scores clips from 11 seconds to 1 hour in short, medium and long buckets, and many models on its official leaderboard score lower on the long bucket. If you only handle short clips, long-video rankings don't matter to you. In document analysis, the more an agent works by looking at a screen and acting on it — as Cursor does — the more returning positions matters over raw accuracy.

6. Reading the benchmarks — MMMU-Pro no longer separates the top

When MMMU, a multi-discipline test with images, was released in late 2023, GPT-4V scored 56%. On its harder successor MMMU-Pro (released September 2024), the top models now sit close to 90%. As measured by the independent firm Artificial Analysis (the reasoning setting differs by model): Gemini 4 Carbon (Medium) 89%, Claude Opus 5.5 (Max, Default Fallback) 88%, GPT-6 Astra (Max) 87% (MMMU-Pro evaluation page, checked November 12, 2026). The top spots are only 1–2 points apart.

Vendors are moving to tests that still separate models. In the "multimodal understanding" section of the Gemini 4 Carbon model card (published November 10, 2026), Google lists no MMMU-Pro and instead shows two other tests, Chartography and Blueprint-Bench 2 (Chartography: Gemini 4 Carbon 72.5%, GPT-6 Astra 71.0%, Claude Opus 5.5 66.3% — Google's measurements). When choosing a model, look at the following four according to your use case.

Whatever the number, read it knowing whether the vendor measured it or a third party did. Vendor announcements tend to pick favorable tests; third-party tables use consistent conditions but cover fewer tests. In the end, testing with your own material, as in §7, is the most reliable check.

7. By use case — what to check before you choose

Here are five use cases and what to check before deciding. They are written as checking steps rather than model names, so they still work when new models come out.

8. Pitfalls and limits

Anthropic's docs spell out the limits of image understanding (Limitations in the Vision docs). They are written about Claude, but other models tend to stumble in the same places.

Don't read a guess from a photo as fact

Even if you ask "read the amount on this receipt," a dark, skewed or small image can lead the model to produce a plausible-looking number. Even models near 90% on MMMU-Pro get roughly one in ten wrong. Amounts, dates and proper names should always be checked by a person against the original image — especially in legal, finance and medicine.

The longer the video, the more it misses in the middle

Even top models on video tests struggle to pull accurate details from the middle of an hour-long video (the same problem as Lost in the Middle in long text). For key scenes, narrow it down by time — "look closely at 30:00–35:00 only" — for better accuracy.

Audio struggles with accents, jargon and overlapping voices

Accuracy is high for standard conversation, but errors rise with strong accents, technical jargon, several people talking at once, and noisy places. For important recordings such as meeting minutes, use a dedicated transcription model (OpenAI's GPT-Transcribe, Google's Gemini 3.5 Transcribe and the like, as of November 2026) and give it proper names up front.

Summary

• Multimodal AI is AI that takes non-text input (images, PDFs, audio, video). What you can send differs by model.
• In the APIs as of November 12, 2026, Claude and OpenAI's general-purpose models accept text, images and PDFs, while Gemini 4 and Qwen's Omni models take audio and video too in one model. OpenAI routes audio to dedicated voice models.
• Image tokens are counted as Claude = 28-pixel patches, OpenAI = 32-pixel patches × 1.2, Gemini = a cap per resolution setting. GPT-6 Astra keeps the original size by default, so phone photos drop to about a fifth with detail: "high" or resizing.
• Video on Gemini is about 360,000 tokens per hour. With Claude and OpenAI you send frames as images and transcribe audio separately.
• MMMU-Pro has the top models bunched near 90%. Check video, documents and audio with their own tests and your own material.
• Poor images, positions, counts, the middle of long videos and accented audio are error-prone. Have a person check amounts and dates.

To start, give the AI you use one screenshot or PDF from your own files, ask it to "copy out the numbers in the table exactly," and compare with the original. Once you see where it slips, you know how far you can trust that model.

FAQ