Visual Encoder — AI Glossary

Reference details

Related terms

• Multimodal Architecture
• Cross-Attention
• Embedding

How it works

1. The image is divided into fixed-size patches, each flattened and projected into an embedding — the step that turns pixels into a sequence the rest of the stack can process.
2. A vision transformer processes that patch sequence with self-attention, so each patch's representation is conditioned on the whole image rather than on a local neighbourhood.
3. Pretraining is usually contrastive against text, which is what aligns visual representations with language in the first place and why the encoder's output is useful to a language model at all.
4. A projection module maps the visual embeddings into the language model's input space, either a simple learned mapping or a module that also compresses many patch embeddings into fewer tokens.
5. Resolution and patch size together determine the token count, which grows with the square of the side length — the single most important cost property of a multimodal system.
6. High resolution is handled by tiling: the image is split into tiles each encoded at native resolution, plus a downscaled overview, which multiplies the token count by the tile count.

Example: Why one image can cost more than a long document

With 14-pixel patches, a 224 x 224 image produces (224 / 14)^2 = 16^2 = 256 patch tokens — cheap. Resolution costs quadratically: a 448 x 448 image gives (448 / 14)^2 = 32^2 = 1,024 tokens, four times as many for twice the side length. Now take a high-resolution image handled by tiling into four 448 x 448 tiles plus a 224 x 224 overview: 4 x 1,024 + 256 = 4,352 tokens for a single image. At roughly 1.3 tokens per word that is about 4,352 / 1.3 = 3,348 words of text-equivalent — a long document's worth of context spent on one picture. The consequences follow immediately: a conversation with five such images carries 5 x 4,352 = 21,760 tokens of image before any text, and at 3 units per million input tokens a million such requests cost 21,760 x 1,000,000 / 1,000,000 x 3 = 65,280 units on images alone. This is why multimodal APIs expose a detail or resolution setting, and why sending a full-resolution screenshot when a 224-pixel thumbnail would answer the question is the most common avoidable cost in a vision pipeline.

Commonly confused with

A visual encoder is not the multimodal model, and the boundary explains most multimodal failure. The encoder turns pixels into embeddings; the language model reasons over them. So a model that misreads a chart may have encoded it at a resolution where the axis labels occupy a fraction of a patch — an encoder-side information loss that no amount of reasoning recovers. Diagnosing a vision failure therefore starts with whether the detail was representable at the resolution used, not with whether the model understood it.

When to use it

Reach for it when:

• Any pipeline passing images to a language model, where the encoder's resolution setting is the main cost and quality lever.
• Choosing resolution deliberately per task — a thumbnail for scene description, high resolution for small text.
• Diagnosing vision failures, where asking whether the detail was representable precedes asking whether it was understood.
• Cost modelling for multimodal traffic, since image tokens frequently dominate the input budget.

Reach for something else when:

• At maximum resolution by default, which multiplies token cost quadratically for detail the task may not need.
• For tasks a dedicated vision model does better — detection, segmentation, precise measurement — where a general multimodal stack is both weaker and more expensive.
• Expecting reliable reading of fine text or dense charts without checking the effective resolution per character.
• With many images in one context, where the token count crowds out the text the model needs to reason with.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Visual Encoder — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path