ModelRefs / Positional Encoding — AI Glossary
Positional Encoding — AI Glossary
A technique injecting sequence-position information into token embeddings so a transformer can reason about word order. Also called position encoding.
Overview
Transformers process tokens in parallel, losing inherent order. Positional encodings—sinusoidal (original), learned absolute, or relative (RoPE, ALiBi)—inject position into embeddings. Choice affects context length, generalization, and long-range attention quality.
Reference details
| Topic | architecture |
|---|---|
| Also known as | position encoding |
| Last reviewed | 2026-06-24 |
Related terms
How it works
- Attention is permutation-invariant, so without added position information a transformer cannot distinguish token order at all.
- Absolute schemes add a position-dependent vector to each embedding, fixing an index to each slot.
- Rotary schemes instead rotate query and key vectors by an angle proportional to position, so the attention score depends on relative distance rather than absolute index.
- Relative and bias-based schemes add a distance-dependent term to attention scores directly, which is cheaper and extrapolates differently.
- The choice sets how far context can be extended after training, because a scheme that encoded absolute indices has no defined behaviour past the longest index it saw.
Example: Why the scheme sets the extrapolation limit
A model trained with learned absolute positions up to 4,096 has no parameters for position 4,097 — the index simply does not exist, so extension requires training new ones. A rotary scheme encodes position as a rotation angle, which is defined at any position, so the same weights can be run longer; quality still degrades because the frequencies were tuned for the training range, which is why extension techniques rescale those frequencies rather than adding parameters. The scheme decides which kind of problem extension is.
Commonly confused with
Confused with context length, which it constrains without being. Context length is how many tokens a model will accept; this is the mechanism by which order is represented, and it determines whether that length can be extended after training. It is also not positional bias: this is the deliberate encoding of position, while that is the empirical finding that content is weighted unevenly by where it sits — one is a design choice, the other a behaviour the choice contributes to.
When to use it
Reach for it when:
- Explaining why a context window can or cannot be extended post-training.
- Comparing architectures on long-context behaviour.
- Reasoning about why attention needs position information at all.
Reach for something else when:
- As a synonym for context length, which is the limit rather than the mechanism.
- Explaining uneven attention over position, which is positional bias.
- Assuming any scheme extends freely — extension quality depends on the scheme.
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Positional Encoding — AI Glossary.
Frequently asked questions
What is Positional Encoding?
A technique injecting sequence-position information into token embeddings so a transformer can reason about word order.
Is Positional Encoding the same as position encoding?
Yes — position encoding are common aliases for Positional Encoding.
What concepts are related to Positional Encoding?
Closely related concepts include rope, alibi, transformer, attention.