Prefix Tuning — AI Glossary

Reference details

Related terms

• Adapter (PEFT)
• Prompt Tuning
• LoRA (Low-Rank Adaptation)

How it works

1. Trainable vectors are prepended to the key and value sequences of every attention layer, not to the input embeddings.
2. The base weights stay frozen, so only those vectors receive gradients — the parameter-efficient part.
3. Because the vectors sit in every layer, they influence computation at all depths rather than only conditioning the first.
4. They occupy positions in the attention sequence, so they consume context length that the actual input could have used.
5. A reparameterisation network often generates the vectors during training to stabilise optimisation, and is discarded afterwards.

Example: Why depth matters more than count here

Prompt tuning adds, say, 20 trainable vectors at the input embedding layer only: 20 x 4,096 ≈ 82,000 parameters. Prefix tuning adds 20 at each of 32 layers, for keys and values both: 20 x 32 x 2 x 4,096 ≈ 5.2M — about 64 times more, and still a tiny fraction of a 7B base. The extra expressiveness comes from acting at every depth, and the cost is that 20 positions of every layer's attention sequence are now spent on the prefix.

Commonly confused with

Routinely merged with prompt tuning, and the difference is where the learned vectors live. Prompt tuning trains vectors at the input embedding layer alone, so the model conditions on them once; this trains vectors in every layer's key-value sequence, so they act throughout the stack — more expressive, more parameters, harder to optimise. Both differ from low-rank adaptation, which modifies weight matrices rather than adding sequence positions, and therefore costs no context length and can be merged into the base weights for inference.

When to use it

Reach for it when:

• Adapting behaviour with very few trainable parameters where low-rank adaptation underperforms.
• The adaptation needs to act at all depths rather than condition the input.
• Many task-specific prefixes are wanted over one frozen base.

Reach for something else when:

• Context length is tight, since the prefix occupies sequence positions in every layer.
• Low-rank adaptation would do — it is usually easier to train and mergeable at inference.
• The behaviour needed is a substantial capability change rather than steering.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Prefix Tuning — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path