IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) — AI Glossary

Reference details

Related terms

• Adapter (PEFT)
• LoRA (Low-Rank Adaptation)
• Prefix Tuning

How it works

1. Three learned vectors are added per transformer block: one scaling the attention keys, one scaling the attention values, and one scaling the feed-forward intermediate activations.
2. Each is applied as an elementwise multiplication against the activation it targets, so the adaptation rescales existing features rather than adding new directions to the weight space.
3. Because the learned object is a vector of length equal to the activation width rather than a matrix, parameter count grows linearly with width instead of with the product of two dimensions.
4. Base weights stay frozen and gradients flow only into the three vectors, so optimiser state is a few hundred thousand entries rather than tens of millions.
5. Vectors are initialised to one, which makes the untrained adapter an exact identity transform and the starting point numerically indistinguishable from the base model.
6. At inference the scaling can be folded into the adjacent frozen weights, so a merged IA³ model runs at exactly base-model latency with no additional matrix multiplications.

Example: Counting the parameters against LoRA on a 7B model

Take a 7B-class decoder with 32 layers, a model width of 4,096 and a feed-forward intermediate width of 11,008. IA³ learns one vector per block for keys, one for values and one for the feed-forward activations, so per block that is 4,096 + 4,096 + 11,008 = 19,200 parameters, and across the stack 19,200 x 32 = 614,400. Against 6.74 billion base parameters that is 614,400 / 6.74e9 ≈ 0.0091%. Now price a conventional LoRA baseline at rank 8 on the query and value projections. Each adapted matrix contributes two factors of 2 x 8 x 4,096 = 65,536 parameters, two matrices per block is 131,072, and across 32 blocks 131,072 x 32 = 4,194,304 — about 0.062% of the base. So the two methods are 614,400 parameters against 4,194,304, a ratio of 4,194,304 / 614,400 ≈ 6.83. The storage consequence is what makes this matter operationally. At fp16 the IA³ adapter is 614,400 x 2 = 1,228,800 bytes, which is 1,228,800 / 1,048,576 ≈ 1.17 MiB, while the rank-8 LoRA is 4,194,304 x 2 = 8,388,608 bytes, exactly 8 MiB. Serving 500 task-specific adapters from memory is therefore about 586 MiB of IA³ against 4,000 MiB of LoRA, which is the difference between holding the whole catalogue resident and paging it.

Commonly confused with

IA³ is not a smaller LoRA, and treating it as one leads to the wrong expectation about what it can learn. LoRA adds a low-rank matrix to the weights, so it can introduce directions the base model did not use; IA³ multiplies existing activations by a scalar per channel, so it can amplify and inhibit what the base model already computes but cannot construct a new feature. That is why it is strongest on few-shot task adaptation, where the needed behaviour is a reweighting of existing capability, and weaker where genuinely new knowledge or a new output format has to be installed. The second confusion is with prompt tuning, which also uses very few parameters: prompt tuning spends its budget on virtual input tokens and consumes context length, while IA³ spends it inside the network and consumes none.

When to use it

Reach for it when:

• For few-shot task adaptation, the setting the method was designed for and where reweighting existing features is usually sufficient.
• When serving a large catalogue of task adapters, where the roughly 6.8x parameter advantage over rank-8 LoRA decides what stays resident in memory.
• Where adapter storage or transfer is the binding constraint, since an adapter near 1 MiB moves differently from one near 8 MiB.
• When inference latency must exactly match the base model, which folding the scalings into the frozen weights achieves.

Reach for something else when:

• For teaching genuinely new knowledge or an unfamiliar output format, where an additive low-rank update has the expressive advantage.
• On large supervised fine-tuning sets, where the extra capacity of LoRA or full fine-tuning is affordable and useful.
• Where the tooling matters more than the parameter count, since LoRA has far wider support across serving stacks.
• As a drop-in replacement expected to match LoRA on every task, given the two methods differ in what they can represent and not only in size.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to IA³ (Infused Adapter by Inhibiting and Amplifying Inner Activations) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path