Activation-Weighted Low-Rank Compression of Transformer Weights

Abstract

Standard SVD compression of transformer weights fails at 3x ratios because it minimizes error in weight space rather than activation space. This work proposes an activation-weighted fitting method that uses calibration data to identify which weight directions the model actually uses. At the same compression budget, the method reduces perplexity degradation by 5–7x across three architectures (GPT-2, Gemma-3, Qwen2.5). On Gemma-3-1B, the compressed model scores below the original’s perplexity — a denoising effect. The method requires no gradient steps and solves each matrix in closed form.

1. Introduction

A large language model stores knowledge in weight matrices — grids of numbers with millions of entries. Compression replaces these with simpler approximations that use less memory.

The standard approach, truncated SVD, asks which directions carry the most mathematical energy and keeps those. But the model does not use all directions equally. Some receive a flood of activation traffic every time the model runs. Others are barely touched. SVD spends its limited budget on directions that look big on paper but do not matter to the model’s computation.

The proposed method asks which directions the model actually uses, and spends the budget on those.

2. Why standard compression fails

When a transformer’s weight matrix is compressed using truncated SVD, each individual layer’s error looks small, but the full model’s performance collapses. On GPT-2 Small, plain SVD at 3x compression raises perplexity by 21%. On Gemma-3-1B, it raises it by 72%.

Two competing explanations have been proposed:

Hypothesis 1 — Drift accumulation

Approximation errors compound through the residual stream. Later layers receive inputs they were never trained on, so even a good per-layer fit does not guarantee a good network.

Hypothesis 2 — Wrong norm

SVD minimizes error in weight space (Frobenius norm), but what matters is error under the activation distribution. SVD wastes rank on directions the model never excites.

3. Method

For a linear layer y = Wx, the approach captures calibration activations using a forward hook, then solves a simple optimization: minimize activation-space error with a weight-space anchor to prevent divergence in unobserved directions.

Mathematical formulation

The objective finds a rank-r approximation Ẍhat; minimizing:

L(Ẍ) = (1/n) Σᵢ ||(W − Ẍ)xᵢ||² + β ||Ẍ − W||²_F

The first term is the activation-space error (what the network actually sees). The second is a weight-space anchor that prevents the fit from diverging in directions the calibration data never excites.

Let G = XXᵀ (Gram matrix) and C = (Y−b)Xᵀ + βnW (cross-moment). The unconstrained minimizer is:

M* = C (G + βnI)⁻¹

The result is truncated to rank r in the G-weighted norm — the norm in which the original objective measures error.

Algorithm

For each target module:

1. Capture input activations X and outputs Y over calibration chunks (forward hooks)
2. Subtract bias from Y if present
3. Compute G = XXᵀ + βnI and C = YXᵀ + βnW
4. Solve M* = CG⁻¹ (eigenspace solve with ridge regularization)
5. Truncate to rank r in the G-weighted norm
6. Write back the compressed factors

One hyperparameter: β (anchor strength). Optimal in [0.1, 0.3]; all results use β = 0.1.

Why the anchor matters

Without the anchor (β = 0), the problem is ill-posed in directions the calibration data never excites. The fit can produce arbitrarily large weights in those directions without any penalty, leading to divergence. The anchor keeps the compressed model close to the original in regions where there is no activation evidence.

5. Results

Same model, same target matrices (attention output projections), same rank (one-sided 3x compression), same evaluation protocol (WikiText-2 test split). The only difference: SVD optimizes in weight space; the proposed method optimizes in activation space.

Cross-scale comparison

PPL = perplexity (lower is better). Negative values mean the compressed model outperforms the original.

Compression as denoising — can compression improve the model?

The Gemma-3-1B result is notable: the compressed model scores below the teacher’s perplexity (−4.45% on 50 texts, −7.34% on 200 texts) with no fine-tuning. Logit-space analysis shows the student is not a copy — it has higher output entropy (2.19 to 2.47) and lower NLL, suggesting it is less overconfident and better calibrated.

Interpretation: directions of W that are never excited by natural text carry no signal but contribute weight-space capacity for spurious logits. Activation-weighted truncation removes them, flattening spurious peaks and moving mass onto correct tokens — denoising.

This claim rests on one architecture and one family. Three-seed replication and a second architecture are needed before claiming it broadly.

Factor quantization — int8 is lossless

On Qwen2.5-7B (rank 1194), quantizing the compressed factors to int8 is effectively lossless (+3.80% vs +3.86% for fp16), achieving 3.0x per-matrix storage reduction. int4 quantization costs an additional +4.4pp but remains within the quality gate.

Ratio frontier — detailed

Both methods compress the same weight matrix by replacing it with a lower-rank approximation. SVD picks the directions with the most mathematical energy, regardless of whether the model uses them. The proposed method uses calibration data to identify which directions receive actual activation traffic, then allocates the budget to those directions instead.

At 2x compression, both methods produce negligible quality loss. The gap opens at 3x: SVD raises perplexity by 21% while the weighted fit adds only 3.6%. Beyond 3x, SVD collapses rapidly — the model becomes essentially useless at 4x (+152%). The weighted fit degrades more gracefully, remaining usable at 4x (+18%) and only becoming impractical at 6x and beyond.

The key observation: weighted 4x beats SVD 3x while storing 25% fewer parameters. The method recovers an entire compression tier that plain SVD destroys.

6. Claims

Supported by current evidence

• Activation-weighted fitting outperforms plain SVD at the same rank budget on attention output projections across three architectures (GPT-2, Gemma-3, Qwen2.5).
• The improvement factor is 5–7x in perplexity delta at 3x compression.
• int8 factor quantization is effectively lossless for the fitted subspace.
• The “drift wall” at 3x compression is largely an artifact of the approximation norm, not a fundamental limit of scale.

Suggestive, needs additional validation

• Compression can act as denoising (Gemma-3-1B beats the teacher). Requires multi-seed replication and a second architecture.
• The hybrid low-rank + quantized residual design outperforms pure int4 on o_proj. Requires MLP-family evaluation where the payoff should concentrate.

Open — not currently claimed

• Algorithmic novelty over ASVD or EoRA — head-to-head comparison not yet run.
• Whole-model compression — only attention output projections are demonstrated at 3x.
• Deployment readiness — warmed-up latency, peak memory, and fused kernel behavior are unmeasured.
• Downstream task performance — only perplexity has been evaluated.

7. What this paper does not claim

• This work does not claim that low-rank compression universally beats quantization for storage. Quantization dominates pure storage; the value here is quality-at-rank and the denoising effect.
• This work does not claim that every transformer matrix is equally compressible. Only attention output projections are viable at 3x; MLP families break even with weighted fitting.
• This work does not claim that activation weighting solves all compression problems. It is one technique with specific scope and tradeoffs.
• This work does not claim deployment readiness. The compressed representation changes the compute graph; latency and memory implications are uncharacterized.
• This work does not claim that the denoising effect is general. It has been observed on one architecture and requires multi-seed replication.

8. Limitations

Scope of compression

Only attention output projections are demonstrated at 3x. MLP matrices — which contain 80% of parameters — break even with weighted fitting at this ratio. Whole-model compression requires budget reallocation across families, which has not been shown.

Statistical robustness

Most results are single calibration/fit runs. A ±1pp robustness test on disjoint calibration data shows stability, but multi-seed statistics are not yet available.

Evaluation scope

Evaluation is limited to perplexity on WikiText-2 (50–200 texts). No downstream-task evaluation, no pre-registered drift metric alongside PPL.

Latency and memory

Factored form changes the compute graph. Warmed-up latency and peak memory are unmeasured. Without a fused low-rank kernel, the storage win does not automatically become a speed win.

9. What did not work

Direct pruning

Deleting 50% of weights (even using activation-aware selection) raised GPT-2 perplexity from 36 to 79 — still 2x worse than the original. At 90% pruning, perplexity exceeds 15,000. Pruning and low-rank compression are fundamentally different operations: pruning removes structure entirely, while low-rank compression redistributes it.

Student-input refitting

Trying to anticipate drift by fitting against the compressed model’s own activations (rather than the original model’s) consistently lost ~1pp across all settings. Anticipating drift is not the lever; fixing the norm is.

MLP compression at 3x

Weighted fitting on MLP matrices (c_fc, c_proj) produced 76–2149% PPL increases at 3x ratio. MLP families do not have enough redundancy at this compression level, even with activation-aware fitting.

Reference neurons

Using reference neurons to encode weight matrices was consistently less efficient than matched SVD at every rank level tested. This confirmed SVD as the right baseline for comparisons.

10. Related work

Activation-aware quantization (GPTQ, AWQ)

What they solve: Post-training quantization — reducing the bit-width of weight entries.

What they optimize: Weight-space rounding error under the second-order (Hessian/activation Gram) statistic of calibration data.

How this differs: This method operates on the low-rank structure rather than quantization grids. The objective is the low-rank analogue of their principle: preserve what activations actually use.

What remains uncertain: The interaction between the low-rank branch and quantization-aware baselines at matched budgets.

ASVD — Activation-aware SVD (Yuan et al., 2023)

What it solves: Reparameterizes weights by activation statistics before SVD, handling outlier channels.

What it optimizes: Activation-aware low-rank approximation for LLM compression.

How this differs: The closed-form anchored fit with weighted truncation is an independent derivation in the same family. The beats-the-teacher denoising effect has not been reported for ASVD.

What remains uncertain: Head-to-head comparison at matched budget and protocol is the sharpest novelty test and remains open.

EoRA — Eigenspace Low-Rank Compensation (Liu et al., 2024)

What it solves: Adds a low-rank residual path to compensate compression error.

What it optimizes: The compensation rank approximates compression error in the eigenspace of input activations.

How this differs: EoRA adds a compensation path on top of a base compression; this method replaces the base compression itself. The two could potentially be combined.

What remains uncertain: Whether this method, EoRA, or a combination performs best at matched budgets.

QEP — Quantization Error Propagation (Arai & Ichikawa, 2025)

What it solves: Error accumulation across layers in quantized models — directly the drift problem, for quantization.

What it optimizes: Sequential layer-wise fitting against drifted inputs rather than teacher inputs.

How this differs: QEP addresses cross-layer error propagation for quantization; this work addresses the per-layer approximation norm for low-rank compression. The two problems are related but distinct.

What remains uncertain: Whether QEP-style cross-layer compensation would further improve low-rank compression.

SVDQuant — Low-rank + quantization hybrid (Li et al., 2024)

What it solves: Absorbs outliers into a low-rank branch, allowing aggressive quantization of the residual.

What it optimizes: Combined low-rank + int4 representation for diffusion models.

How this differs: This work shows the low-rank branch itself survives int8 losslessly, reinforcing the hybrid design space. The weighted-fit branch outperforms an SVD branch at matched storage.

What remains uncertain: Full hybrid evaluation on MLP families where the payoff should concentrate.

Glossary

References

1. Frantar et al. “GPTQ: Accurate post-training quantization for generative pre-trained transformers.” arXiv:2210.17323, 2022.
2. Frantar & Alistarh. “SparseGPT: Massive language models can be accurately pruned in one-shot.” arXiv:2301.00774, 2023.
3. Lin et al. “AWQ: Activation-aware weight quantization for LLM compression and acceleration.” arXiv:2306.00978, 2023.
4. Yuan et al. “ASVD: Activation-aware singular value decomposition for large language model compression.” arXiv:2312.05821, 2023.
5. Liu et al. “EoRA: Fine-tuning-free compensation for compressed LLM with eigenspace low-rank approximation.” arXiv:2410.21271, 2024.
6. Arai & Ichikawa. “Quantization error propagation: Revisiting layer-wise post-training quantization.” arXiv:2504.09629, 2025.
7. Li et al. “SVDQuant: Absorbing outliers by low-rank components for 4-bit diffusion models.” arXiv:2411.05007, 2024.
8. Merity et al. “Pointer sentinel mixture models.” arXiv:1609.07843, 2016.