The one-sentence version
When compressing a neural network, don't ask "what's mathematically biggest?" — ask "what does the model actually use?" The answer changes which parts of the network you preserve, and it makes compression 5–7× less destructive.
The idea in 30 seconds
Neural networks store their knowledge in massive matrices — grids of numbers with millions of entries. To make the model smaller, we replace these matrices with simpler approximations that keep the most important patterns.
The standard approach (SVD) keeps the directions with the most mathematical energy. But the model doesn't use all directions equally. Some directions carry most of the computational traffic; others are barely touched.
Our method keeps the directions with the most activation traffic — the ones the model actually uses. Same compression budget, very different results.
Imagine compressing a road network. Standard SVD preserves the widest roads — even if some are abandoned highways. Our method preserves the busiest roads — the ones that carry real traffic. Same map size, but the important connections survive.
Why this matters
Previous attempts at low-rank compression of transformers hit a "drift wall" — at 3× compression, the model's performance collapses. This was thought to be a fundamental limit: compress too much, and errors compound through the network.
We show the wall is mostly an artifact of the wrong compression objective. When you optimize in the right geometry (activation space instead of weight space), the wall largely disappears.
The problem
Plain SVD at 3× compression raises perplexity by 21% on GPT-2 and 72% on Gemma — making models nearly unusable.
Our result
Activation-weighted fitting at the same 3× compression raises perplexity by only 3.6% on GPT-2 and actually improves Gemma by 4.5%.
The method in three steps
No equations required. Here's the intuition behind each step.
Observe what the model does
Feed the model some sample text and record which directions in each weight matrix get activated. This creates an "activation profile" — a map of where the computational traffic flows.
Solve for the best compressed version
Instead of finding the mathematically simplest approximation (SVD), find the simplest approximation that preserves the activation patterns you observed. This is a one-shot calculation — no training, no gradients.
Keep rank where it matters
When deciding which directions to keep and which to discard, prioritize the directions that carry activation traffic. The result: the model's important computational pathways survive compression.
How to read the paper
The full paper is available as a Markdown draft or LaTeX source. Here's how to approach it depending on your background.
If you know basic machine learning
Start with the introduction (Section 1) to understand why low-rank compression fails. Then skip to Section 5 (Results) to see the cross-scale comparison table. The key figure is the perplexity delta table — it shows our method beating SVD by 5–7× across three model sizes. Section 8 (Limitations) is essential reading to understand the scope of the claims.
If you are an ML researcher
Read the paper in order. Section 3 (Method) presents the closed-form solution and its derivation. Section 4 (Experimental Setup) describes the protocol — all results use identical calibration, evaluation, and control conditions. Section 5.4 (Matrix-family study) shows where the method works and where it doesn't. Section 6 (Analysis: Compression as Denoising) presents the most surprising finding. Section 7 (Storage Accounting) honestly positions the method against quantization. Check the pre-submission checklist for open gaps.
Key equations to understand
The core optimization objective:
This says: find a compressed version Ŵ that produces small errors on the calibration activations (first term), while staying close to the original weights in directions where we have no evidence (second term, the anchor).
The closed-form solution:
Where G = XXᵀ is the activation Gram matrix and C = (Y−b)Xᵀ + βnW is the cross-moment. This is computed in the eigenspace of G with ridge regularization.
Glossary
- Low-rank
- A matrix represented using fewer underlying dimensions. Instead of storing every entry independently, you store two smaller matrices whose product approximates the original. This is a form of compression.
- SVD (Singular Value Decomposition)
- A standard mathematical method that breaks a matrix into its most important directions, ranked by how much "energy" each direction carries. It's the default tool for low-rank approximation.
- Activation
- The signal produced inside a neural network when an input passes through a layer. Activations tell you which parts of the network are "firing" for a given input.
- Calibration data
- Representative inputs used to observe the model's activation patterns before compression. The method needs to "see" what the model does in order to know what to preserve.
- Rank
- The number of independent directions retained by the compressed representation. Rank-256 means keeping 256 main patterns. Higher rank = more faithful to original but less compression.
- Perplexity (PPL)
- A language-model metric measuring how well the model predicts text. Lower is better. A PPL of 36 is good; a PPL of 15,000 means the model is essentially guessing randomly.
- Activation-weighted
- A compression objective influenced by the directions emphasized by actual model activations, rather than by mathematical magnitude alone.
- Gram matrix
- A matrix summarizing which directions appear frequently in the calibration activations. It defines the geometry in which the compression error is measured.
- Weight-space anchor
- A regularization term that prevents the compressed model from producing arbitrarily large weights in directions the calibration data never excites. Without it, the fit diverges.
- Drift
- The phenomenon where approximation errors in early layers compound through the network, causing later layers to receive inputs they weren't trained on. This is why per-layer compression quality doesn't guarantee whole-model quality.
Main results
All results are on attention output projections at one-sided 3× compression, evaluated on WikiText-2.
| Model | Baseline PPL | Plain SVD | Weighted fit |
|---|---|---|---|
| GPT-2 Small (124M) | 56.47 | +21.34% | +3.58% |
| Gemma-3-1B | 70.40 | +71.86% | −4.45% |
| Qwen2.5-7B | 12.29 | +26.31% | +3.84% |
Negative values mean the compressed model outperforms the original. The improvement factor over SVD is 5.9×, 16.2×, and 6.9× respectively.
What does "PPL increase" mean?
Perplexity measures how confused the model is when predicting the next word. A PPL increase of +3.58% means the model is 3.58% more confused than the original. A PPL decrease of −4.45% means the compressed model is actually less confused than the original — it's a better language model after compression.
What does "beats the teacher" mean?
When we say the compressed Gemma-3-1B "beats the teacher," we mean the compressed model achieves lower perplexity on held-out text than the original uncompressed model — without any fine-tuning. This is unusual: compression normally makes models worse, not better. Our interpretation is that activation-weighted truncation acts as denoising, removing weight directions that never fire on natural text but contribute noise to the model's predictions.
Limitations
Where this result stops being supported.
- Scope: Only attention output projections are demonstrated at 3×. MLP matrices (80% of parameters) don't have enough redundancy at this compression level.
- Seeds: Most results are single calibration runs. Multi-seed statistics are not yet available.
- Evaluation: Only perplexity on WikiText-2. No downstream tasks, no drift metrics.
- Latency: The compressed representation changes the compute graph. Speed and memory implications are unmeasured.
- Novelty: Head-to-head comparison against ASVD and EoRA (the closest related methods) has not been run.
What this paper does not claim
- That low-rank compression universally beats quantization for storage.
- That every transformer matrix is equally compressible.
- That activation weighting solves all compression problems.
- That the method is ready for deployment.
- That the denoising effect is general (it's observed on one architecture).
Further reading
Full paper draft
Available as Markdown or LaTeX. Contains all equations, proofs, experimental details, and appendices.
Source code & experiments
GitHub repository with standalone experiment scripts, results, and run contracts.