Architecture Is the Circuit, Weights Are the Knob Positions — and Attention Variants Are Hacks Around n²
Can you use a model as soon as its weights are public, and why GQA matters for long context
"What even are weights? If they're public, can I just use them in transformers?" and "What's an Attention variant?" are really one question. Short answer: the architecture is the circuit diagram, and the weights are the positions of every knob on it. The diagram is published in papers. Finding where each knob goes is what all the training money pays for. After reading this you'll be able to check whether an open model is usable with four questions, and know what GQA or Sliding Window in a model card is trying to cut down.
Two things in a model file
blueprint (config.json) layers, dimensions, heads <- the Transformer "pattern". Usually public
weights billions of numbers set by training <- everything the model knows
For Llama 8B that's 8 billion numbers. Even with the same architecture you can't swap in another model's weights. Each knob was tuned assuming where all the other knobs are.
Weights are public. Can you use them? Four checks
Is the architecture implemented in a library? Well-known ones usually are, in HF transformers.
Is the tokenizer published too? Without the rules for turning text into numbers, the weights are useless.
Does the license allow your use? Some, like Llama, come with conditions, so read the license itself.
Do you have the hardware? An 8B model runs on a laptop once quantized. The 70B class needs several high-end GPUs.
Claude, GPT and Gemini keep their weights closed, so you use them through an API. Llama, Qwen and Mistral publish theirs, so they run locally.
Why Attention gets modified: n²
Original Attention has every token look at every other token once. Picture a meeting where every attendee has a one-on-one with every other attendee. With n tokens you get n² pairs. 1,000 tokens means a million pairs. 100,000 tokens means ten billion. Make the context 100 times longer and the work grows 10,000 times.
So people built variants that look at less, split the looking, or do the same looking faster.
| Variant | Idea | What it cuts | Used in |
|---|---|---|---|
| Multi-Head (original) | Look from several angles (heads) at once | Nothing, it's the baseline | Original Transformer, BERT |
| MQA / GQA | Heads share Keys and Values | KV cache memory | Llama 2 70B, Llama 3, Mistral 7B |
| Sliding Window | Only look at the nearest N | Compute on long inputs | Mistral 7B |
| Sparse Attention | Only look at some fixed positions | Compute | GPT-3 (alternating with dense) |
| Linear Attention | Approximate n² as roughly n | Very long inputs | Mostly research |
| FlashAttention | Same result, different order of GPU memory reads | Memory traffic time | Many recent training and inference libraries |
FlashAttention doesn't approximate anything. The paper's title says "Exact Attention". It speeds things up by cutting trips to GPU memory, not by computing less.
Why GQA matters: work out the KV cache yourself
A generative model stores the Keys and Values of all previous tokens each time it produces a new one. That's the KV cache. If every head keeps its own copy, GPU memory fills up fast in long conversations.
I ran the numbers from Llama 3 8B's config.json: 32 layers, 8 heads that hold Keys and Values (32 query heads), head dimension 128, fp16 (2 bytes).
per token = 2 (K, V) x 32 layers x 8 heads x 128 dims x 2 bytes = 131,072 bytes = 128KB
without GQA, all 32 heads keep their own = 512KB
at 8,192 tokens -> 1GB with GQA / 4GB without
That's calculated from the config, not measured. Still, the 4x gap falls straight out of the architecture. It's why GQA is close to standard in long-context models.
It's the same kind of move as adding caching or query tuning to an MVC app. The pattern stays, the insides get reworked.
If the Transformer itself is fuzzy, start with Transformer. How the same weights get boxed up for distribution is in ONNX is a package format.
If you're trying to put an open model on a local machine or an internal server and can't make memory, license and hardware line up, feel free to ask negabaro.
Key Concepts
A model file is a blueprint (config.json) plus weights. All the knowledge is in the weights
Before using open weights, check implementation, tokenizer, license and hardware
Attention is n². 100 times the context means 10,000 times the work
KV cache = 2 x layers x KV heads x head dim x bytes x tokens. GQA shrinks the KV heads term