tok/s ≈ B / (N · q)

Decode wall

tok/s ≈ B / (N · q)

B = HBM bandwidth, N = parameters, q = bytes per weight

Autoregressive decode re-reads the weights every token. On a 3090 (936 GB/s) a 27B model at 4-bit is ~87 tok/s if you actually hit the bus. If you measure less, you are leaving bandwidth on the table.

Prefill

FLOPs ≈ 2 · N · S

S = sequence length; tok/s ≈ peak FLOP/s / FLOPs per token

Prefill is compute-bound until the sequence is short or the quant is extreme. Tensor Cores matter here. They barely matter at batch-1 decode.

KV cache

bytes = 2 · L · n_kv · d_h · S · q_kv

L layers, n_kv heads, d_h head dim, S sequence, q_kv bytes per element

Long context is a RAM tax, not a FLOP tax. GQA, quantized KV, and a smaller n_kv buy sequence. They do not buy a smarter model.

Roofline

I = FLOPs / bytes; achieved = min(peak_FLOP, I · B)

I = arithmetic intensity. Ridge point = peak_FLOP / B

If I is below the ridge, more Tensor Cores do nothing. Fuse, tile, or raise batch until I climbs, or buy bandwidth.

Speculation

E[tokens] = (1 − α^{γ+1}) / (1 − α)

α = draft accept rate, γ = draft tokens per step

A fast draft on a slow target wins. A slow draft on a fast 7B is a tax. Measure α on your prompt, not on HumanEval.

Occupancy

occ = active_warps / max_warps

Limited by registers, shared memory, and the block size you picked

nvidia-smi at 100% can still be stalled on the scoreboard. Nsight is the instrument. Occupancy is the first number it should teach you.