Part 1 - Fit, Quality, and Speed:
A Practical Framework for Choosing Quants

Most quantization comparisons eventually run into the same problem: a single number does not tell you which quant to download. It would be nice if it did. It does not.

A smaller model is not automatically better. A faster model is not automatically better. A model that stays closer to the original model’s token distribution is not automatically better either. Each number answers a useful question, but none answers the whole deployment question.

What users actually want to know is simpler:

Which model should I run on my hardware?

That is the question we try to answer at ByteShape.

For us, model evaluation has three parts:

1. Size decides whether the model fits.
2. Quality and speed decide which fitting model is best.
3. The answer can change depending on the hardware and workload.

Size matters until the model and its context fit. After that, it stops being the main character. Users usually care more about the quality-speed trade-off: how much capability the model keeps, and how fast it runs.

TL;DR

• Model size is the fit gate. If a model does not fit the target hardware at the required context length, it is not a practical option.
• Quality and throughput determine the trade-off. Among models that fit, we compare downstream benchmark performance and tokens per second.
• We do not rank releases by fidelity metrics such as KLD or perplexity. They are useful diagnostics, but they mainly measure how much a quantized model diverges from the reference model. They do not tell us whether those changes are helpful, harmful, or irrelevant. For quality, the direction and consequence of the changes matter more than their volume.
• We focus on metrics that help users decide. Our release tables prioritize model size, downstream benchmarks, prompt-processing speed, token-generation speed, hardware details, and reproducible results.

1 - What does it mean for a model to be good?

Modern models can write, code, answer questions, follow instructions, use tools, analyze documents, and generate images. Measuring all of that with one number is impossible.

Some tasks are relatively easy to score. A multiple-choice answer can be checked against a key, a math problem against an exact result, and generated code against unit tests. This is why benchmarks such as MMLU, GSM8K, BFCL, HumanEval, and LiveCodeBench are useful.

Other tasks are much harder to judge. If the prompt is “write a short poem” or “generate an image that feels warm and nostalgic,” there may be no single correct answer. At that point, even humans start grading on vibes.

Benchmarks are imperfect, but they give us a reproducible way to ask the question that matters: did the model actually do the task?

2 - Quantization adds another question

With quantization, we are not only asking whether the model is good. We are asking how good the model is while making it smaller, faster, and easier to run.

In other words:

How much capability can we preserve within a given memory, latency, or power budget?

For local inference, the most visible questions are usually:

• does the model fit on my hardware,
• how much context can I use,
• how fast does it process the prompt,
• and how fast does it generate new tokens?

The goal is not to reproduce BF16 as closely as possible. A quant can move noticeably away from the reference distribution and still preserve, or occasionally improve, benchmark performance. A much smaller change can also flip a correct response into an incorrect one. What matters is the practical outcome, not the numerical size of the change.

For users, the practical question is not:

Which quantized model is closest numerically to BF16?

It is:

Which quantized model performs well, fits my hardware, and runs fast for my workload?

3 - Three kinds of metrics

Quantized model comparisons often mix together three different questions.

3.1 - Task-quality benchmarks

These ask whether the model actually completed the task. Did it answer correctly, write working code, follow the instruction, or produce the required format?

Benchmarks are expensive to run, but they are the closest measure of the outcome users care about.

3.2 - Fidelity diagnostics

These ask how far the quantized model moved from the higher-precision reference. Examples include perplexity, KL divergence, top-token agreement, and logit-distance metrics.

They are useful for detecting large changes or obviously degraded models. But they measure similarity, not whether the change improved or harmed the answer. In Part 2, we test several variants of these metrics directly.

3.3 - Deployment metrics

These ask whether the model is practical to run. They include file size, memory use, supported context length, prompt-processing speed, token-generation throughput, and hardware compatibility.

A useful model must both perform well and run acceptably on the target hardware.

ByteShape evaluations use all three categories, but for different purposes: size determines what fits, benchmarks measure quality, throughput measures speed, and fidelity metrics serve as diagnostics rather than the final ranking.

4 - Why perplexity and KLD are diagnostics, not rankings

Perplexity and KL divergence are common in quantization comparisons because they are relatively cheap to compute and produce a clean number. But they answer a narrower question than the one users usually care about.

4.1 - Perplexity

Perplexity measures how much probability a model assigns to the observed tokens in an evaluation corpus. Lower perplexity means those tokens were considered more likely.

It is a useful diagnostic: a large increase after quantization can indicate substantial degradation. But higher perplexity does not necessarily mean worse task quality. Probability can move away from the reference continuation while the model still produces a correct answer, and low perplexity on a corpus does not guarantee that the model will follow instructions or solve the downstream task correctly.

The evaluation setup also matters. For instruction-tuned models, scoring the full prompt and response includes prompt tokens that were given to the model rather than generated by it, so response-only scoring is often a more relevant diagnostic.

4.2 - KL divergence

KL divergence measures how much the quantized model’s token distribution differs from the higher-precision reference.

That makes it a fidelity metric, not a direct measure of quality. KLD counts any distributional change, whether it improves the answer, breaks it, or only shifts probability among tokens that would never be generated.

A low-KLD quant is closer to BF16 under the chosen evaluation setup. That is useful diagnostic information, but it does not establish which quant is better for a task or deployment target.

5 - A toy example: why fidelity and task quality can disagree

The easiest way to see the difference is to separate two questions: did the model move, and did the move change the answer?

Consider a simplified one-token task. The prompt is:

What is the capital of France?

Assume the model can choose between four possible next tokens: Paris, London, Madrid, and Rome. The correct answer is Paris.

The table below shows the probability distribution of what the next token should be given the prompt.

For the illustrative perplexity value, we use the correct token, Paris, as the reference continuation. For KLD, we compare each model’s full distribution with the original.

Model A is extremely close to the original distribution. Its KLD is almost zero. But the small shift between Paris and London changes the top token, so the task result flips from correct to incorrect.

Model B is less faithful to the original distribution. A distributional metric registers a larger change. But the change moves probability toward the correct answer, so the model becomes more confident in the right output. Its single-token perplexity is also lower because it assigns more probability to the reference answer.

Model C changes the lower-probability part of the distribution while leaving the top answer unchanged. Its single-token perplexity is identical to the original in this example, because the probability assigned to Paris did not change. But its KLD is the largest of the three optimized models because the rest of the distribution moved. Depending on the decoding settings, this difference may have little or no effect on the generated answer, but a fidelity metric still counts it as drift.

This is the central issue with using KLD, perplexity, or logit-distance metrics as the final ranking for quantized models. They can tell us whether a quantized model changed relative to the original model. They cannot, by themselves, tell us whether the change was helpful, harmful, or just movement in a part of the distribution nobody was going to sample anyway.

That is why downstream benchmarks remain central to our evaluation.

6 - What we report in ByteShape releases

For ByteShape releases, we first identify which models fit the target hardware and context length. Among those options, we compare downstream benchmark quality and measured throughput.

Our release tables therefore focus on:

• model size,
• supported context length,
• downstream benchmark quality,
• prompt-processing speed,
• token-generation throughput,
• tested hardware,
• and enough setup detail to reproduce the results.

Benchmarks remain central because they measure user-visible outcomes: whether the model answers correctly, writes working code, follows instructions, or produces the required format. Throughput tells us how practical that quality is on a particular device.

No single number decides the winner. The goal is to filter by what fits, then compare quality and speed among the models that remain.

Coming next: hard, cold data

In Part 2, we stop using toy examples and look at real quantized models. Across 14 variants of KLD, perplexity, and related fidelity metrics, the same pattern appears: they can identify clearly degraded models, but they fail to rank near-baseline quants.