Projects / Small Model Arena
Small Model Arena
Small models, measured the way people actually run them.
Serves one quantized checkpoint, measures it, and compares it to its full-precision original. Thinking on and off.
Where it stands
Building.
No results to publish yet. They land here when they're real.
Why I built it
My small model research needs to know how models under 30B params perform in realistic settings, like running on V100s.
What it is
It takes one Hugging Face checkpoint that is already in the quant you name, serves it on one GPU, and writes a resumable run folder. A crash resumes at the step that didn't finish.
Safetensors go to vLLM by default, with SGLang as an option, and GGUF goes to llama.cpp.
It measures what matters for small models: perplexity, KL against a full-precision reference, quality suites like IFEval, GSM8K, MMLU-Pro and GPQA, and speed with GPU telemetry.
Each model runs at least three ways: BF16, 4-bit with thinking off, and 4-bit with thinking on. Models that do better one way still get a fair result.