Benchmark Methodology Guide
Learn proper LLM performance testing practices and avoid common benchmarking mistakes.
Installation
- Make sure Claude is on your device and in your terminal.
Skills load from
~/.claude/skills/when Claude Code starts up — so you need it on your machine first. If you don't have it yet, install it once with the command below, then runclaudein any terminal to verify.One-time setupnpm i -g @anthropic-ai/claude-codeAlready have it? Skip ahead.
- Paste into Claude Code or into your terminal.
This copies the whole skill folder into
~/.claude/skills/bench-ddalcu/— the SKILL.md plus any scripts, reference docs, or templates the skill ships with. Works for every skill.What this command does
- Makes a new, empty temporary folder.
- Downloads the skill's folder,
.agents/skills/bench, fromgithub.com/ddalcu/mlx-serve(branchmain), the GitHub repository it's listed from. You get the version that is there now, not a copy we keep. - Copies that folder to
~/.claude/skills/bench-ddalcu/. - Deletes the temporary folder.
Installing runs none of the skill's files; it only downloads and copies them. When you later use the skill, Claude follows its SKILL.md and may run scripts that come with it. Read them first.
Faster alternative (instruction-only skills)
Skips the clone and grabs only the SKILL.md file. Don't use this if the skill ships Python scripts, reference markdowns, or asset templates — they won't be downloaded and the skill will fail when it tries to load them.
Quick install (SKILL.md only)Sign up to copy - Restart Claude Code.
Quit and reopen Claude Code (or any other agent that loads from
~/.claude/skills/). New skills are picked up on startup. - Just ask Claude.
Skills auto-activate when your request matches the skill's description — no slash command needed. Trigger phrases live in the skill's own frontmatter; you can read them in the “What this skill does” section above.
Prefer to read the source first? Open on GitHub.
When Claude uses it
mlx-serve benchmarking methodology — bench.sh/llmprobe usage, comparison-trap rules (same-methodology cells only, spec-decode variance, thermal lies, engine naming), perf-claim etiquette. Use before running benchmarks or making any performance claim.
What this skill does
Benchmarking
llmprobe is the measurement layer. tests/bench.sh boots mlx-serve (one model at a time: boot, probe, kill, settle) and llmprobe takes every number via --bench-only. We do not hand-roll timing loops — llmprobe discards a warmup per scenario, reports median-of-3 as median (min-max), refuses to fabricate a number when usage is missing, records the machine it ran on, and applies the same protocol to every engine.
./tests/bench.sh # every model (~did we regress)
./tests/bench.sh --only qwen38-27b # one row
./tests/bench.sh --url 127.0.0.1:1234 -m <id> # a server someone else started
./tests/bench.sh --full # median of 3 per rung, to 64k
Each cell is mlx-serve at its FASTEST. --mtp is forced wherever the checkpoint ships an MTP head, because it is default-OFF on MoE targets and that is where it pays most (35B-A3B reads 157 without and 191 with). Everything else is already on by default. The mode that actually engaged is read off the server's own [spec-stats] mode= lines and named beside the number — a mode that silently stops engaging shows up as a bare cell, which is the regression signal.
Another engine = another URL. Start LM Studio / oMLX / MTPLX / llama-server yourself, then --url host:port -m <id>. Same script, same probe, nothing about their binaries, ports or version strings lives in the bench.
The only artifacts: the paste-ready rows bench.sh prints at the end, which go into benchmarks.md (one column per release in the history table, plus the cross-engine table rewritten when a comparison is run), and the saved llmprobe reports + server logs under ~/claude-tmp/bench-<tag>/.
Comparison traps (these cost real days)
- Only diff same-methodology cells. Columns through 26.7.12 are the old in-repo harness; 26.8 on is llmprobe — different prompts, different warmup, different rate math. Never diff across that boundary. Same rule inside one column: a forced-spec cell and a shipping-defaults cell are not the same measurement.
- Boot the server with
--warmup-full(bench.sh and bench_versions.sh pass it to a binary that lists it). A Metal kernel compiles at its first use and Metal keeps it per build, so without the flag a build's first pass on a model carries one-time compiles (a cache-hit turn several times slower, a long prompt at a fraction of its rate) and the second pass does not. A binary without the flag: discard its first pass per model. - "Reproducible ≠ not variance" for spec-decode cells — sample across runs and boot orders before any regression claim. A cell that reads the same twice can still be variance.
- Attribute before believing. Check whether the change could physically reach the cell that moved; reachability is faster to check than another bench and is what makes a repeat a confirmation.
- Never quote a win without naming the engine it is over — vs LM-GGUF a row reads +33%; vs oMLX the same row is +1.6%.
- Thermal soak lies harder than drift — same-session ratios only. llmprobe's own sustained-load check catches drift WITHIN a cell; run the comparison engine right after ours, not hours later, or say so beside the number.
- An A/B arm is proven by ENGAGEMENT lines in its own log, never by its launch env. zsh does not word-split
env $VAR, so a multi-switch arm's first switch swallowed the rest as its value and the "composed" arm silently ran the fast path — reading a 2x win as "neutral" for half a session (live 2026-07-30, story in docs/qwentts-cache.md). - A spec-decode collapse that survives SERVER restarts is machine state, not the build (M3 Ultra, PR #223: 5.45→2.0 tok/step persisted across restarts and an ANE-OFF control; a macOS reboot fully restored it) — reboot and re-baseline before attributing.
- Power source is a variable on laptops: measure on AC at high charge — battery→AC roughly doubled one tester's decode MID-RUN (PR #223, M5 Max); Latin-square or A-B-A the arms on laptops so a power/thermal shift reads as drift, not an effect.
- A bench's port wait-list must equal its kill-list. LM Studio's server is a persistent daemon you never kill (its MODEL is freed by
lms unload --all), so waiting on its port burns the full timeout on every stop — measured 11 of 20 min on one run. This is why bench.sh no longer manages other engines at all.
Related skills
Documentation Co-Authoring
anthropics
Guide structured workflows for writing docs, proposals, and technical specs collaboratively.
MCP Server Builder
anthropics
Build protocol servers that connect language models to external APIs and services.
Skill Builder & Optimizer
anthropics
Create, edit, and optimize Claude skills with performance testing and benchmarking.
Multi-Component Web Artifacts
anthropics
Build complex React artifacts with Tailwind CSS and shadcn/ui components.