The best sites to compare LLMs are Chatbot Arena for human-preference rankings, Artificial Analysis for speed and pricing data, and the Hugging Face Open LLM Leaderboard for open-model benchmarks.
But each one answers a different question, and using the wrong one for your decision will waste your time.
I track these leaderboards weekly as part of my editorial workflow.
When a new model drops and I need to know whether it is actually better or just better-marketed, I have a short rotation of sites I check before forming an opinion.
Some are useful. Some look useful but measure the wrong thing. Here is where I actually go, and why.
Chatbot Arena: The Closest Thing to an Honest Ranking
Chatbot Arena (formerly LMSYS) works like this: you ask a question, two anonymous models answer it, and you vote for the better response. Neither you nor the models know who is competing until after you vote.
The votes feed a Bradley-Terry statistical model (often loosely called “Elo”) that produces a ranking.
As of mid-2026, Arena has collected over 7.1 million blind human votes across 369 ranked models. That sample size is what makes it hard to dismiss.
What I like about it: the methodology is transparent, the voting is blind, and the “Style Control” filter adjusts for the tendency voters have to pick longer, more polished responses over shorter, more accurate ones. That filter alone makes Arena more honest than most benchmarks, because without it, verbosity wins.
Arena runs nine categories now (text, code, WebDev, vision, image editing, search, text-to-video, and image-to-video), so you can check whether a model that ranks well in conversation also holds up in code generation.
The answer is usually no, which is the kind of insight that saves you from a bad API choice.
The limitation. Arena measures vibes. “Which answer do you prefer?” is a real signal, but it is not the same as “Which model will perform best on my production workload?” A model that writes beautiful explanations might hallucinate numbers. Arena cannot tell you that.
A 2025 study titled “The Leaderboard Illusion” raised concerns that large labs can test many private variants and benefit from preference data access. Arena disputes this, but the criticism is worth knowing about.
Best for: getting a quick read on which models are actually competitive right now, especially when a new release drops and you need to cut through the marketing.
Artificial Analysis: Where the Money Questions Get Answered
Artificial Analysis is the site I open when the question is not “which model is smartest?” but “which model gives me the best output per dollar at acceptable speed?”
It tracks over 100 LLM API endpoints across both open and proprietary models, measuring:
- Quality (a composite index based on MMLU, HumanEval, Chatbot Arena scores, and other benchmarks)
- Speed (output tokens per second, reported as median and percentiles over 14 days)
- Latency (time to first token)
- Price (per-token input and output costs, plus a blended rate)
- Context window size
Each endpoint gets tested 8 times per day under six workload combinations (different prompt lengths and parallelism levels).
That testing methodology is more rigorous than most alternatives, which tend to report a single throughput number with no load context.
The interactive charts are the real value here.
You can plot quality against price, speed against quality, or any other combination, and instantly see which models cluster together and which are outliers. It is the fastest way I know to answer “is GPT-4o actually worth 3x the price of Claude Haiku for my use case?”
The limitation. Quality scores lean on benchmarks reported by model creators, not independent evaluations.
Artificial Analysis is transparent about this, but it means the quality axis is only as honest as the labs’ self-reporting.
Speed and price data are independently measured, though.
Best for: developers and teams choosing an API provider.
If you are comparing inference costs across providers or trying to find the sweet spot between quality and latency, start here.
Hugging Face Open LLM Leaderboard: The Open-Source Standard
The Open LLM Leaderboard runs standardized evaluations using the EleutherAI Language Model Evaluation Harness across benchmarks like MMLU-Pro and GPQA.
It is free, it is transparent, and it is the default reference for anyone working with open-weight models.
The leaderboard includes archived runs, so you can track how a model’s performance has changed across versions.
The community around it is active, and the comparator Spaces let you do side-by-side analysis without setting up your own evaluation pipeline.
The limitation. It only covers open models. If your shortlist includes GPT-4o, Claude, or Gemini alongside Llama or Mistral, this leaderboard cannot settle the comparison.
It is also benchmark-heavy in a way that may not reflect real-world performance. A model that scores well on MMLU-Pro might still struggle with your specific task.
Best for: open-source model selection. If you are deciding between Llama 3.1 405B and Mixtral 8x22B for self-hosted inference, this is where you start.
Scale AI SEAL Leaderboard: Private Benchmarks That Cannot Be Gamed
Most LLM benchmarks have a contamination problem.
If the test questions appear in the training data, high scores mean nothing. Scale AI’s SEAL Leaderboard attacks this by using private, unpublished datasets that model developers have never seen.
Domain experts write the prompts from scratch, vetted evaluators grade the outputs, and the whole process goes through multiple rounds of quality control.
Models are generally excluded if their developers had access to the prompts through API logging.
SEAL currently covers coding, instruction following, math (using GSM1k, a private alternative to the widely-contaminated GSM8k), and multilingual tasks.
The limitation. Private datasets solve contamination but create a trust problem in the other direction. You have to take Scale’s word that the evaluations are fair, because you cannot reproduce them. Scale publishes its methodology but not the actual test data.
The leaderboard also updates less frequently than Chatbot Arena or Artificial Analysis, so it can lag behind new model releases by weeks.
Best for: cutting through benchmark inflation. When a model claims a high score on a public benchmark and you want to know if that holds on data it has never seen, check SEAL.
OpenRouter: Compare and Then Actually Use the Models
OpenRouter is primarily a unified API that routes requests to different model providers, but its comparison tool is quietly one of the better ways to evaluate models side by side.
You can compare benchmark availability, token pricing, context windows, latency, and throughput across dozens of models in a single view. The data updates regularly because OpenRouter is routing real traffic through these providers.
The real advantage is that comparison and usage live in the same place. You can identify a promising model, test it through the OpenRouter playground, and switch your production traffic to it through the same API. No vendor-specific SDK changes required.
The limitation. OpenRouter is a middleman, so you pay a small markup on top of provider pricing. For high-volume production use, going direct to the provider is usually cheaper. The comparison data is also limited to models available through OpenRouter’s routing layer.
Best for: developers who want to compare and test in one workflow. Particularly useful if you are running multiple models and want a single API with fallback routing.
Promptfoo: Build Your Own Benchmark
The sites above compare models on generic tasks. Promptfoo lets you compare them on your tasks.
It is an open-source CLI tool where you define test cases in YAML, run them against 60+ providers, and get structured results with assertions, scoring rubrics, and LLM-as-judge evaluations.
It plugs into CI/CD pipelines, so you can regression-test your prompts whenever you update a model.
I mention it because every other tool on this list answers the question “which model is generally best?” Promptfoo answers “which model is best for what I am actually building?” Those are different questions, and the second one matters more once you are past the research phase.
The limitation. Results are only as good as your test cases. Building a meaningful evaluation suite takes time, and LLM-as-judge scoring adds cost and potential bias. It is a tool for teams with a specific use case, not for casual model browsing.
Best for: production teams evaluating models against their own data and prompts. If you are deciding between two models for a specific product feature, Promptfoo gives you a defensible answer instead of a leaderboard number.
Which One Should You Actually Use?
It depends on what you are deciding.
| Decision | Start here |
| “Which model is best right now?” | Chatbot Arena |
| “Which API gives the best quality per dollar?” | Artificial Analysis |
| “Which open model should I self-host?” | Hugging Face Open LLM Leaderboard |
| “Are these benchmark scores real?” | Scale SEAL |
| “Which model works best for my specific task?” | Promptfoo |
| “I want to compare and test in one place” | OpenRouter |
My actual workflow: I check Chatbot Arena first for a directional read, then Artificial Analysis to narrow by price and speed, then run my own tests in a playground before committing.
The leaderboard tells you what is probably good. Your own testing tells you what is actually good for you.
One thing I have learned from tracking these sites: the gap between the top 5 models on any leaderboard is usually smaller than the gap between a well-written prompt and a lazy one.
If you are spending hours comparing models but minutes on your prompts, you have the priority backwards.
FAQ
Are LLM benchmarks reliable?
It depends on the benchmark. Public benchmarks like MMLU have known contamination issues where models may have trained on the test data. Private benchmarks like Scale SEAL avoid this but trade off transparency. Crowdsourced rankings like Chatbot Arena avoid contamination entirely but measure preference rather than accuracy. Use multiple sources.
How often do LLM leaderboards update?
Chatbot Arena updates continuously as votes come in. Artificial Analysis refreshes speed and pricing data daily. Hugging Face updates when new model submissions are evaluated. Scale SEAL updates several times a year. No single leaderboard gives you a complete real-time picture.
Is there a free way to test LLMs side by side?
Yes. Chatbot Arena lets you compare two models head-to-head for free. OpenRouter offers a playground with pay-per-use pricing (some models have free tiers). Promptfoo is free and open source. You do not need a paid subscription to do meaningful model comparison.
What is the difference between benchmarks and human evaluation?
Benchmarks run models through standardized tests and score the outputs. Human evaluation (like Chatbot Arena) asks real people which output they prefer. Benchmarks are more consistent and scalable. Human evaluation captures nuance that benchmarks miss, like whether an answer is actually helpful versus technically correct.
Which LLM comparison site is best for non-technical users?
Chatbot Arena. You do not need to understand benchmarks, APIs, or evaluation frameworks. You ask a question, read two answers, and pick the one you like better. The AI agents and automation landscape can feel overwhelming, but Arena strips it down to a simple side-by-side comparison anyone can use.
