LLM Benchmarks Explained: Full List

Your situation	Benchmark	What it measures	Planning use
Checking broad knowledge	MMLU	Multiple-choice questions across many academic subjects, scored zero-shot and few-shot	Use as a general knowledge check between a base model and your tuned version
Checking sentence completion	HellaSwag	Choosing the right continuation of a sentence, with adversarially filtered wrong options	Pair with MMLU to see whether a change affected knowledge, completion, or both
Checking truthfulness	TruthfulQA	Whether a model avoids false answers that resemble common misconceptions	Use when your model will answer factual questions from the public
Checking code generation	HumanEval	LeetCode-style coding problems, reported with Pass@k	Use only if code generation is part of your product
Comparing against other open models	Open LLM Leaderboard	A set of Eleuther-framework benchmarks run through a shared harness	Use to shortlist base models, then re-measure yourself
Measuring your own checkpoint	lm-evaluation-harness	A unified framework for testing generative language models	Use to run the same tasks on your base and fine-tuned models

Benchmark descriptions follow Arize’s guide to LLM leaderboards and benchmarks and Together AI’s overview of evaluating LLMs.

LLM benchmarks explained in one line: each test measures a single narrow skill, so choose the benchmark that matches your use case and check how a score was produced before you trust it.

This list covers the benchmarks a fine-tuner meets most often. For the training side, see how to how to fine tune an llm. Judge-based scoring is a separate method, covered in llm as a judge.

Benchmarks for general knowledge and sentence completion

These two benchmarks appear in almost every model card. They look similar because both are multiple choice, but they test different things, and the difference decides which one you should read.

MMLU

MMLU is the broad knowledge test. Arize’s overview describes it as questions across 57 diverse subjects covering humanities, social sciences, STEM and more, with models evaluated in zero-shot and few-shot settings. Together AI adds that each question asks the model to choose between four multiple-choice options.

It suits fine-tuners who want to confirm that a tuned model still holds broad knowledge. It cannot tell you anything about free-form writing or instruction following, because the model only picks an option. Symflower notes that MMLU is considered to have noisy data and data contamination, so a small gain on it is weak evidence.

HellaSwag

HellaSwag tests sentence completion. Symflower describes a dataset of 70,000 multiple-choice questions, where four choices are offered for how a given sentence might continue. Arize explains that it was built with Adversarial Filtering, which makes it difficult for state-of-the-art models.

It suits you when you want a quick signal about whether a model still picks sensible continuations after tuning. It says little about whether the model knows facts. As with any fixed question set, the contamination risk covered below applies.

Choosing between HellaSwag and MMLU

Both use a pick-one format, so the answer format does not separate them. What separates them is the skill behind the question.

• Choose MMLU when you care about subject knowledge across many fields, for example when comparing candidate base models.
• Choose HellaSwag when you care about whether the model finishes a sentence the way the filtered data expects, for example as a sanity check after tuning on conversational text.
• Report both when you tune a model, because a change that helps one and hurts the other is exactly what you want to notice.

Benchmarks for truthfulness and coding

Knowledge scores do not cover two failure modes that matter in products: confident falsehoods and broken code. TruthfulQA and HumanEval each target one of them.

TruthfulQA

According to Arize, TruthfulQA includes 817 questions covering 38 categories, designed around false answers that resemble common misconceptions. It suits you if your model will field factual questions from the public.

Its limit is scope. It checks whether the model resists tempting wrong answers, not whether it is accurate on your own domain, so a good score does not replace testing on your own material.

HumanEval

HumanEval tests coding abilities through LeetCode-style problems, and Galileo points to Pass@k as the metric developer tools report on it. It suits teams tuning a model for code completion or generation.

Do not read it as a verdict on your codebase. Puzzle-style problems differ from work in a real repository, and like every public problem set it can leak into training data.

Leaderboards and evaluation harnesses

Leaderboards publish scores for many models, and harnesses are the tools that produce those scores. Knowing which is which explains both why leaderboards are useful and where they mislead.

Open LLM Leaderboard

As Arize describes it, the Open LLM Leaderboard assesses open LLMs on four main benchmarks from the Eleuther framework, and Hugging Face relies on the EleutherAI Language Model Evaluation Harness to run them. The same source says the results are easily reproducible and well documented.

It suits shortlisting open base models before you spend compute on tuning. It does not reflect your task, your prompt format or your data.

lm-evaluation-harness

The lm-evaluation-harness is, in Arize’s words, a unified framework designed to test generative language models. Its value for fine-tuners is that the leaderboard’s own tooling is available to you, so you can score your checkpoint the same way. The same source observes that papers rarely include the code or detail needed to replicate their evaluations, which is the gap a shared harness closes.

How far to trust leaderboard scores

Trust them as a pointer, not a measurement of your use case. Together AI reports that when different implementations of MMLU were compared, they got different results and model rankings. Galileo adds that inconsistent hardware, dependencies or prompt templates can skew scores by double-digit percentages.

• Trust the ordering of far-apart models more than small gaps between neighbours, since setup differences can be large.
• Distrust numbers from mixed setups. A score from your own prompt template is not comparable with a leaderboard figure.
• Re-run finalists yourself under one configuration before deciding.

Benchmark contamination

Benchmark contamination is the problem that can quietly invalidate everything above, so treat it as a warning attached to every item in this list.

What it is

Together AI defines data contamination as benchmark questions appearing in a model’s training data. A model that has seen the questions can score well by recall rather than ability. A contaminated score then overstates what the model can do on questions it has not seen.

Detecting and limiting it

Detection works by comparison. The same source describes GSM1K, which contains similar problems that are guaranteed to be novel. The logic carries over to your own work: if a model does well on a public set but worse on equivalent fresh problems, treat the public score with suspicion.

For your own evaluation, Galileo advises guarding against leakage by withholding a final test split that never touches prompt engineering. That keeps at least one score uncontaminated by your own iteration.

Running a benchmark on your fine-tuned model

Use the evaluation harness so your numbers are comparable with published ones, and change as little as possible between the base and tuned runs.

• Pick tasks that match your goal. Choose MMLU for knowledge, HellaSwag for completion, TruthfulQA for truthfulness and HumanEval for code, rather than running everything.
• Keep the setup identical. Score the base model and your checkpoint on the same hardware, dependencies and prompt templates. Hold your llm fine tuning hyperparameters fixed between runs you intend to compare, so a score change traces to one variable.
• Record the configuration. NVIDIA’s benchmarking documentation makes the point that a throughput number means nothing without the configuration that produced it, and advises recording the engine and resolved arguments before collecting data. The same habit applies to accuracy scores.
• Repeat the run. Galileo warns that single test runs mislead and recommends at least three full passes to average out variance.

Read the result as a comparison between your two models on a narrow skill, not as a ranking against the world.

Frequently asked questions

Question	Answer
What is MMLU-Pro?	A harder variant of MMLU. Symflower explains that models must answer multiple-choice questions with 10 choices instead of the 4 in basic MMLU.
What do unspecialized humans score on MMLU?	Arize reports unspecialized humans obtaining 34.5% accuracy, which gives a rough reference point for model scores.
Where do HellaSwag items come from?	Symflower says the questions are based on ActivityNet or WikiHow content.
Can I reproduce a paper’s reported benchmark score?	Often not directly. Arize notes that papers rarely include the necessary code or sufficient detail to replicate their evaluations, so a shared harness is the safer route.