Completeness in LLM evaluation
Did the answer cover everything the question required? Completeness measures the proportion of the required points that the answer actually states. It is the dimension a confident, correct, perfectly grounded answer can fail outright — and the one users notice last, because a partial answer looks exactly like a full one until you act on it.
Overview
Complete next to what
Completeness is undefined without a statement of what the answer needed to hold. So the guide is not optional. In practice that means a key-points list per evaluation query. The facts a good answer must include, written by whoever understands the domain.
Writing that list is the real work, and it is worth doing carefully because it is reusable. The same list drives correctness (are the stated facts right?) and completeness (were they all stated?). So one artefact serves two dimensions.
It also forces a decision your users have already made implicitly. Is the caveat about digital goods a needed point or a nice-to-have? Teams often discover during this exercise that they do not agree on what a good answer is. This is more valuable than any score.
Completeness is scored as the fraction of needed key points that appear in the answer, checked either by an LLM judge or by embedding likeness against each point. It is free of correctness, groundedness, and relevance. An answer can score perfectly on all three and still omit the one fact that makes it actionable. This page walks through a concrete case. This shows how a partial answer scores on each dimension, explains where incompleteness comes from (retrieval failures, aggressive chunking, length limits, and the model stopping early), and covers the practical traps in optimising for it. The temptation to pad answers with each fetched fact. This raises completeness at the cost of relevance and user trust.
Did the answer cover everything the question required?
In this guide
- Complete next to what
- Why the other dimensions cannot see it
- Where incompleteness comes from
- Gauging it, and the trap in optimising it
- Did it cover all of it the question needed?
- Gauging it
- Why answers come out incomplete
- Counting what the answer left out
- Things to try with Completeness in LLM evaluation
- What to recall
- The tension with relevance and concision
- Multi-hop questions
- Practical guidance
- Questions people ask about Completeness in LLM evaluation
- Completeness in LLM evaluation: recap in one screen
Parameters
Visualisation
—Readout
What to watch
- Completeness counts required points covered, not claims made.
- Adding true, grounded, on-topic claims does not raise it.
- It is the mirror image of relevance — read them together.
Completeness in LLM evaluation: A Practical Guide
What is completeness, and why can a perfectly correct answer still fail on it?
Complete next to what
Completeness is undefined without a statement of what the answer needed to hold. So the guide is not optional. In practice that means a key-points list per evaluation query. The facts a good answer must include, written by whoever understands the domain.
Writing that list is the real work, and it is worth doing carefully because it is reusable. The same list drives correctness (are the stated facts right?) and completeness (were they all stated?). So one artefact serves two dimensions.
It also forces a decision your users have already made implicitly. Is the caveat about digital goods a needed point or a nice-to-have? Teams often discover during this exercise that they do not agree on what a good answer is. This is more valuable than any score.
Completeness is scored as the fraction of needed key points that appear in the answer, checked either by an LLM judge or by embedding likeness against each point. It is free of correctness, groundedness, and relevance. An answer can score perfectly on all three and still omit the one fact that makes it actionable. This page walks through a concrete case. This shows how a partial answer scores on each dimension, explains where incompleteness comes from (retrieval failures, aggressive chunking, length limits, and the model stopping early), and covers the practical traps in optimising for it. The temptation to pad answers with each fetched fact. This raises completeness at the cost of relevance and user trust.
Why the other dimensions cannot see it
Think of an answer that states only "Refunds are issued within 14 days." when the question was about the full refund policy. It is correct. It is grounded. It is fully apt. It scores 1.0 on three dimensions and leaves out the condition that makes it actionable.
This asymmetry is the reason completeness is worth gauging apart. The other three dimensions all penalise saying the wrong thing. Only completeness penalises not saying the right thing, and omission is the harder failure to notice,. This is because there is nothing on the screen to catch your eye.
It is also the failure with the worst results in practice. A user who reads a wrong answer may check it. A user who reads a partial answer has no sign that anything is missing and acts on it.
Where incompleteness comes from
Retrieval, most often. If a needed point was never in the fetched context, the model cannot state it without hallucinating. Low completeness alongside low recall@k is a retrieval problem. And no prompt change will fix it. This is the single most common cause.
Chunking. A policy split across two chunks where only one was fetched gives a confidently half-right answer —. This is just what parent-document retrieval and generous chunk overlap exist to prevent.
Length limits. A max-tokens cap or a "be concise" cue trades completeness for brevity, often without anyone deciding to.
The model stopping early. Given ten chunks, models reliably use the first few more than the rest. A needed point in chunk eight is fetched, in context, and still absent from the answer —. This is why MRR is worth tracking next to recall.
Gauging it, and the trap in optimising it
The measurement is mechanical once the key-points list exists. For each needed point, ask a judge whether the answer states it, and take the share. Per point rather than per answer, so the output tells you which point was dropped — that is the actionable part.
The trap is that completeness is trivially gamed by verbosity. An answer that dumps the entire fetched context scores 1.0. If completeness is the only dimension you optimise, you will get long, hedged, exhaustive answers that score beautifully and that nobody wants to read.
Which is why it is read against relevance. The pair behaves like precision and recall: completeness is the recall of needed data, relevance is its precision. And a system is only good when both are high. Reporting either alone is the same mistake as reporting recall without k.
Did it cover all of it the question needed?
Completeness asks whether the answer includes all the data needed. It is the counterweight to relevance: an answer can be perfectly on-topic and still leave out something key.
Question: "What are the eligibility requirements for parental leave?"
Context: "Employees must have 26 weeks' smooth service by the qualifying week, must give 8 weeks' notice, and must provide a MATB1 certificate."
Answer A: "You need 26 weeks' smooth service, 8 weeks' notice and a MATB1 certificate." — complete
Answer B: "You need 26 weeks' smooth service." — correct, grounded, apt, and incomplete
Answer B passes correctness (what it says is true), groundedness (it is backed) and relevance (it fixes the question). Only completeness catches it — which is why the metric exists as a separate measurement.
Incompleteness is more dangerous than being wrong in one set way: a partial answer looks authoritative. And the user has no sign that something is missing.
Gauging it
Completeness needs a guide for what "all of it" means, and there are two sources.
Against a guide answer. Decompose the guide into needed points and check how many appear in the model's answer:
completeness = needed points covered / total needed points
Against the fetched context. Check whether the answer used all the apt data available in the context. This measures the generation stage in detail, without needing hand-written guides.
Question: {question}
Reference answer: {reference}
Model answer: {answer}
List the distinct pieces of information in the reference answer. For each,
state whether the model answer covers it.
Reply as JSON: {"points": [{"point": "...", "covered": true|false}]}Scoring per point rather than per answer is what makes the metric actionable: "3 of 4 needed points, missing the notice period" tells you something a single number does not.
The distinction between the two guides matters diagnostically. If the answer is incomplete next to the context, generation is at fault. If the context itself was incomplete, retrieval is. And no prompt change will fix it.
Why answers come out incomplete
The context was incomplete. Retrieval returned two of the three chunks needed. This is the most common cause, and it is a recall problem — especially for multi-hop questions whose answer spans documents.
The apt chunk was ranked low. Present at spot 9 of 10. Where models attend to it less. The "lost in the middle" effect. A reranker fixes this.
The model stopped early. Answering the first part of a multi-part question and treating that as done.
Length constraints. An cue to be concise, or a low token limit, can truncate needed detail. This is a real tension with relevance.
Chunking split the requirements. A list of five conditions cut after the third. So the fetched chunk looks complete and is not.
That last one is worth watching for, because it is invisible in the answer. The model faithfully reports what it was given, and what it was given was a fragment that reads as whole.
Counting what the answer left out
Completeness is the only dimension that scores what is ABSENT, which is why the others cannot see it. This measures it by decomposing the question, then shows the trap. The metric can be gamed by saying more. And the way out is not a better completeness score.
Once the needed points are listed, the next step is checking how the other three dimensions score an answer that only covers half of them.
That gap is also a trap: completeness alone rewards a model for simply saying more, whether or not the extra words are useful.
The fix is not a better completeness score on its own, but needing completeness and relevance to clear a floor together.
Finally, completeness also surfaces a failure that has nothing to do with generation at all. A multi-hop question whose answer needs facts from two separate documents.
Things to try with Completeness in LLM evaluation
- Three of the five claims are needed points and the guide lists four. Completeness is 0.75 — one needed point is simply absent from the answer.
- Click the off-topic phone claim on and off. Completeness does not move at all: adding material that was not needed cannot improve it.
- Compare with relevance on the same claim list. The claims that hurt relevance are just the ones completeness ignores. This is why the pair has to be read together.
What to recall
Completeness is the share of needed points an answer really states. And it needs a key-points guide to be set at all. It is the one dimension that penalises omission rather than error. This makes it the failure users notice last and act on first. Low completeness is often a retrieval or chunking problem rather than a generation one. Measure it per point so you know what was dropped, and always read it against relevance. Optimised alone it rewards dumping the entire context into the answer.
These four are often measured with an LLM as judge. A strong model is shown the question, the fetched context, the answer and sometimes a guide, and asked to score one dimension at a time. Judging one dimension per call is not a stylistic choice. A single prompt asking for all four gives linked, mushy scores, because the model settles on a general impression and applies it all over.
Two habits make judge scores trustworthy. Ask for a decision and a justification, so a human can audit disagreements. And calibrate against a small human-labelled set before believing any of it. A judge that agrees with your annotators 70% of the time is not gauging what you think it is gauging.
The tension with relevance and concision
Completeness and relevance pull in opposite directions, and pushing either too hard damages the other.
| High completeness | Low completeness | |
|---|---|---|
| High relevance | The target | Terse, missing detail |
| Low relevance | Padded, buried answer | Poor on both |
Two ways to have both:
Structure the answer. Direct answer first, then the backing detail. Nothing is missing, and nothing is buried. Needing a set format — a short answer then a bulleted list of conditions — reaches this reliably.
Distinguish needed from optional. Instruct the model to include all of it the question asked for and nothing it did not. That is a narrower cue than "be thorough", which gives padding.
The practical target is not maximum completeness. It is enough completeness — all of it the question needed, and no more.
Multi-hop questions
The case where completeness fails hardest is a question whose answer needs combining two documents.
"Which of our offices are in countries where the standard VAT rate is above 20%?"
That needs the office list and the VAT rates, which are almost certainly in different documents. A single retrieval pass ranked by likeness to the whole question may return one and not the other,. This is because neither document is a good match for the combined query.
Three approaches:
Decompose the question into sub-questions, retrieve for each, and combine the context. The most reliable fix.
Iterative retrieval — retrieve, let the model find what is still missing, retrieve again. More capable and slower.
Increase k and hope both appear. Cheap, unreliable.
Gauging completeness apart is what makes multi-hop weakness visible. On single-hop factoid sets it looks fine; on multi-hop questions it collapses. And the aggregate hides which.
Practical guidance
Score per point, not per answer. The list of missed points is the actionable output.
Measure against both the guide and the context. The gap localises the failure to retrieval or generation.
Include multi-part and multi-hop questions in the evaluation set, on purpose. A set of single-fact questions will show high completeness and tell you nothing.
Track it alongside relevance and answer length. A completeness improvement that doubled the answer length may not be an improvement.
Watch chunk boundaries for lists and conditions — splitting an enumerated list is a silent completeness failure.
Questions people ask about Completeness in LLM evaluation
Is completeness the same as recall? Related but distinct. Recall is about fetched documents, completeness about the made answer's coverage.
How do I know what "complete" means? From a guide answer, or from the apt data present in the fetched context. The second needs no hand-written guide.
Does asking for longer answers improve it? It improves coverage and damages relevance. Structure the answer instead.
Why do multi-part questions fail? The model fixes the first part, and retrieval often only covered that part. Decompose them.
Should a refusal be scored on completeness? Not meaningfully. Exclude refusals and track them as their own metric.
What is a good score? Above 0.9 on single-hop questions is achievable; multi-hop is often much lower and is where the useful work is.
Completeness in LLM evaluation: recap in one screen
- Completeness asks whether the answer covers all of it the question needed.
- A partial answer can pass correctness, groundedness and relevance — only this metric catches it.
- Score per needed point; the list of missing points is what tells you what to fix.
- Measure against both the guide and the fetched context to localise the failure.
- It pulls against relevance — structure the answer rather than lengthening it, and test multi-hop questions on purpose.