A Driver's License Question Beat Every Real One: Why Retrieval Scores Cannot Gate a RAG System

Most retrieval-augmented generation systems carry a comfortable assumption. If the retrieval score is high, the retrieved context is probably relevant and the answer is probably grounded. If the score is low, the system is out of its depth and should refuse. Set a threshold somewhere sensible, and you have a safety mechanism.

I built a small Romanian question-answering system, measured that assumption, and it did not survive contact with the data.

This piece is the measurement, the reason it breaks the standard design, and what I ended up gating on instead. The code is public and the result is reproducible in about a minute on a laptop: ro-cited-answers on GitHub.

What the system is

A question-answering demo over five documents on Romanian customs and EORI registration procedure, drawn from public information published by Poșta Română and the Romanian Customs Authority.

Everything runs locally. An open-weight 2B model on Ollama, an embedding model for retrieval, on a four-core VPS in the EU. No external API, no key, no data leaving the machine. The design rule is that an answer which cannot be traced to a source document is refused before anyone reads it.

I did not build it as a product. I built it because a parcel of mine was genuinely stuck in customs at BSI București, and the questions in the test suite are the ones a real person actually needed answered.

The measurement

The script scores seven legitimate questions and seven impostors against the corpus. Legitimate questions are about customs, EORI numbers, parcel returns. Impostors are about things the corpus knows nothing about.

If a retrieval score is a usable safety signal, the two populations should separate cleanly. Every real question should score above every impostor, and the gap between them is where you put the threshold.

They do not separate. They overlap.

	Question	Score
Lowest legitimate	"Cand am nevoie de cod EORI pentru un colet?"	0.697
Highest impostor	"Ce documente imi trebuie pentru un permis de conducere?"	0.773
Separation margin		-0.076

A question about driver's license paperwork scored higher against a Romanian customs corpus than every genuine customs question did.

The margin is negative. That is the whole finding. It is not that the threshold was badly tuned, or that a bit more calibration would fix it. When the populations overlap, there is no value you can pick that admits every real question and rejects every impostor. One threshold is too strict and loses real answers. Any looser one lets the driver's license question through.

Why this breaks the common design

A lot of RAG systems treat the retrieval score as the guardrail. Retrieve, check the similarity, and if it clears the bar, hand the context to the model and trust that grounding follows.

If the score cannot distinguish a question your corpus can answer from one it cannot, then that guardrail is decorative. It will fire on some things and feel like it is working, which is worse than not having it, because it produces confidence without producing safety.

Note what the impostor question does. It is in the same language, the same register, the same bureaucratic domain of official documents and required paperwork. Embeddings capture that similarity faithfully. They are doing their job. Asking a similarity score to also tell you whether a specific fact is present in a specific document is asking it a question it was never designed to answer.

So in this system the retrieval gate is deliberately not load-bearing. Its threshold is set from the measurement above, it catches the obviously unrelated case, and nothing important rests on it.

What to gate on instead: the answer, not the confidence

If you cannot trust a score computed before the answer exists, check the answer itself after it exists.

Four gates run in sequence, and an answer must clear all of them or it is refused, with the refusal logged.

Gate A, retrieval. The best-matching document must clear a similarity threshold. Deliberately weak, for the reason above.

Gate B, citation. Every source the model points at must resolve to a document that was actually retrieved. Catches invented citations.

Gate C, numbers. Every number in the answer must appear verbatim in a cited source. Numbers are where a hallucination does financial damage, and they are cheap to check exactly.

Gate D, grounding. Every substantive sentence must be attributable to one cited document, and the printed source list is rebuilt from the documents that actually carried a sentence.

Gate D is the load-bearing one, and the reason is worth stating plainly.

Gates that depend on the model cooperating with an instruction are brittle. You can tell a model to cite its sources in a particular format, and a large model will usually comply, and a 2B model frequently will not. Gate D does not ask the model for anything. It takes the answer as a finished artifact and checks it against the source text. It works the same whether the model cooperated, ignored the instruction, or never understood it.

That is the transferable idea here, and it is the same rule I run in production diligence work: cite the source or cut the claim. Check the output against the evidence, not the model against its instructions.

The same model, twice, minutes apart

Here is what the gates are actually for.

Ungrounded, asked what EORI means, the model answered that it is a "European Union Chamber of Commerce and Industry Number."

That is invented. EORI is not a chamber of commerce number. The answer is fluent, plausible, correctly shaped, and wrong, which is the exact failure mode that costs money in regulated work.

Grounded through the four gates, same weights, same machine, minutes later, it answered that EORI stands for Economic Operators Registration and Identification, that the number is assigned free of charge, and that the average processing time is 5 days with a maximum of 30. Then it printed the source document and the date that source was last checked.

Same model. Same hardware. The gates decided everything.

This is also why I am unconvinced by the framing that sovereign AI, by itself, gets you trustworthy AI. There is a serious argument in Europe about open weights, national compute, and models governed under national law, and it is worth winning. But sovereignty decides where the weights sit. It decides nothing about whether a particular answer is true. A model can be entirely sovereign and confidently wrong at the same time.

Three things that cost real time

A small model will not echo a string source id. Asked to cite a document by its identifier, the model answered the question correctly and signed it "SURSE: 1, 2." A gate demanding string ids threw away two correct answers. The fix was to number the fragments and map the number back to the real document in code, where it cannot be got wrong. If a gate can be enforced in code instead of in a prompt, enforce it in code.

Once citing is cheap, models pad their citations. With numbered fragments, the model answered a parcel-return question correctly from the right document, then also cited an unrelated one. The answer was trustworthy and the citation list was noise, which is precisely backwards for a system whose entire value proposition is traceability. Citations are no longer taken on the model's word. Each sentence is attributed to its best-matching cited document, the printed list is rebuilt from what actually carried a sentence, and padding is stripped and logged.

Morphology is not ungroundedness. Romanian inflects heavily. Exact word matching scored a verbatim quote at exactly 0.5000 against a 0.50 threshold, which passed only because the comparison was written as less-than rather than less-than-or-equal. That is not a safety margin, that is luck. Prefix matching on content words lifted the same quote to 0.83, while a genuinely ungrounded sentence still scored 0.00.

Every decision, accepted or refused, is appended to an audit log with the retrieval scores, the gate that fired, and the per-sentence attribution. A refusal you cannot reconstruct afterward is just a different kind of unverifiable output.

What this does not do

I would rather state the limits than have you find them.

It is slow. CPU-only inference on four shared cores runs at roughly 2 tokens per second. Fine for a demonstration, not fine for production traffic.

The grounding is lexical, not semantic. Gate D compares content words by a fixed-length prefix, so it forgives Romanian inflection but not synonymy. A correct paraphrase sharing little vocabulary with its source would be refused. The system fails toward refusing, which is the right direction, but it is a real limitation and not a subtle one.

The corpus is five documents. The overlap finding in particular should be re-measured on a larger corpus before anyone treats it as a general result. I am reporting what I measured, not a law of nature.

A refusal is not an answer. Refusing well is the point, but a system that refuses everything is also useless. Of the four test cases, two must be refused and two must be answered, precisely to hold both sides honest.

None of this is legal advice, and the corpus is a demonstration fixture.

The part worth taking away

If you are building anything that answers questions from documents and the stakes are higher than trivia, the question to ask about your own system is simple. When it refuses, what is it actually checking?

If the answer is a retrieval score, go measure whether your in-corpus and out-of-corpus questions actually separate. Score a handful of real questions and a handful of plausible impostors from an adjacent domain and look at the two populations. It takes an afternoon. You may find, as I did, that the number you have been trusting cannot tell the two apart.

Then move the check to where it can be enforced: the finished answer, against the source text, in code.

That is the same discipline behind the diligence work we do, where a figure in a hundred-page deal document either traces to something real or it gets flagged rather than repeated. If you want to see it applied to acquisition documents instead of customs procedure, there is a cited sample brief with every claim tied to its source, no login required.

The repository is MIT licensed and the measurement script is the first thing worth running: github.com/MariusGithub13/ro-cited-answers.