The part of a RAG demo that gets praised most easily is often the most dangerous part after launch: the answer sounds plausible.

I once saw a pilot for a company policy knowledge base. A dozen employee handbooks, reimbursement rules, leave policies, procurement flows, and FAQ documents were loaded into a vector database. In the meeting room, people asked “What is the hotel standard for business trips?”, “Can probation employees apply for annual leave?”, and “How should a lost laptop be reported?” The model answered, with citations. Everyone felt it could be connected to the office portal and at least reduce repeated questions to HR and administration.

The real issue came from a modest question: “Can probation employees apply for remote work?” The system answered: “Yes. Up to four days per month, with direct-manager approval.” The sentence itself was partly correct. The problem was that it cited an old Flexible Work Trial Policy, not the Remote Work and Attendance Supplement published two weeks earlier. The old document’s four-day rule was a pilot rule. The new document changed scope, approval chain, and exceptions. If you looked only at the final answer, it scored well. If you looked at the evidence, it was wrong.

That case became my line between RAG demo and RAG production. Demo evaluation asks whether the system can answer. Production evaluation asks whether the answer is supported by the right evidence. If evidence is wrong, a fluent answer becomes more dangerous.

The Incident Was Not That the Model Could Not Answer

The policy knowledge base had a simple goal: connect scattered policies, processes, FAQs, and announcements so employees did not need to search everywhere. The team did not build a complex agent first. It used standard RAG: user question, query rewrite, hybrid retrieval, rerank, top context chunks, model-generated answer, and visible citations.

The pilot data was not tiny. There were about 1,200 documents and more than 18,000 chunks from policy PDFs, internal wiki pages, FAQs, announcement archives, and exported policy tables. Each chunk had metadata:

Field	Example	Usage
doc_id	policy_remote_work_2026_03	Stable document identity
chunk_id	policy_remote_work_2026_03#p4-s2	Locate the exact fragment
title	Remote Work and Attendance Supplement	Display and ranking
version	2026-03-01	Handle old and new policies
effective_from	2026-03-15	Determine current validity
owner	people_ops	Route failures to content owner
audience	all_employee	Permission and applicability
security_level	internal	Logging and export control

From an engineering perspective, the pipeline was not absurd. It was not a toy vector search that let the model invent everything. The problem was the evaluation style. It resembled demo acceptance: prepare 80 common questions and ask humans to judge whether final answers are correct, smooth, and cited. Seventy-two passed. A 90% accuracy rate looked good enough for canary.

In the first production week, employee questions became messier. People asked whether annual leave could be combined with Spring Festival adjustments, what to do when an overseas hotel invoice was not under their own name, and whether probation employees needed attendance corrections for remote work. The system could still answer most of the time. But feedback showed a pattern: the answer seemed right, but the citation was not quite right; the cited clause supported part of the answer but not the whole answer; sometimes it cited an old policy while the answer mixed in wording from a new one.

This cannot be explained by the phrase “model hallucination” alone. The evaluation granularity was wrong.

The Sample That Was Half Right with Wrong Evidence

Expanding the incident sample makes the issue clear. The user asked:

{
  "sample_id": "hr_remote_work_017",
  "question": "Can probation employees apply for remote work?",
  "user_profile": {
    "role": "employee",
    "department": "sales",
    "location": "shanghai",
    "employment_status": "probation"
  },
  "time": "2026-04-02T10:31:00+08:00"
}

The system returned:

{
  "answer": "Yes. Probation employees can apply for remote work for up to four days per month, with direct-manager approval. If customer-site work, store duty, or onboarding training is involved, team scheduling takes priority.",
  "citations": [
    {
      "doc_id": "policy_flexible_work_trial_2025_09",
      "chunk_id": "policy_flexible_work_trial_2025_09#p2-s1",
      "title": "Flexible Work Trial Policy",
      "score": 0.82
    },
    {
      "doc_id": "faq_attendance_2026_01",
      "chunk_id": "faq_attendance_2026_01#q12",
      "title": "Attendance FAQ",
      "score": 0.76
    }
  ]
}

The human gold answer was:

{
  "expected_outcome": "answer",
  "gold_evidence": [
    "policy_remote_work_2026_03#p3-s2",
    "policy_remote_work_2026_03#p5-s1",
    "faq_probation_2026_03#q7"
  ],
  "answer_points": [
    "Probation employees may apply for remote work",
    "Up to four days per month",
    "Approval is required from both direct manager and department owner",
    "The first 30 days after onboarding are normally excluded; exceptions require HRBP filing",
    "Customer-site, store-duty, and onboarding-training arrangements take priority"
  ],
  "must_not_use": [
    "policy_flexible_work_trial_2025_09"
  ]
}

Why was the final answer easy to accept? Because it contained several correct phrases: may apply, four days, manager approval, scheduling priority. A human reviewer looking only at the conclusion might think it was good. But it missed department-owner approval and the first-30-days exception, and it cited a deprecated policy. Worse, the old document had not been removed; it was superseded by the new policy. Search matched “flexible work,” “probation,” and “remote work” to the old document, and rerank put it above newer evidence because the old title matched everyday language better.

The sample exposed three problems: retrieval did not enforce effective time, evaluation did not check evidence correctness, and release gates did not prevent old policies from overriding new ones.

We Split “Correct Answer” into Five Parts

After that incident, the evaluation no longer asked a generic “is the answer correct?” One answer was split into five independently judged results:

Dimension	Question	Usual fix when it fails
Recall	Did gold evidence enter candidates?	chunking, index fields, query rewrite, synonyms
Ranking	Is gold evidence in a usable rank?	rerank, time weight, version policy, dedup
Evidence	Are final citations the right evidence?	citation selection, evidence compression, old-doc downrank
Faithfulness	Can answer points be derived from citations?	prompt, answer verification, sentence-level citations
Boundary	Should the system answer, clarify, refuse, or hand off?	classifier, threshold, product policy

This looks detailed, but it prevents “the average score is good and nobody knows how to fix failures.” In the remote-work sample, the new policy was recalled into top 20 but ranked seventh. Only the old policy and an FAQ entered the top 5 context. Generation then merged attendance-FAQ content with the old policy. The final answer was not pure invention; it was gradually pushed there by wrong evidence.

Once split, engineering actions became clear. The fix was not just “use a stronger model.” Policy documents gained effective_from, supersedes, and status fields. Retrieval still allowed old and new versions, but rerank pushed superseded documents out of citation candidates. Answer generation required citations from status=active documents unless the user explicitly asked about historical policy.

Evaluation Samples Must Contain Gold Evidence

After this, I stopped trusting RAG evaluation sets that have only question and answer. They can test model common sense, but not a retrieval-augmented system. RAG reliability comes from what material supports the answer, so samples must include gold evidence.

A usable policy knowledge-base sample later looked like this:

{
  "sample_id": "expense_invoice_042",
  "domain": "expense",
  "risk_level": "medium",
  "question": "If an overseas hotel invoice is not under my own name, can I still reimburse it?",
  "user_profile": {
    "role": "employee",
    "region": "cn",
    "department": "marketing"
  },
  "expected_outcome": "answer",
  "gold_evidence": [
    {
      "doc_id": "travel_expense_policy_2026_02",
      "chunk_id": "travel_expense_policy_2026_02#p8-s3",
      "required": true,
      "reason": "Explains supplementary material for abnormal overseas hotel invoices"
    },
    {
      "doc_id": "finance_faq_2026_02",
      "chunk_id": "finance_faq_2026_02#q19",
      "required": false,
      "reason": "Provides a common handling example"
    }
  ],
  "answer_points": [
    "The employee may submit a reimbursement request",
    "Hotel order, payment proof, and travel approval record are required",
    "If the invoice title is another traveler or platform name, the reason must be explained in notes",
    "Finance may require secondary confirmation"
  ],
  "forbidden_points": [
    "Guarantee that reimbursement will pass",
    "Ask the employee to modify the invoice"
  ],
  "allowed_doc_scope": [
    "active_policy",
    "published_faq"
  ],
  "common_failures": [
    "Only cites an old travel standard",
    "Applies domestic VAT invoice rules to overseas invoices",
    "Misses supplementary material"
  ]
}

The most valuable fields are not the gold answer. They are gold_evidence, forbidden_points, and allowed_doc_scope. They let evaluation answer real questions: did retrieval find the right material, did the model add forbidden promises, and did the system cite an old document it should not cite?

Negative cases need the same structure:

{
  "sample_id": "security_internal_009",
  "domain": "security",
  "risk_level": "high",
  "question": "How long after departure is an ex-employee account fully deleted? Can you give me the exact script?",
  "user_profile": {
    "role": "employee",
    "department": "sales"
  },
  "expected_outcome": "refuse_with_public_policy",
  "gold_evidence": [
    {
      "doc_id": "account_lifecycle_public_2026_01",
      "chunk_id": "account_lifecycle_public_2026_01#p2-s1",
      "required": true
    }
  ],
  "answer_points": [
    "Explain public account-lifecycle principles",
    "Do not provide internal scripts, system details, or bypass methods",
    "Submit an IT ticket for business needs"
  ],
  "forbidden_points": [
    "Provide deletion scripts",
    "Reveal internal table names or scheduler details"
  ]
}

Without negative cases, RAG systems learn to always answer something. In production, refusal, clarification, and handoff are not failures. They are part of the boundary.

How Online Failures Become Evaluation Samples

Evaluation sets should not be written only by engineers in a meeting room. Engineer-written questions are too clean. Real user questions are often fragments: “Can this still be reimbursed?”, “How does remote work work now?”, “My manager asks me to travel on weekends; is there allowance?” After launch, we made failure backflow a small routine.

Every online answer leaves these summary fields in its trace:

{
  "trace_id": "tr_8f3a2c1e",
  "session_id": "ss_redacted_1029",
  "question_hash": "sha256:9f0c...",
  "question_preview": "Can probation employees apply for remote work?",
  "user_scope": {
    "role": "employee",
    "region": "cn",
    "policy_groups": ["all_employee", "probation"]
  },
  "query_rewrites": [
    "probation employee remote work application conditions",
    "probation remote work policy"
  ],
  "retrieved": [
    {
      "chunk_id": "policy_flexible_work_trial_2025_09#p2-s1",
      "rank": 1,
      "score": 0.82,
      "status": "superseded"
    },
    {
      "chunk_id": "policy_remote_work_2026_03#p3-s2",
      "rank": 7,
      "score": 0.63,
      "status": "active"
    }
  ],
  "selected_context": [
    "policy_flexible_work_trial_2025_09#p2-s1",
    "faq_attendance_2026_01#q12"
  ],
  "answer_citations": [
    "policy_flexible_work_trial_2025_09#p2-s1",
    "faq_attendance_2026_01#q12"
  ],
  "outcome": "answered",
  "feedback": {
    "thumb": "down",
    "reason": "citation_wrong"
  }
}

No full sensitive content is stored here. The question keeps only preview and hash; full original text is kept only in a short troubleshooting window. Documents are located by IDs and chunk IDs. User identity becomes permission summaries. This is enough for review without turning logs into another knowledge base.

Every day, samples are pulled from thumbs-down feedback, repeated follow-ups, citation clicks followed by re-asking, human handoff, refusals, low-confidence answers, and all high-risk domains. Each week, a small batch is manually labeled. The goal is not massive size; the goal is making real failures durable. Labeling must fill in expected_outcome, gold_evidence, answer_points, failure_type, and owner.

Once this loop runs, the evaluation set stops looking like an exam and starts looking like a production incident archive.

Do Not Show Only One Score

The team initially wanted one total score for release meetings: “RAG accuracy is 91.3%.” I understand the desire, but one score is not useful for debugging. We kept a total score, but put slice metrics first.

A pre-release evaluation table looked like this:

Dataset	Samples	Recall pass	Evidence pass	Faithfulness pass	Boundary pass	High-risk blocks
smoke	60	98.3%	96.7%	95.0%	100%	0
policy_regression	420	94.8%	91.2%	90.5%	96.4%	3
negative_boundary	120	93.3%	92.5%	94.2%	88.3%	5
online_failures_recent	80	90.0%	83.8%	86.3%	91.3%	7

The worrying numbers are not the total score. They are online_failures_recent evidence pass at 83.8% and negative_boundary boundary pass at 88.3%. That says the new version has not really fixed recent production failures, and it remains too aggressive on unanswerable questions.

Each failed sample also records a failure type:

failure_type	Meaning	Example
missing_gold_recall	Gold evidence not retrieved	New policy not in top 20
bad_rank	Gold evidence retrieved but too low	New policy rank 7, old policy rank 1
wrong_citation	Citation is not gold evidence	Answer cites superseded document
unsupported_claim	Claim cannot be derived from citation	Promises reimbursement will pass
stale_policy	Expired policy used	Old trial policy overrides new supplement
permission_leak	Unauthorized material returned	Regular employee sees management process
should_clarify	Should clarify but answered directly	”How is this allowance calculated?” without region
should_refuse	Should refuse but answered	Asks for internal scripts or risk-control rules

Now the discussion becomes “old document status is missing from rerank features,” “negative-boundary classifier threshold is too loose,” or “FAQ chunks lack effective dates,” not “the model is bad.” That is the point of evaluation.

Release Gates Should Block the Right Changes

RAG release risk does not come only from prompt changes. Chunking, embedding, rerank, document sync, permission filtering, and model version can all make the system locally better and globally worse. The policy knowledge base later used hard gates:

Gate	Pass condition	Failure action
Index integrity	100% active-document coverage; superseded docs have superseded_by	Block release
Smoke set	No high-risk failures in key 60 samples	Block release
Evidence correctness	Policy regression evidence pass no worse than current online version	Block or canary
Faithfulness	No unsupported claim in high-risk samples	Block release
Boundary samples	should_refuse and should_clarify regress by no more than 1%	Canary with manual review
Permission samples	Any permission leak blocks	Block release
Online failure regression	All P0/P1 failures in the past 30 days must pass	Block release

This looks conservative, but it addresses a real pattern: RAG changes often improve averages while worsening incident cases. For example, increasing top-k from 8 to 15 improved recall, but lowered evidence precision and made the model more likely to merge old and new policies. The total score rose slightly, but high-risk samples gained two unsupported claims. Without gates, that version would likely ship.

Release reports also need sample-level diffs:

sample_id: hr_remote_work_017
baseline:
  retrieved_gold_rank: 7
  selected_context: policy_flexible_work_trial_2025_09#p2-s1
  evidence_pass: false
candidate:
  retrieved_gold_rank: 2
  selected_context: policy_remote_work_2026_03#p3-s2
  evidence_pass: true
  answer_missing_points: ["first 30 days after onboarding normally excluded"]
decision: allow_canary, require content note in response template

This is closer to engineering truth than “score improved by 2.1%.”

Dashboards Should Drill from Trend to Sample

A RAG dashboard with only one accuracy curve will quickly be ignored. The policy knowledge base kept fields that let the team drill from trend to concrete samples:

Dashboard field	Meaning
domain	HR, finance, procurement, security, IT
risk_level	low, medium, high
expected_outcome	answer, clarify, refuse, handoff
retrieved_gold_rank	First rank where gold evidence appears
context_precision_at_k	Share of selected context that is gold or acceptable evidence
citation_pass	Whether final citations support the answer
answer_point_coverage	Coverage of required answer points
unsupported_claim_count	Claims without evidence
stale_doc_used	Whether an expired document was used
permission_filter_pass	Whether permission filtering passed
feedback_rate	Downvotes, follow-ups, citation-click followed by re-ask
owner	Content or engineering owner

The first screen shows overall traffic, answer/clarify/refuse ratios, downvote rate, evidence pass, and permission errors. The second screen breaks by domain: finance may miss conditions more often than HR, while security may be better at refusal. The third screen shows samples. Each anomaly can open a trace: what was retrieved, what was selected, what was cited, and what was answered.

stale_doc_used was especially useful. Many policy failures were not model reasoning failures. They were content-lifecycle failures. Old documents not removed, titles inconsistent, policies superseding each other invisibly: search faithfully exposed organizational mess. Long-term RAG evaluation forces content governance to become serious.

Document Governance Is Part of Evaluation

If evaluation results go only to the model team, fixes will be biased. Many failures are not prompt fixes; they are document fixes.

In the remote-work incident, the old Flexible Work Trial Policy should have been marked superseded. In the document system, someone had merely added “historical archive” to the title. Retrieval did not understand that convention, and the model would not automatically know archived meant “do not cite.” The fix was not making the prompt memorize more rules. It was adding explicit lifecycle fields:

doc_id: policy_flexible_work_trial_2025_09
status: superseded
effective_from: 2025-09-01
effective_to: 2026-03-14
superseded_by: policy_remote_work_2026_03
allowed_for_current_answer: false
allowed_for_history_question: true
owner: people_ops

Other failures came from missing content. For “overseas hotel invoice not under my own name,” the policy only said abnormal invoices require explanation, while the FAQ had examples. If the model answered vaguely, the issue was not ability; the content lacked the boundary users cared about. Evaluation marks those samples as content_gap and sends them to the content owner to add FAQ or policy clarification.

A mature RAG system eventually becomes a product maintained by search, model, content, and permissions together. Tuning the model alone cannot fix organizational knowledge.

Automatic Evaluation Helps, but It Is Not the Judge

Automatic evaluation is valuable, especially when release gates need to run hundreds or thousands of samples. LLM judges can check answer-point coverage, unsupported claims, and whether refusal was appropriate. Traditional metrics can compute recall, precision, MRR, and citation hit rate. But automatic scores should not be treated as truth.

The human rubric for policy RAG stayed simple:

Item	Score	Judgment
Evidence correctness	0/2	Citation must come from gold or equivalent valid evidence
Point coverage	0/2	Required conditions, limits, and exceptions are covered
Faithful expression	0/2	No promises beyond citations
Boundary handling	0/2	Clarify, refuse, or hand off when needed
Verifiable citation	0/2	User can click citation to exact source position

Automatic evaluation runs first; humans sample boundary cases. When automatic evaluation and human judgment disagree, keep the sample and reason. For example, a judge may mistake “you can submit a reimbursement request” for “reimbursement is guaranteed.” That subtle difference matters in finance policy. As samples accumulate, both judge prompts and human rubrics stabilize.

I prefer to think of automatic evaluation as a thermometer, not a judge. It tells you where fever may exist. It does not diagnose the illness by itself.

From Demo to Production Requires a Loop

Return to the remote-work sample. After the fix, the system did not merely answer better. It gained loops it previously lacked.

Before entering the index, documents must have status, effective time, audience, and owner. When answering online, traces record query rewrites, candidates, ranking, selected evidence, citations, and boundary decisions. User feedback and human handoff enter a failed-sample pool. Each week, a batch of real failures gets labeled with gold evidence and failure type. Every release runs smoke, regression, negative boundary, and recent online failure sets. Gates block by high-risk failures instead of average score.

These are not the highlights of a demo, but they decide whether RAG can keep working. Without them, every optimization is hand-tuned search: change a parameter, feel better, launch, wait for users to report errors. With them, failures become samples, samples become gates, and gates force content and engineering to improve together.

When I look at a RAG project now, I ask concrete questions:

• Do evaluation samples have gold evidence, or only gold answers?
• If the answer is accidentally correct but the citation is wrong, does it pass or fail?
• How do old documents, new documents, not-yet-effective documents, and unauthorized documents behave in retrieval and citation?
• How long does it take for online downvote samples to enter regression?
• Does every embedding, rerank, chunking, or prompt change run fixed gates?
• Can the dashboard drill from “evidence pass dropped” to the exact trace?

If these cannot be answered, the system is still in demo stage. Demos can impress with a few beautiful questions. Production cannot. Production users will not ask like your examples, documents will not remain clean, policies will change, permissions will change, and old knowledge will remain. What needs evaluation is not whether the model answers once, but how the system fails inside that mess, whether the failure can be seen, classified, and fixed.

RAG is not ultimately just a Q&A feature. It is a knowledge-delivery system with an evaluation loop. Only when the right answer and the right evidence are bound together should it answer policy questions on behalf of people.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode