What to Do When Your RAG Assistant Retrieves the Wrong Policy

A retrieval-augmented generation (RAG) assistant can give a confident answer from the wrong document. The model may be working correctly; the retrieval layer may have supplied an outdated policy, the wrong market version or an incomplete passage.

You do not need to rebuild the entire assistant to investigate. This week, create a small set of known questions, inspect the retrieved sources and add controls where the failure actually occurs.

Treat the wrong answer as a retrieval incident

Suppose a customer asks, “Can I return a customised sofa after delivery?” Your assistant retrieves a general returns page and answers “Yes, within 14 days.” A separate custom-order policy says customised items are excluded.

The problem is not necessarily that the LLM cannot understand the exception. It may never have received the exception in its context. If the wrong passage is retrieved, asking the model to “be more accurate” will not solve the underlying issue.

Classify the failure before changing prompts:

• Retrieval failure: the correct document or passage was not returned.
• Ranking failure: the correct passage was returned, but a weaker passage appeared first.
• Data failure: the source is outdated, incomplete or contradictory.
• Generation failure: the correct evidence was available, but the answer ignored it.
• Policy failure: the assistant should have refused or escalated instead of answering.

This classification prevents expensive trial and error. Save the user’s question, the retrieved passages, document metadata and final answer for every failed example.

Build a small evaluation set from real questions

Do not begin with hundreds of artificial test prompts. Export a practical sample from support tickets, website searches, sales conversations and unanswered inbox questions. Include questions that contain exceptions, regional differences and missing information.

For example, a store serving Türkiye and the Gulf might create 30 test questions covering delivery areas, returns, warranty, payment methods, installation and product availability. The number is an example, not a requirement; the useful principle is to test the situations that cause business risk.

For each question, record four expected outcomes:

1. The answer: What facts must be included?
2. The source: Which document or policy is authoritative?
3. The scope: Which country, language, product or customer type applies?
4. The fallback: What should happen if the information is missing?

Add difficult cases deliberately. Test “What is your return policy?” as well as “Can I return a customised item bought in Dubai?” A system that performs well only on broad questions is not ready for customer-facing use.

Score retrieval separately from the answer

Use a simple review sheet with three ratings:

• Source found: Was the authoritative passage retrieved?
• Evidence used: Did the answer reflect that passage accurately?
• Action safe: Did the assistant avoid making an unsupported promise?

This lets you see where performance is falling. If the source is missing in most failures, work on chunking, metadata or search. If the source is present but ignored, inspect the prompt, context formatting and model choice.

Fix document structure before changing models

Many RAG problems are caused by documents that are difficult to retrieve. A long PDF containing delivery, returns and warranty rules may be readable to a person but hard to match to a specific question.

Improve the source material first:

• Split large policies into meaningful sections with clear headings.
• Keep each chunk focused on one question or rule.
• Repeat essential scope in the text, such as “UAE returns policy” or “Türkiye installation terms.”
• Add effective dates and review dates.
• Mark superseded documents as inactive instead of leaving them searchable.
• Store product, market, language and customer-type metadata.
• Preserve exceptions with the main rule rather than placing them in an unrelated appendix.

Chunking should follow meaning, not an arbitrary character count. A return rule and its exclusions should usually remain together. If a section is too large, divide it by topic while repeating the conditions required to interpret each part.

Use metadata as a gate, not just a label

Metadata becomes more useful when it filters retrieval. If a user is asking about a UAE order, a UAE policy should be preferred or required before a global policy is considered. If the question concerns a specific product category, do not rank a generic policy above a category-specific rule without a reason.

Be careful with location inference. A customer’s language does not always identify their market. If the market is unclear and the answer changes by country, ask a clarifying question rather than guessing.

Add a source-and-fallback guardrail

Your assistant should not treat every retrieved passage as permission to answer. Give it explicit rules for evidence and uncertainty.

A practical instruction might require the assistant to:

• Answer only from the provided sources for policy questions.
• Mention the relevant market or product scope when it affects the answer.
• Never invent a deadline, price, stock status or eligibility condition.
• Say when the sources conflict or do not contain the answer.
• Ask one clarifying question when the customer’s market or product is unclear.
• Escalate sensitive cases to a human team member.

The fallback should be operational, not vague. “I’m not sure” is less useful than “I can’t confirm whether customised items are eligible for return. Please share your order number so our support team can check the order terms.”

For high-risk topics such as refunds, legal terms, medical claims or financial decisions, route uncertain cases to a person. A slightly lower automation rate is often preferable to an unsupported promise that creates a costly correction later.

Control cost while improving accuracy

Better retrieval does not always require a larger or more expensive model. First reduce the amount of irrelevant context sent to the model. Returning three focused passages is usually easier to evaluate than returning 15 loosely related documents.

You can also separate tasks:

• Use a lightweight step to classify language, market or intent.
• Apply metadata filters before semantic search where possible.
• Retrieve a small candidate set, then rerank only when the question is ambiguous.
• Use a stronger model for complex or escalated cases rather than every message.
• Cache stable answers only when the underlying policy has not changed.

Track cost alongside quality. For example, if an assistant handles 1,000 questions in a month, compare the average retrieval and model cost with the number of human escalations and corrected answers. These are illustrative measures; choose the business outcomes that matter to your team.

A practical RAG repair checklist

Use this sequence for your first investigation:

1. Collect 20–30 real questions that produced risky or incorrect answers.
2. Save the retrieved passages and metadata for each question.
3. Mark the authoritative source and expected fallback.
4. Fix outdated, duplicated or poorly structured documents.
5. Add market, language, product and effective-date metadata.
6. Test filters and retrieval before changing the model.
7. Add rules for uncertainty, conflicts and escalation.
8. Re-run the same questions and record source, evidence and safety scores.
9. Keep the set as a regression test for future changes.

Do not delete failed examples after the assistant improves. They are the most valuable tests you have.

How ADMOV can help

ADMOV can help you set up and integrate an LLM system around your private business data, including document preparation, RAG architecture, model selection, evaluation sets, metadata filters and guardrails. We can also connect the assistant to your existing website, CRM or internal workflow so uncertain answers reach the right person instead of becoming silent errors.

Book a free call to review your RAG setup and identify the first retrieval failures to fix.