What to Do When Your AI Assistant Gives Different Answers

An AI assistant that gives different answers to the same customer question is difficult to trust. The problem may come from the model, retrieved data, changing prompts, or an unclear business rule—not necessarily from the model being “bad.”

This week, build a small evaluation set and use it to identify the source of inconsistency. You will have a clearer decision about what to fix, which model to use, and where a firm guardrail is needed.

First, separate variation from error

Not every different answer is a problem. If a customer asks for product recommendations, several valid answers may be acceptable. If they ask about a return deadline, delivery fee or warranty condition, variation can create operational and legal risk.

Create two categories:

• Flexible answers: recommendations, explanations, brainstorming and tone of voice.
• Fixed answers: prices, eligibility rules, shipping conditions, refund windows and internal process steps.

The goal is not to make every response identical. The goal is to make every response correct where the business rule is fixed and appropriately useful where several answers are valid.

For example, if a store’s return window is 14 days, an answer saying 14, 15 or 30 days is not acceptable. If the customer asks which of three sofas may suit a small living room, several suggestions could be reasonable.

Build a 30-question evaluation set

Use real questions from support tickets, Instagram comments, WhatsApp messages, sales calls or your product catalogue. Do not create only easy questions. Include the situations that expose weak retrieval and unclear instructions.

A practical set could include:

• 10 common customer questions
• 5 questions involving product attributes
• 5 policy or eligibility questions
• 5 questions with missing or ambiguous information
• 5 questions that the assistant must refuse, escalate or ask to clarify

Label each question with the expected behaviour. The label does not always need to be a complete answer. It may say:

• Must contain: “14-day return period”
• Must not claim: an unverified delivery date
• Must ask: the customer’s country
• Must escalate: a damaged-item dispute
• Must cite: the relevant internal policy or product record

This creates a usable test, rather than a vague review based on whether one person “likes” the answer.

Add realistic language variations

If your business serves Türkiye and the Gulf, write some questions in the languages and styles your customers actually use. Include short messages, spelling mistakes, mixed Arabic and English, Turkish abbreviations and informal phrasing where relevant.

Do not translate every question mechanically. A customer asking “kargo ne kadar?” and another asking “How much is shipping to Riyadh?” may test different data and routing rules, even when the intent is similar.

Run the same questions more than once

Send each evaluation question through the assistant several times using the same version of the prompt and knowledge source. Record the answer, retrieved documents, model used, response time and estimated cost.

For a small business, three runs per question may be enough for an initial check. If you have 30 questions, that produces 90 responses—large enough to reveal patterns without creating a major testing project.

Score each response using a simple rubric:

• 2 — Pass: correct, relevant and follows the required behaviour
• 1 — Review: broadly useful but incomplete, unclear or missing a citation
• 0 — Fail: factually wrong, invents information, exposes restricted content or ignores an escalation rule

Use examples rather than impressions. “Sounds confident” is not a reliable quality measure. “States the correct return period and does not promise a refund before inspection” is measurable.

Diagnose the source before changing the model

When a response fails, classify the failure. This prevents the common mistake of switching models when the real issue is missing or badly structured business data.

If the retrieved information is wrong

The assistant may be receiving an outdated policy, the wrong product variant or several conflicting documents. Check the retrieved passages before editing the prompt.

Useful fixes include:

• Assigning an owner and update date to each policy
• Removing duplicate or obsolete documents
• Separating product records by SKU or variant
• Adding country, language and customer-segment metadata
• Requiring the assistant to say when no trusted source is available

If the correct answer is not present in the knowledge base, no model can reliably retrieve it. Update the source first.

If the information is right but the answer is wrong

The instruction may be ambiguous. Replace broad directions such as “be helpful” with explicit rules:

• Use only the supplied product and policy information for factual claims.
• Do not infer stock, delivery dates or discounts.
• Ask one clarifying question when a required field is missing.
• Escalate complaints involving refunds, damage or suspected fraud.
• Keep the answer in the customer’s language unless asked otherwise.

Then rerun the failed questions. Change one control at a time so you know which adjustment helped.

If quality is good but cost or speed is poor

Do not send every request to the most capable model by default. Create simple routing rules based on risk and complexity.

For example:

• Use a lower-cost model for intent detection, language classification and extracting order numbers.
• Use a stronger model for complex policy explanations, multi-step product comparisons or sensitive escalations.
• Use a fixed response or structured lookup for exact values such as shipping fees and return periods.

These are operating choices, not universal technical rules. Test them against your evaluation set and measure both quality and cost.

Add guardrails for high-risk answers

A good score on common questions does not remove the need for controls. Add a final check before the response reaches the customer.

The check can verify that:

• A price, discount or delivery promise came from an approved source.
• Required fields are present.
• The answer does not reveal internal notes or personal data.
• A refund, complaint or legal issue is routed to a human.
• The language matches the customer’s request.

For fixed business rules, consider using structured data or a deterministic workflow instead of asking the LLM to calculate or remember the answer. The model can explain the result, while the system supplies the value.

A practical evaluation checklist

Use this sequence with your team:

1. Collect 30 real questions and label the expected behaviour.
2. Mark which answers are flexible and which must be exact.
3. Run each question three times and save the full trace.
4. Score responses as pass, review or fail.
5. Classify failures as data, instruction, model, routing or guardrail issues.
6. Fix the highest-risk failure category first.
7. Rerun the same set before changing anything else.
8. Keep the test set as a release check for future prompt, data or model changes.

If you do not have 30 usable questions, begin with 10 high-risk examples and expand the set as real conversations arrive. A small repeatable test is more valuable than a large test that nobody maintains.

How ADMOV can help

ADMOV can help you set up an LLM evaluation workflow around your real customer questions, private business data and operational rules. We can structure the knowledge sources, define retrieval and escalation controls, compare model routes, and create a repeatable review process for Arabic, Turkish and English interactions.

The result is not just a model choice. It is a practical system that shows when the assistant is reliable, when it should ask for clarification, and when a human must take over. Book a free call to review your setup.