When Your AI Agent Lies to You: Tackling Real-World LLM Hallucinations

Authors: Sravan Jayanthi, Joscha Koepke

Overview

We @ Connectly AI have been deep in the weeds of building sales and customer support conversational AI since the dawn of the ChatGPT era. After developing several AI agents in production, including Sofia Lite & Sofia V3, we've systematically identified the key pain points around hallucination and reliability. This is what led us to develop Sofia QA, the latest AI product from Connectly AI, which is built on the wealth of real-world data and learnings from AI agents in production to ensure high reliability and accuracy within the bot. Through extensive work done to diagnose and mitigate hallucination within the design and architecture of the product, Sofia QA has had strong adoption and stickiness with 77% answer rate across 25 thousand user sessions and customer feedback highlights a 5/5 setup experience, effective inquiry resolution, and improved answer quality.

In this blog, we attempt to precisely define the failures we observed and our strategy for evaluating and catching these during the design and implementation of the Sofia QA.

Background

In this section, we highlight the major shortcomings of the previous iterations of the Sofia bot and categorize the nature of the failure modes. Hallucinations tend to be used as an overloaded catch-all term that describes a wide-range of behaviors. We define a language modelhallucination as when a model performs behaviors or makes statements that cannot be verified or supported.

Sofia V3, an Autonomous Agentic Architecture

Read to learn more about Sofia V3

The unconstrained nature of Sofia V3's agentic architecture made it difficult to achieve reliable and predictable behavior across a wide range of tasks, from RAG-form question answering to image-based product recommendations. This lack of constraint also resulted in a lack of explainability regarding generated instructions and LLM execution strategy, which in turn made it challenging to train non-experts on the bot's capabilities and how to fine-tune its behavior.

Table 1: Analysis of Agentic Failures

Analysis of agentic failure modes

*Links for above*: AmiEnt, CLAMBER, mCoT, MAGICORE, Reflexion, PARC, HaluEval 2.0, CDQG, LLM Effective Negotiators, Hallucination Survey, PersuaBot, HiAgent, Sierra AI's Tau-bench, Claude 3.7, RuleBench

Table 2: Analysis of non-expert configuration challenges

Difficulty for non-experts to configure an agentic AI app

Sofia QA

Sofia QA was designed to tackle all these major challenges by providing factual, knowledge-based answers, reducing the workload on customer agents, and delivering clear analytics to showcase its impact. It is reliable, easy to set up, self-improving, and maintainable without significant engineering support.

Attribution Strategies

Attribution techniques for Large Language Models (LLMs) aim to improve factual reliability by linking generated text to verifiable sources. These methods ensure that responses are backed by provided content, reducing the underlying risk of hallucination.

These were the 3 strategies we explored for attribution:

1. In-line Response Citations -- This approach integrates citations directly into responses as they are generated, improving transparency. The Nova RAG Prompt Design structures prompts to enforce inline attributions by guiding LLMs to reference retrieved sources explicitly.
2. Chain-of-Thought -- CoT has the LLM decompose its thinking process, ensures it is using citations linking to high-quality, relevant evidence rather than arbitrary or low-relevance sources. This method adds additional reasoning on-top of the in-line citation approach.
3. Post-attribution (Citation Generation with LLM Attribution or Entailment Model) -- Rather than embedding citations during generation, this approach applies an attribution model after the response is drafted. LLMs or entailment models like RoBERTa or other hallucination evaluation models (Patronus, MiniCheck) to analyze statements, match them to retrieve documents, and insert citations post-hoc.

Experimentation

We leverage Metrics-Driven Development to have a measurable baseline dataset to design and optimize our system against. We perform this experimentation on a sample customer dataset of over one hundred and twenty one questions in Spanish. Development of this RAG framework would require identifying the strongest retrieval, citation, and answer generation methods for this specific use-case of multilingual conversational RAG.

A. Retrieval Quality

We explore 3 retrieval approaches:

• One-shot retrieval: This technique retrieves information in a single step, using a hybrid search approach that combines sparse and dense vector search. This is the simplest and fastest retrieval technique.
• Reranking: This technique involves an initial retrieval step, followed by a reranking step that uses an LLM re-ranker to reorder the retrieved results. This technique can improve the quality of the results, but it can also be noisy and add additional time.
• Filtering: This technique uses an LLM to filter the retrieved results, removing any results that are deemed to be irrelevant. This technique can also improve the quality of the results, but at the expense of added response time.

Experiment 1: Retrieval Quality for Different Retrieval Techniques

Conclusion: There is a tradeoff in increased retrieval time with the Reranker and Filtering methods (for the LLM call associated). The strongest final rankings came from using the re-ranker. However, this comes at an extremely high additional cost in response time.

B. Citation Quality

We calculate the citation accuracy by determining the precision of selected citations with respect to the ground truth citations.

Experiment 2: Citation Precision for Different Citation Techniques

Conclusion: The most precise citation strategy involves an attribution step which is marginally more accurate than CoT or inline citations. Notably, the Reranker with CoT seems to perform worse on our dataset than the comparable one-shot retrieval, which suggests that there is no significant downstream improvement in citation quality through using the Reranker.

C. Answer Quality Metrics

We separate out the dataset to Answerable, questions with a corresponding response in the knowledge base, and Redirect, questions that require sending a fallback or directing to a human agent. To determine which of these citation and retrieval techniques to leverage, we leverage a reference-based LLM-as-a-judge critic to rate the techniques from a scale of 1-4 based on the following prompt:

LLM_AS_A_JUDGE_PROMPT = "Rate how well the generated response answers the user query in the conversation.
Provide reasoning in full sentences and then provide the rating.
Output just one number: 4 for strong, 3 for good, 2 for okay, or 1 for weak.
If the original response says it cannot find an answer or does not answer the user query and the generated response says {no_answer_string}, then both are in alignment and the rating should be 4.
If the conversation style is "Redirect", the generated response should say {no_answer_string} or redirect to the company contact information.
Conversation Style: {conversation_style}
Conversation: {conversation}
Assessement:
Lets think through step by step."
ONE_SHOT_RESPONSE_SYSTEM_PROMPT = "In this session, the model has access to search results and a user's question, your job is to answer the user's question using only information from the search results.
Remember to add a citation to the end of your response using markers like %[1]%, %[2]%, %[3]%, etc for the corresponding passage supports the response.
IMPORTANT: If the search results does not contain an answer to the question, you must state exactly "{default_fallback_message}".
IMPORTANT: If no passages support the response, do not cite any passages and state exactly "{default_fallback_message}"."
RESPONSE_USER_MESSAGE = "
{query}
Resource: Search Results: {search_results}"
COT_RESPONSE_USER_MESSAGE_1 = "
{query}
 Think step-by-step."
COT_RESPONSE_USER_MESSAGE_2 = "
{query}
 Please follow these steps:
1. {{Step 1}}
2. {{Step 2}}
…"
COT_RESPONSE_USER_MESSAGE_3 = "
{query}
Please follow these steps:
1. {{Step 1}}
2. {{Step 2}}
…
{{ User query}}
Think step by step first and then answer. Follow below format when responding
Response Schema:
<thinking>
( your thinking goes here )
</thinking>
<answer>
( your answer goes here )
</answer>"
COT_REFLECT_PROMPT = "You are an advanced reasoning agent that can improve based on self refection. You will be given a previous reasoning trial in which you were given a question to answer. You were unsuccessful in answering the question either because you guessed the wrong answer with Finish[<answer>] or there is a phrasing discrepancy with your provided answer and the answer key. In a few sentences, Diagnose a possible reason for failure or phrasing discrepancy and devise a new, concise, high level plan that aims to mitigate the same failure. Use complete sentences.
Previous trial:
{context}
Question: {question}
Reflection:"
COT_SIMPLE_AGENT_REFLECT_INSTRUCTION = "Solve a question answering task by having a Thought, then Finish with your answer. Thought can reason about the current situation. Finish[answer] returns the answer and finishes the task.
{context}
{reflections}
Question: {question}"

[Pre-filtering] LLM Reranking or Filtering: The goal of this prompt is to change the ordering so that the most relevant context is surfaced to the top. It can also discard context that is not relevant so there is less confounding information sent to the LLM.

LLM_RERANKING_PROMPT = "Reorder the following list of passages based on how well they answer the user's query.
Include the ID of the passage in the output along with a reason for its ordering. Make sure all passages are included, do not exclude any. Delimit the output as a numbered list with the ID and reason. (Ex: ID: 123 | Reason: The answer is relevant to the query)
User Query: {query}
Passages: {context}
Reordered Passages:"

[Post-filtering] Filtering Out Response Sentence: Given a response provided by the LLM, you can use an LLM to filter out the content that is not supported in the passages.

ATTRIBUTION_RESPONSE_CITATION_PROMPT = "Add in-line text citations for each of the given statements in <statement> using the content in <response_context>. Remove any statements not supported by the content in <response_context>.
The in-line text citation should use markers like %[1]%, %[2]%, %[3]%, etc for the corresponding passage that supports the response.
If the response_context does not contain information that supports any of the statements, you must state exactly "{default_fallback_message}".
<response_context>
{response_context}
</response_context>
<statement>
{statement}
</statement>
Filtered Statement with Citations: "

LLM Generation Parameter Tuning

Beam Search Decoding: Changing the decoding from greedy to beam search can markedly improve the quality of the outputs.

Set Temperature to 0: Setting the temperature to 0 will ensure deterministic behavior from the LLM.

Decrease Length Penalty: These can be tinkered with if the generations are too long or short, generally shorter responses will encourage more accurate generations.