Sofia V3: Next Generation Conversational Sales AI

Motivation

Connectly AI is a B2B conversational AI company that offers sales AI SaaS for major e-commerce brands such as Alibaba, Columbia, and American Eagle.

The flagship product offered by Connectly AI is the Sofia AI Sales Assistant, an advanced conversational sales AI that enables product discovery, recommendation, and customer assistance, driving 30% higher conversions for outbound marketing campaigns and inbound contacts to businesses.

Connectly's Flagship Sofia AI Sales Assistant

Challenge

After initial success with Sofia AI, our customers wanted it to go beyond a basic salesperson to become a sales "expert". To become an expert, Sofia AI needed more advanced multi-hop reasoning and multi-step planning. For example, in order to recommend the best Labor Day outfit Sofia might need to first read several blogs, browse a product catalog, and finally recommend outfits.

Furthermore, an AI "expert" cannot be a one-size-fits-all model; it must be uniquely tailored to each business. Our business customers need to be able to share domain knowledge and specify instructions in natural language (i.e. increase sales of a certain product, foster business to business interactions, target lead generation, etc). For example, a business might say "our customers respond well when you recommend at least one option from the clearance rack for each outfit" or "for the next two weeks, you can use promotional code 'sofia15' to entice hesitant customers to buy."

Previous Architecture: Sofia V2

The previous Sofia AI assistant was designed with two stages: tool use and response. The architecture was designed as a multiplexer of specialized tools, where each "action model" would have logic that determines whether to trigger that tool. The architecture was optimized for low latency and costs by using smaller task-specific models to decide whether or not a tool should be invoked based on the conversations. There are several pros and cons associated with this design.

Sofia V2 Architecture

At the final stage, an LLM would be tasked with analyzing a large conversation, sets of tool call results, custom prompt logic, and complex set of conditions or rules on how to respond and attempt to curate a reply.

There were several key challenges with this design:

1. It can't perform multi-hop reasoning because multiple tools can only be invoked in parallel, not in sequence. For example, it could read style guides and look for product recommendations simultaneously, but it could not read a style guide and then look for a recommendation.
2. There are unrecoverable failure points. If the intent "action" model for product recommendations didn't detect a recommendation intent, the bot would be unable to serve recommendations.
3. When multiple tools ran in parallel with the response model, LLM would get overloaded with too much context. For example, it would be asked to summarize extracts from a knowledge base and follow up on recommendations.

Next Generation V3 Architecture

In Sofia V3 we aim to design an AI sales assistant that is a highlyversatile conversational chatbot. Our intelligent sales AI can accomplish merchant-specified objectives and tackle complex multi-hop queries using state of the art agentic reasoning techniques.

Description of Flow

One of the important goals in the architecture design was to ensure each AI module had singular, well-defined responsibility and was not over-encumbered with multiple tasks. To enable this, we provided for a set of custom merchant-specified rules that would be assigned to different modules (we will discuss this in-depth later).

When a user asks Sofia for a suggestion of gift ideas or assistance with their order, this is first passed to the instruction extraction agent which generates a contextual goal for Sofia to accomplish. Then, this instruction is passed to the planning & tool calling agent which would curate a low-level plan and execute tools to accomplish the task. Finally, the tool results would be yielded along with the instruction to the response agent which crafts a message back to the user.

Sofia V3 Architecture

Example Conversation with Sofia

Below is an example 2-turn interaction between a user and Sofia AI for a wine seller within an intuitive conversational experience. The high-level objective for Sofia is to explain and sell the main product, being a wine subscription service.

User: Wines to pair with Margherita pizza.

Sofia: To pair with margherita pizza, I recommend the following wines: Chianti Classico, Pinot Grigio, and Sauvignon Blanc. Do any of these catch your eye?

User: I'll go with the Pinot Grigio, what are the subscription options?

Sofia: Of course! Here is a Pinot Grigio from the Burgundy region of France <link>. I recommend one of our 3 subscription plans to get the best value. Would you like to learn more?

Instruction Extraction Agent

Instruction Extraction Agent Flow

The first module would be an instruction extraction agent, which would be tasked with identifying the objective of the system based on the "Behavior Rules" and the user's query.

The Behavior Rules describe the high-level behavior of the system, such as how it should respond to different types of input. Below is the example for our wine store:

• Provide information about our store's history, wine portfolio, and stores
• Explain the different subscription plans and their associated benefits, including free shipping, discounts, and corkage waivers.

Given the context of the conversation and goals the business outlined for Sofia, it designs an appropriate instruction for the given stage of the conversation flow.

Turn 1 Instruction: Review the wine pairing guide and suggest several wines from our portfolio that would pair well with a Margherita pizza.

Turn 2 Instruction: Search the product catalog for Pinot Grigio and suggest purchasing a subscription plan.

The instruction critic observes the crafted task and determines whether it fully captures and adheres to the given behavior rules. This is a technique described in Reflexion (Shinn et. al 2023) that improves the reasoning ability of the LLM and ensures that the instruction extractor is more robust at capturing the latent intent and the business's goals.

After leveraging a critic to verify the accuracy of the extracted instruction, it will be passed to the tool-calling agent and response agent to guide their next steps.

Planning & Tool Calling Agent

Planning & Tool Calling Agent Flow

The Plan & Tool Calling agent is tasked with leveraging a suite of tools provided to the sales assistant to attempt to fulfill the instruction. This involves crafting a plan, executing tools, and observing their outputs to determine their acceptability.

In our example, Sofia was instructed to share suggestions for users by tapping to a knowledge base of catalogs and guides about wines most suitable for pizza along with a product inventory of available wines.

Turn 1 Planning & Tool Calling Trace

Thought: I need to suggest a suitable wine that will pair well with Margherita pizza. I should search the knowledge base to find a product guide that describes wine selections.

Tool Call: knowledge_base_search(instruction="Find wines from our portfolio that pair well with a Margherita pizza")

Tool Output: Recommended wines: Chianti Classico, Pinot Grigio, and Sauvignon Blanc.

Turn 2 Planning & Tool Calling Trace

Thought: I should find a Pinot Grigio wine within our product inventory for the user.I should then inform them about the subscription service to purchase that specific Pinot Grigio product.

Tool Call: recommendation_engine(query="Pinot Grigio")

Tool Output: Here is a link to our selection of Pinot Grigio: <link>.

Response Agent

Response Agent Flow

The final module is the Response agent, which would have access to the response rules, and would be tasked with understanding the conversation and the observed tool outputs to write a response back to the user.

Response Rules describe the formatting and style of the system's response. Below is an example of how the business can specify tone or level of detail in the response for the wine seller.

• Format responses in a clear and concise manner. Use bullets or numbered lists as needed.
• Use a friendly and professional tone.
• Engage the user by asking relevant follow-up questions

Turn 1 Response: To pair with margherita pizza, I recommend the following wines: Chianti Classico, Pinot Grigio, and Sauvignon Blanc. Do any of these catch your eye?

Turn 2 Response: Of course! Here is a Pinot Grigio from the Burgundy region of France <link>. I recommend one of our 3 subscription plans to get the best value. Would you like to learn more?

Note how it structures the response and adheres to the goal of being concise and direct to the point while engaging the user with relevant follow-ups.

System Tuning: Models and Prompts

To ensure we use the optimal models and prompts, we construct datasets and use metrics-driven development to improve iteratively.

Dataset Generation

While the methods and structure of our dataset remain proprietary. To encourage future work on these systems, we want to highlight one synthetic dataset-generation technique that we found particularly useful:

Following the technique in SMM-QG, we instruct the generative model to design both successful & unsuccessful tool outputs to test both cases.

SMMQG's Sampling & Dataset Generation Methodology (Wu et. al 2024)

This was useful as it allowed us to have a standard dataset to develop with and to optimize for in our metrics-driven development methodology.

Evaluation: Metrics-Driven Development

Note:Our LLM architecture is proprietary, thus for this section we'll use common foundational models to showcase our metrics driven development approach*.*** Additionally, we hope this can add more information to our common knowledge of how popular foundational models perform.

In order to assess the success, we define clear measurable scores that want our system to optimize for in a) instruction extraction & b) success at rule following.

We evaluated the success of the instruction extraction & critic by leveraging GPT-4 as a judge to rate the quality of the generated instructions on a scale of 4 over 3 conversations with 5 turns on average.

Table of Instruction Ratings for Popular Foundation Models

We wanted to benchmark the quality of the conversations, particularly in adhering to the defined categories of rules, with the previous architecture. We benchmark on cases with successful tool outputs (with Successful Tool Outputs) and on negative cases (with Unsuccessful Tool Outputs) where tools fail & determine if they still succeed in adhering to the given rules.

Table of Rule-Following Ratings comparing Sofia V2 and Sofia V3 with GPT-4o

These results show that there are clear monotonic improvements in the new architecture design in adhering to the custom assistant rules during the conversation. Additionally, in cases when the tools fail, we still are able to continue to engage the user in the conversation without degrading performance in rule-following.

Ablation Across LLMs

One interesting study we performed was to ablate across different LLM models within the stack. Different models have tradeoffs in their ability to support multilingual, tool-calling, or complex reasoning and we sought to find the best mix of models while minimizing response time and cost.

Ablation Across Models used within Sofia V3

We find that we can leverage a mix of these models, each with appropriate capability at the task at hand, to maximize performance while also minimizing response time.

Average Conversation Time Breakdown

We plan to study further optimization in the response time of the model to make it feel instantaneous for end users.

Conclusion

The Sofia V3 provides advanced capabilities to allow merchants to program an autonomous AI sales "expert" in pure natural language. Businesses can both design custom goals for the sales assistant and criteria for how it should interact with users and it intelligently accomplishes these tasks within the conversation. The next-generation version of Sofia AI is now available at https://www.connectly.ai/ and more advanced features are being rolled out monthly!

Literature Review

Building contemporary LLM applications is a challenging task and we've identified the best known techniques for designing and building a performant, production-ready Generative AI system at lighting speed.

Synthetic Dataset Generation, Wu, Jayanthi et. al 2024:

• We leverage the SoTA technique for building high-quality human-like synthetic datasets to rapidly design a benchmark to test and iterate our design upon. This work, Synthetic Multimodal Question Generation, details a rigorous approach for building user-specified query, response datasets for closed-domain tasks like question-answering or search.

Reflexion, Shinn et. al 2023:

• Reflexion describes an approach to improve the quality of LLMs at difficult tasks requiring reasoning by leveraging a critic model that evaluates whether the task was accomplished successfully. The critic produces a "reflection" that is passed back to the original model to re-attempt the task with a critique of its previous approach in an effort to get it to design a more robust approach to solving the task.

Verbal Reinforcement Learning in Reflexion (Shinn et. al 2023)

Self-Reflection, Renze and Guven 2024 & Self-Refine, Madaan et. Al 2023::

• Many advanced RAG or Agentic LLM pipelines leverage self-reflection to increase the reliability of the underlying language model tasks. Similar to Reflexion, this taps into a key ability of contemporary foundation models at being more performant at discriminative tasks than generative ones. Therefore, in crafting function calls or solving contextual Q&A, leveraging implicit feedback from the LM can enable it to validate the correctness of its response.

LangChain example of Self-Reflection

GPT-as-a-Judge, Thankur et. al 2024:

• In a study of all major closed and open-source LMs, the model with the strongest alignment with human-annotated judgements was found to be GPT-4. Therefore, we leverage GPT-4 as a close proxy for human judgements when scoring the quality of generations for various tasks studied.

Authors

Sravan Jayanthi, Joel Simonoff, Yandong Liu