RAG-Based Enterprise AI Chatbot

Overview

The enterprise insurance platform I worked on at LTM served over one million users across policy management, premium calculation, claims, and billing. At that scale, support fielded a high volume of daily inquiries, and most were the same questions (policy status, premium amounts, claim progress, coverage details): repetitive, structured, answerable from data, yet every one landed in an agent's queue.

I built the AI chatbot that changed that: a Spring Boot orchestration layer with WebSocket-backed real-time conversations, a transformer-based NLU model for intent and entity extraction, a LangChain/OpenAI pipeline for complex policy inquiries, Redis-backed session state, and a confidence-threshold escalation path that handed agents full conversation context instead of dropping users mid-interaction, shadow-deployed against live traffic for two weeks before cutover. The result: a 60% reduction in support ticket volume, freeing agents for the cases where they add value.

---

The Problem

Most inbound support volume ("what is my current premium?", "what's the status of my claim?", "what does my policy cover for X?") had deterministic answers sitting in the database, specific to the user's policy, requiring no human judgment. Yet every inquiry was submitted as a ticket, queued, and answered by a human reading from the same database the system already had access to. Agents spent most of their time on mechanical lookup-and-respond work instead of the cases that needed their expertise.

The business case was obvious. The engineering challenge was building a chatbot safe for a production insurance platform, where a hallucinated policy detail or a confidently wrong claim status carries real consequences.

---

Goals

• Autonomously handle the high-volume repetitive inquiry categories (policy status, premium, claim status, coverage, billing).
• Classify intent and extract entities with a transformer-based NLU layer before routing.
• Use LangChain/OpenAI for complex inquiries needing reasoning across documents, but avoid generative responses for factual inquiries where accuracy is non-negotiable.
• Back conversation state with Redis so sessions survive drops and carry full context into agent handoffs.
• Escalate below a defined confidence level, with the complete history. No cold transfers.
• Shadow-deploy against real traffic to validate accuracy before exposing users.

---

Technical Architecture

System Overview

The chatbot sits between the frontend and the existing backend services, wrapping rather than replacing them: all factual data (policy details, claim status, premium records, billing history) is retrieved from existing services via internal API calls. It classifies the question, retrieves data, composes a response, and decides whether to answer or escalate, across four layers: WebSocket conversation, NLU classification, the response pipeline (templated retrieval and RAG), and session/escalation.

WebSocket Conversation Layer

The conversation interface runs over WebSocket connections via Spring Boot's WebSocket support (@EnableWebSocketMessageBroker). Persistent connections beat HTTP request-response for latency and state: they remove per-message round-trip overhead (which matters when NLU already adds latency), and the connection is a natural session boundary whose disconnection event triggers a finalization that writes the conversation summary to the audit log.

The handler deserializes incoming messages into a ConversationMessage, looks up or initializes the connection's Redis session, and dispatches to NLU. The response path is asynchronous: NLU and response pipelines run on a separate thread pool, so long-running inference never blocks the handler.

NLU Classification Layer: Transformer-Based Intent and Entity Extraction

Every message passes through the NLU layer before any response logic runs: intent classification and entity extraction.

Intent classification uses a fine-tuned transformer-embedding model trained on historical support tickets (real inquiries labeled to intent categories). The taxonomy:

Training on historical tickets rather than synthetic examples gave strong performance on real phrasing: users asking about claim status rarely say "claim status," they say "my claim," "the accident claim I submitted," "case number 1234." The classifier outputs a predicted intent (which routes to a pipeline) and a confidence score (which feeds escalation).

Entity extraction runs in parallel, identifying policy number, claim ID, coverage type, date range, dependent name, and payment method. It combines regex for structured identifiers (policy numbers and claim IDs follow known formats) with NER over the embeddings for semantic entities (coverage types, dates), storing them in the Redis session to parameterize downstream retrieval calls.

Response Pipeline: Templated Retrieval vs. RAG

The pipeline is split by intent type: templated retrieval for factual inquiries, and LangChain/OpenAI RAG for complex, document-grounded ones.

Track 1: Templated Retrieval (factual intents)

For intents where the answer is a deterministic lookup (policy status, premium, claim status, billing balance), the system retrieves data from internal APIs and populates a fixed template, e.g. for a premium.inquiry:

"Your current monthly premium for Policy [POLICY_NUMBER] is $[AMOUNT], effective [DATE]. Your next payment of $[AMOUNT] is due on [DATE]."

This is explicitly not generative: the template is fixed, the values come straight from the database, and the model only classifies and extracts.

This was contested. The initial product direction favored a fully generative approach for fluency; I argued for accuracy and auditability, and prevailed. "Your premium is approximately $450" when it's actually $447.23 isn't a fluency difference, it's a factually wrong statement about a contractual obligation. Templating eliminates that risk, and every field traces back to the exact API call that produced it.

Track 2: LangChain / OpenAI RAG (complex policy inquiries)

For intents that require reasoning across policy documents (coverage inquiries, policy change implications, cancellation terms), the system uses a LangChain/OpenAI RAG pipeline. It indexes the policy corpus (product guides, terms and conditions, coverage schedules, FAQs) in a vector store; policy.coverage and policy.change intents trigger a semantic search, and the chunks, question, and entities become a structured prompt for the OpenAI model.

The chain is configured with explicit grounding: answer only from retrieved content, cite the specific policy section, and flag when the content lacks a clear answer rather than inferring beyond it. Combined with the confidence threshold, this keeps RAG safe while answering coverage questions a template can't handle.

Redis Session State

Every conversation is backed by a Redis session keyed on the WebSocket connection ID, storing:

• Conversation history: the full message thread, in order.
• Extracted entities: all entities accumulated across turns.
• Classified intents: the intent for each user turn.
• Escalation state: whether the session was routed to an agent, and if so to whom and when.

Redis over in-memory state bought connection resilience and agent handoff. On reconnect, the session restores and the conversation continues. On escalation, the agent dashboard pulls the full history and entities, so the agent sees what the user said, what the chatbot understood, what data was retrieved, and why escalation triggered, before typing a word. Session TTL is 30 minutes of inactivity, extended to 4 hours once a session escalates.

Confidence-Threshold Escalation

Escalation is the system's most important safety property. Three conditions trigger a handoff:

1. Low intent confidence: the classifier's score falls below the threshold (tuned in shadow mode).
2. Entity extraction failure: a required entity is missing (a claim.status intent with no claim ID can't retrieve data).
3. Retrieval failure or empty result: a factual API call errored or returned empty, or RAG found no chunks above the similarity threshold.

In any case, the chatbot doesn't guess: it sends a transition message, marks the session escalated, and routes to the agent queue with the full session record and failure reason.

The threshold was the core internal disagreement. The initial direction was to set none and let the chatbot attempt every response, on the theory that a low-confidence answer might still help. My position: a wrong answer about an insurance policy is categorically worse than an escalation, since it damages trust, creates liability, and can lead users to wrong coverage decisions. The shadow data settled it; low-confidence responses had a substantially higher error rate.

Shadow Mode Rollout

For two weeks before any user exposure, every real inquiry was also routed to the chatbot, which logged a response that was never shown while the live workflow continued unchanged. This produced chatbot responses paired against ground-truth agent responses, each evaluated for correct policy data, correct intent classification, and the right would-escalate-vs-would-answer decision.

It caught two systematic issues before they reached users: premium phrasings consistently misrouted to policy.status instead of premium.inquiry (fixed by augmenting the training set and retraining), and a legacy policy-number format from one product line that the regex didn't match (fixed by updating the pattern and validating against the full corpus).

---

Key Technical Challenges

Challenge 1: The Generative vs. Retrieval Architecture Decision

The most consequential decision wasn't technical, it was a product and risk question that needed a technical answer: fluency (fully generative) versus accuracy and auditability. Splitting the pipeline by risk profile took more engineering than a unified approach, but was the only safe design.

The shadow data confirmed it: zero factual errors on templated responses, acceptable error rates on grounded RAG, and an unacceptable hallucination rate on the generative-only baseline.

Challenge 2: Training the NLU Classifier on Real User Phrasing

Classifiers trained on synthetic examples fail in production because real users don't talk like documentation: "What is my current monthly premium?" is the example, "How much am I paying you guys" is what a user says.

The historical tickets solved this but needed real preparation, since the corpus was noisy (mislabeled, multi-intent, incomplete sentences). Building a clean set required a manual labeling pass for ground truth, a quality filter to drop ambiguous tickets, and deliberate augmentation of underrepresented intents so the classifier wasn't biased toward the highest-volume ones. It outperformed a comparable synthetic-trained classifier on real shadow-mode traffic, which was the whole point.

Challenge 3: Escalation Context That Actually Helps Agents

The naive escalation is a warm transfer where the user re-explains everything, negating much of the triage benefit. As described above, the Redis session record makes handoffs useful, not ceremonial: the agent arrives knowing what the user wants and why the bot couldn't finish, without a single re-establishing question.

This meant designing the session schema with the agent dashboard as a consumer, not just internal chatbot state: field types and naming were specified against the UI's requirements.

---

Outcome and What It Demonstrates

From day one, the chatbot handled the majority of repetitive inquiries autonomously, cutting support ticket volume by 60% and shifting agents to the cases where their expertise mattered. What it demonstrates:

Production AI System Design. Real traffic on a high-stakes platform demands more than a working model: hybrid pipelines by risk profile, confidence-gated escalation, and shadow deployment.

Risk-Calibrated Architecture. Templated retrieval for factual responses and RAG only for document-grounded inquiries encodes that accuracy requirements vary by consequence. A wrong policy amount is not a wrong product description.

Training Data as Engineering Work. Turning a messy ticket corpus into a clean, balanced, quality-filtered training set was as much engineering as building the classifier, and directly determined the system's quality.

Escalation as a Product Feature, Not a Fallback. Designing the escalation path with the same care as the happy path is what makes the system safe to deploy.

---

Tech Stack Summary