Why Most RAG Demos Fail in Production

You've seen the 50-line RAG demo. Load a PDF, embed it, ask a question, get an answer. It works in a notebook. It falls apart the moment you add real users, large documents, concurrent requests, and an expectation that the system stays up.

The gap between a RAG prototype and a production RAG system is where the real engineering lives.

This post documents the full architecture of a production-grade RAG Knowledge Assistant — a platform where users upload PDFs and books, interact via conversational AI with source citations, and get AI-generated flashcards and quizzes. The decisions here are real ones I made while building it, not theoretical recommendations.

Let's go deep.

The 30,000-Foot View

Before diving into components, here's the system at a glance:

Every architectural decision flows from one constraint: the user experience must feel fast, even when the underlying pipeline is not.

Part 1: The Ingestion Pipeline

Why Async Ingestion is Non-Negotiable

The naïve approach is synchronous: receive file → process → respond. For a 200-page PDF, that means holding an HTTP connection open while you OCR, chunk, and embed the entire document. That's a 30–120 second request. That's unacceptable.

The solution is a task queue pattern using Celery + Redis:

`@shared_task(
    bind=True,
    autoretry_for=(Exception,),
    retry_backoff=True,
    retry_kwargs={"max_retries": 3}
)
def ingest_document(self, document_id: int):
    with transaction.atomic():
        doc = Document.objects.select_for_update().get(id=document_id)
        doc.status = "PROCESSING"
        doc.save()

    try:
        text = extract_text(doc.file)
        chunks = semantic_chunk(text, chunk_size=512)
        embeddings = embed_batch(chunks)

        qdrant_client.upsert(
            collection_name="documents",
            points=[...]
        )

        doc.status = "INDEXED"
        doc.save()

        notify_document_ready(document_id, doc.user_id)

    except Exception as e:
        doc.status = "FAILED"
        doc.error_message = str(e)
        doc.save()
        raise `

The API layer queues the task and immediately returns a 202 Accepted with a document ID. The user sees a loading state. When ingestion completes, a WebSocket event pushes the status update to the frontend. Zero polling.

The select_for_update() is important — it prevents concurrent tasks from double-processing the same document if a worker crashes and the task is retried.

Semantic Chunking vs. Fixed-Size Chunking

Most tutorials use fixed-size chunking (every N tokens, with overlap). It's fast but ignores document semantics. A 512-token chunk might split a paragraph across two chunks, separating a claim from its evidence. Semantic chunking respects sentence and paragraph boundaries. The implementation uses recursive splitting with split_on_sentences=True, targeting 256–512 token windows with 128-token overlap:

chunk_size=512, overlap=128, split_on_sentences=True

The overlap ensures that context at chunk boundaries isn't lost during retrieval.

The payoff: fewer hallucinations, because retrieved chunks contain coherent reasoning units rather than mid-sentence fragments.

Embedding Infrastructure: Why TEI Over API Calls

For embedding generation, the system uses HuggingFace Text Embeddings Inference (TEI) running locally, not an external API. The model is sentence-transformers/bge-large-en-v1.5.

Why not just call OpenAI's embedding API?

1. Latency: local inference adds ~5ms vs. ~80–150ms for an API round-trip per batch

2. Cost: embedding 100k chunks per day at OpenAI rates adds up; local inference is flat-cost

3. Consistency: the same model version always, no silent upstream changes

4. Data privacy: document content never leaves your infrastructure

The TEI container is registered in docker-compose alongside Qdrant, Postgres, and Redis:

Redis:

tei:
  image: ghcr.io/huggingface/text-embeddings-inference:0.4.0
  ports:
    - "8080:80"
  environment:
    MODEL_ID: sentence-transformers/bge-large-en-v1.5

Part 2: The Retrieval Pipeline

This is where most RAG systems get it wrong. A single similarity_search(query, k=5) call is not a retrieval pipeline. It's a starting point.

Stage 1: Query Contextualization

Conversational RAG has a silent failure mode: the user asks "What did he say about that?" — and the retrieval system has no idea what "he" or "that" refers to.

The fix is query contextualization: rewrite the incoming query using the last N messages of chat history before retrieval.

def contextualize_query(query, chat_history):
    history_context = "\n".join([
        f"{msg.sender}: {msg.text}"
        for msg in chat_history[-5:]
    ])

    rewritten = llm.generate(
        f"Rewrite this query as a standalone question                     given context:\n"
        f"{history_context}\n"
        f"Query: {query}"
    )
    return rewritten

This adds ~200ms latency (an LLM call), but it's the difference between a retrieval system that works in multi-turn conversations and one that doesn't.

Stage 2: Intent Routing

Not every message needs the full RAG pipeline. "Hello!" doesn't need vector search. "Summarize this document" doesn't need retrieval at all.

An Intent Router — a lightweight LLM classifier — categorizes each query before dispatching it:

This optimization alone saves 60–70% of LLM generation calls for a typical user session.

Stage 3: Query Expansion

Even after contextualization, a single query embedding is a single point in vector space. If the user phrases their question differently from how the document phrases the answer, cosine similarity will miss it.

Query expansion generates 3–4 semantic variations of the contextualized query, runs parallel vector searches for each, and merges the result sets:

def expand_query(query):
    variations = llm.generate_variations(query, count=3)
    return [query] + variations

def retrieve_chunks(queries):
    all_results = []
    for q in queries:
        q_embedding = tei_client.embed(q)
        results = qdrant_client.search(
            collection_name="documents",
            query_vector=q_embedding,
            limit=8,
            with_payload=True
        )
        all_results.extend(results)

    # Deduplicate
    unique_chunks = {r.payload["id"]: r for r in all_results}
    return list(unique_chunks.values())

Parallel execution using asynciokeeps the latency impact to ~180ms, not 4×180ms.

Stage 4: Cross-Encoder Reranking

Vector similarity gives you recall. Cross-encoder reranking gives you precision.

Bi-encoder models (used in vector search) embed query and document independently and compare them with cosine similarity. It's fast, but misses subtle semantic relationships. Cross-encoders jointly encode the query-document pair, which is slower but dramatically more accurate.

The architecture separates these two stages deliberately:

• Vector search (bi-encoder): cast a wide net — retrieve top-8 candidates per query variation (~30–40 chunks total after dedup)

• Cross-encoder reranking: score each candidate jointly with the query, keep only chunks scoring ≥ 4/10

def rerank_chunks(query, chunks):
    scores = cross_encoder.predict([
        [query, chunk.payload["text"]]
        for chunk in chunks
    ])

    ranked = sorted(
        zip(chunks, scores),
        key=lambda x: x[1],
        reverse=True
    )

    return [chunk for chunk, score in ranked if score >= 4]

The score threshold is intentional: it prevents low-relevance context from being injected into the LLM prompt, which is a leading cause of hallucinations in RAG systems.

Part 3: Generation and Response Delivery

Context Assembly and Citation

Once high-relevance chunks pass the reranker, they're assembled into a structured prompt with page-level citations:

python

context = "\n---\n".join([
    f"[{c.payload['page']}] {c.payload['text']}"
    for c in context_chunks
])

prompt = f"""Based on this context:
{context}

Answer the question: {query}

Include source citations like [page X]."""

Citations are first-class, not an afterthought. Every generated response is grounded in specific pages of the source document, which the frontend renders as clickable references.

Streaming via WebSockets

The LLM generation (Llama 3 via Groq) runs in streaming mode. Tokens are pushed to the frontend via Django Channels WebSocket as they arrive — first token to screen in ~300ms, regardless of response length.

class DocumentConsumer(AsyncWebsocketConsumer):
    async def connect(self):
        self.user_id = self.scope["user"].id
        self.group_name = f"user_{self.user_id}"
        await self.channel_layer.group_add(
            self.group_name,
            self.channel_name
        )
        await self.accept()

    async def document_status_update(self, event):
        await self.send(text_data=json.dumps({
            "type": "document_indexed",
            "document_id": event["document_id"],
            "status": event["status"],
            "chunks": event["chunks"]
        }))

WebSockets serve double duty: ingestion status updates and streaming LLM responses. One persistent connection per user session, multiplexed via Django Channels group messaging.

Part 4: From Retrieval to Active Learning

Retrieval solves one problem — finding information. It doesn't solve a second, equally important one: helping users actually retain it.

While testing the platform, a pattern kept appearing. Users would ask questions, get accurate answers, and leave. The system was working, but it wasn't doing anything to convert that retrieved knowledge into something durable.

So I built a content generation layer directly on top of the RAG pipeline. Instead of treating indexed documents as static knowledge sources to query against, the system can transform them into structured learning material on demand.

Question Generation

The workflow is deliberately simple — simplicity here is a feature, not a shortcut:

Document
    ↓
Chunk Retrieval (reranked)
    ↓
Context Assembly
    ↓
LLM Question Generation
    ↓
Structured Question Set

The system retrieves and reranks chunks from the indexed document, assembles a focused context window, and prompts the LLM to generate practice questions grounded in that specific content — not the model's general knowledge. Users can test their understanding of their own material without manually authoring a single question.

Flashcard Generation

For active recall, a separate pipeline converts document knowledge into Q&A pairs:

Document
    ↓
Retrieve Key Concepts
    ↓
Extract Definitions and Relationships
    ↓
Generate Q/A Pairs
    ↓
Interactive Flashcard Set

The distinction from a generic "summarize this" prompt matters: the system specifically targets concepts, definitions, and relationships within the retrieved chunks — the building blocks of durable recall — rather than just condensing prose.

Engineering Considerations

Three problems surfaced during implementation worth naming explicitly.

Context selection quality directly determines output quality. A bad retrieval result produces a bad flashcard. The fix was to run generation from reranked chunks, not raw vector search results. The cross-encoder's relevance scoring acts as a quality gate before any generative call is made.

Hallucination control is structural, not prompt-based. Questions and flashcards are generated only from retrieved document context, with the prompt explicitly constrained to that window. The model cannot pull from its parametric knowledge. This keeps generated study material aligned with the user's source document, not with what the model happens to know about the topic.

Generation results are persisted and served asynchronously. Running a full rerank + generation pass on every visit would be expensive and slow. Instead, generated question sets and flashcard decks are stored in PostgreSQL after first generation and retrieved on subsequent views — the expensive LLM call happens once, not repeatedly.

Why Build This at All?

Most document-chat systems stop at question answering. The goal here was to support the full learning loop:

Upload → Understand → Revise → Test

Retrieval handles "Understand." Flashcards handle "Revise." Generated questions handle "Test." Without all three, the platform is just a smarter Ctrl+F.

Part 5: Resilience Patterns

Circuit Breaker for External AI Services

Embedding and LLM inference services can fail transiently. Without a circuit breaker, a degraded TEI or Groq endpoint causes cascading failures — every incoming request queues up waiting for a timeout.

A simple circuit breaker implementation:

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_timeout=60):
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.state = "CLOSED"

    def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.recovery_timeout:
                self.state = "HALF_OPEN"
            else:
                raise CircuitBreakerOpen("Service temporarily unavailable")

        try:
            result = func(*args, **kwargs)
            if self.state == "HALF_OPEN":
                self.state = "CLOSED"
                self.failure_count = 0
            return result
        except Exception:
            self.failure_count += 1
            if self.failure_count >= self.failure_threshold:
                self.state = "OPEN"
                self.last_failure_time = time.time()
            raise

Three states:

• CLOSED: normal operation, failures counted

• OPEN: service assumed down, calls rejected immediately with a meaningful error

• HALF_OPEN: single probe allowed to check recovery

Celery Retry Strategy

`@shared_task(
    autoretry_for=(TimeoutError, ConnectionError),
    retry_backoff=True,
    retry_backoff_max=600,
    retry_jitter=True,
    max_retries=5
)
def process_chunk(chunk_id, embedding):
    pass

retry_jitter=True is the key detail here. Without jitter, multiple failed tasks all retry at the same backoff interval, creating thundering herd behavior against a recovering service. Jitter spreads retries across a time window.

Concurrency Control

Without rate limiting, a burst of concurrent LLM requests will either exhaust the Groq rate limit or DOS your local inference server.

MAX_CONCURRENT_LLM_CALLS = 20
llm_semaphore = asyncio.Semaphore(MAX_CONCURRENT_LLM_CALLS)

async def generate_with_rate_limit(query, context):
    async with llm_semaphore:
        return await groq_client.generate_async(query, context)

Performance Profile

End-to-end query latency on a warm system:

LLM generation dominates. This is why streaming matters — the user sees words appearing at ~300ms, not a 1.8s blank screen.

The 45ms vector search (across parallel queries) is a direct result of Qdrant's HNSW index. A naive PostgreSQL full-scan equivalent would be ~800ms at comparable dataset sizes.

What I'd Do Differently

1. Semantic caching

For queries semantically similar to previous ones, return the cached answer without running the full pipeline. A Redis layer with pgvector-based similarity lookup on query embeddings would cut latency by 60–80% for repeat question patterns.

2. Chunking strategy needs per-document calibration

A fixed 512-token semantic chunk works well for prose. For academic papers with structured sections, a section-aware chunker would improve retrieval quality. For code-heavy documents, a syntax-aware chunker matters. The current implementation doesn't distinguish document types.

3. Qdrant collection design

Currently a single documents collection with document_id as a payload filter for multi-tenant isolation. At scale, per-user collections would provide better isolation and allow collection-level configuration tuning (HNSW m and ef_construct parameters) per workload pattern.

Stack Summary

Closing Thoughts

The delta between a RAG proof-of-concept and a production RAG system isn't model quality — it's infrastructure. It's what happens when the embedding service goes down, when a user uploads a 500-page book, when 50 users query simultaneously.

The patterns in this architecture aren't novel. Async task queues, circuit breakers, retry with jitter, semantic caching — these are standard distributed systems patterns. What's interesting is how they compose specifically for the RAG problem: a workload with unusually high variance in task duration, external AI service dependencies, and real-time user-facing latency requirements.

Build for failure. Instrument before you need it. Stream everything you can.

If you want to explore the codebase, it's on GitHub. Questions, critiques, and architecture debates welcome in the comments.

Tags: rag, system-design, llm, django, celery, vector-databases, ai-engineering, python, backend