Building a Production RAG Pipeline with Vector Databases, Hybrid Search & Reranking: Step-by-Step

1. The Production Failure of Naive RAG

Retrieval-Augmented Generation (RAG) is the dominant architecture for grounding LLMs on private enterprise knowledge. However, naive RAG pipelines—taking raw PDF documents, splitting them into fixed 500-character chunks, embedding them with standard models, and performing basic cosine similarity lookups—consistently fail when deployed in enterprise production.

The failure modes are well-documented: context fragmentation (splitting critical tabular data across arbitrary boundaries), keyword blindness (dense embeddings failing on exact part numbers or error codes), and lost-in-the-middle attention degradation (passing 20 irrelevant chunks to an LLM context window).

Engineering a production AI chatbot solution requires a multi-stage retrieval architecture: Structure-Aware Document Ingestion, Hybrid Sparse-Dense Indexing, Reciprocal Rank Fusion (RRF), and Cross-Encoder Reranking.

2. Stage 1: Document Ingestion & Structure-Aware Chunking

Arbitrary character splitting destroys document semantics. Production ingestion must preserve document topology:

Semantic Chunking Rules

• Markdown & Table Preservation: Tables must be extracted and preserved as complete standalone JSON/Markdown objects with injected header context.
• Parent-Child Document Indexing: Ingest small child chunks (128 tokens) for hyper-accurate vector search, but retrieve the larger parent chunk (512–1024 tokens) to pass complete context to the LLM.
• Metadata Context Injection: Every chunk is prepended with hierarchical metadata (e.g., Document: 2026_Enterprise_Security_Policy.pdf | Section: Section 4.2 API Authentication) before embedding.

3. Stage 2: Hybrid Sparse + Dense Retrieval (BM25 + Vector)

Dense embeddings (such as OpenAI text-embedding-3-large) excel at capturing high-level semantic meaning (e.g., understanding that "remuneration" relates to "salary"). However, they frequently fail on precise lexical keywords (e.g., product SKU SKU-8849-X or error code ERR_AUTH_TIMEOUT_504).

Production systems combine BM25 sparse lexical search with dense vector embeddings using convex score combination:

import numpy as np
from pinecone import Pinecone
from pinecone_text.hybrid import hybrid_convex_scale

pc = Pinecone(api_key="your_pinecone_api_key")
index = pc.Index("production-enterprise-rag")

def hybrid_retrieve(query_text: str, top_k: int = 25, alpha: float = 0.75):
    """
    alpha = 1.0 -> Pure Dense Vector Search
    alpha = 0.0 -> Pure Sparse BM25 Keyword Search
    alpha = 0.75 -> Optimal production balance
    """
    # 1. Generate 3072-dim dense embedding
    dense_vector = embedding_client.embeddings.create(
        input=query_text,
        model="text-embedding-3-large"
    ).data[0].embedding

    # 2. Generate BM25 sparse token weights
    sparse_vector = bm25_encoder.encode_queries(query_text)

    # 3. Apply convex scaling
    scaled_dense, scaled_sparse = hybrid_convex_scale(dense_vector, sparse_vector, alpha=alpha)

    # 4. Query Pinecone index
    response = index.query(
        top_k=top_k,
        vector=scaled_dense,
        sparse_vector=scaled_sparse,
        include_metadata=True
    )
    return response["matches"]

4. Stage 3: Cross-Encoder Reranking (Cohere Rerank v3)

Vector databases return the top 25 candidate chunks based on bi-encoder dot products. However, passing 25 chunks to an LLM context window causes cognitive saturation and 10x token inflation. We pass candidates through a Cross-Encoder Reranker:

• Unlike bi-encoders which encode query and document separately, cross-encoders compute simultaneous multi-layer self-attention across the query and document together.
• The reranker re-scores and re-orders the candidate list, selecting only the top 3 to 5 highest-relevance chunks.
• Hallucination rates drop by over 91% compared to un-reranked vector search. Learn more in our Why Off-the-Shelf AI Chatbots Fail Guide.

import cohere

co = cohere.ClientV2(api_key="your_cohere_api_key")

def rerank_context_chunks(query: str, retrieved_chunks: list[dict], top_n: int = 4) -> list[dict]:
    documents = [chunk["metadata"]["text"] for chunk in retrieved_chunks]
    
    rerank_response = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=documents,
        top_n=top_n
    )
    
    selected_chunks = []
    for hit in rerank_response.results:
        selected_chunks.append({
            "text": documents[hit.index],
            "relevance_score": hit.relevance_score,
            "chunk_id": retrieved_chunks[hit.index]["id"]
        })
    return selected_chunks

5. Vector Database Comparison for Enterprise Scale

Vector Database	Best For	Hybrid Search Support	Hosting Model
Pinecone Serverless	Zero-maintenance cloud scale, instant indexing	Native (Dense + BM25)	Fully Managed Cloud
Qdrant	High-throughput payload filtering, complex geo/data filters	Native (Sparse/Dense)	Managed Cloud / Self-Hosted Docker
Supabase (pgvector)	Relational apps where embeddings live with user SQL tables	Via pg_trgm + pgvector	Managed Cloud / Self-Hosted PostgreSQL
Weaviate	Multimodal image/audio/text search pipelines	Native (BM25 + Dense)	Managed Cloud / Kubernetes

6. Automated Evals: Measuring RAG Quality with Ragas

Production RAG systems must be continuously evaluated against three core metrics:

1. Context Relevance: Measures the signal-to-noise ratio of retrieved chunks (target: >0.90).
2. Faithfulness (Groundedness): Verifies that all generated statements are derived exclusively from retrieved context (target: >0.96).
3. Answer Relevance: Verifies that the response directly addresses the user's initial query (target: >0.94).

7. Build Your Production RAG Pipeline with IKONIC LABS

Eliminate hallucinations and unlock the true intelligence within your corporate data. Explore complete multi-agent integration in our Scaling Multi-Agent AI Systems Guide and calculate development investment in our Custom AI Agent Pricing Guide. Book a technical scoping session with IKONIC LABS to architect an enterprise-grade RAG pipeline in under 48 hours.