Why RAG Is a Data Engineering Problem (And How to Build Production Retrieval Pipelines)

RAG Failures Are Data Failures

Every week, another team announces their RAG implementation is "hallucinating" or "returning irrelevant results." They blame the LLM. They try a bigger model. They add more prompt engineering. Nothing improves.

The problem is almost never the model. It is the retrieval pipeline.

RAG (Retrieval-Augmented Generation) is fundamentally a data engineering problem disguised as an AI problem. The quality of your RAG system is bounded by the quality of your ingestion, chunking, embedding, indexing, and retrieval pipeline. Get the data engineering right, and even a modest model produces excellent results. Get it wrong, and GPT-5 cannot save you.

I built a complete RAG pipeline to demonstrate these principles. The source code is at rag-knowledge-base-pipeline. This article walks through every layer of the pipeline and the engineering decisions that determine success or failure.

Architecture: A Data Pipeline That Happens to Feed an LLM

The architecture should look familiar to any data engineer. It is an ETL pipeline with a different consumer:

[Sources]        [Ingestion]       [Processing]       [Indexing]        [Retrieval]
 PDF, HTML   -->  Extract    -->   Chunk + Clean  -->  Embed + Store -->  Hybrid Search
 Markdown        Parse             Enrich              pgvector          RRF Fusion
 APIs            Validate          Deduplicate         BM25 Index        Re-rank
    |               |                  |                  |                  |
    v               v                  v                  v                  v
                    [PostgreSQL + pgvector]

Notice there is no vector-database-as-a-service in this diagram. PostgreSQL with pgvector handles both the relational metadata and the vector embeddings. More on that decision later.

Chunking Strategy: Where Most RAG Pipelines Fail

Chunking is the single most impactful decision in a RAG pipeline, and it is purely a data engineering concern. The LLM never sees your chunking logic. It only sees the consequences.

Why Naive Chunking Fails

The most common approach is fixed-size chunking: split text into 512-token blocks with some overlap. This is fast to implement and terrible in practice because it:

• Splits sentences mid-thought, destroying semantic coherence
• Separates context from the statements that depend on it
• Creates chunks where the first half is about topic A and the second half about topic B
• Loses document structure (headers, sections, lists)

Semantic-Aware Chunking

The pipeline implements a hierarchical chunking strategy that respects document structure:

def chunk_document(
    document: ParsedDocument,
    max_tokens: int = 512,
    overlap_tokens: int = 50,
) -> list[Chunk]:
    sections = split_by_structure(document)
    chunks = []
    for section in sections:
        if count_tokens(section.text) <= max_tokens:
            chunks.append(create_chunk(
                text=section.text,
                metadata=section.metadata,
            ))
        else:
            sub_chunks = split_by_sentences(
                section.text,
                max_tokens=max_tokens,
                overlap_tokens=overlap_tokens,
            )
            for sub in sub_chunks:
                sub.metadata.update(section.metadata)
                chunks.append(sub)
    return chunks

The key principles:

1. Respect document structure first. Split on headers, sections, and paragraphs before resorting to sentence splitting.
2. Preserve metadata. Every chunk carries its source document, section hierarchy, page number, and creation date. This metadata is critical for citation and freshness.
3. Sentence-boundary splitting. When a section exceeds the token limit, split at sentence boundaries, never mid-sentence.
4. Contextual overlap. The overlap between chunks includes the last 1-2 sentences of the previous chunk, maintaining continuity.

Chunk Enrichment

Before embedding, each chunk is enriched with contextual information that improves retrieval:

• Section title prepended: If a chunk is from a section titled "Return Policy," that title is prepended to the chunk text before embedding
• Document title included: The source document title provides global context
• Entity extraction: Key entities (product names, dates, policy numbers) are extracted and stored as filterable metadata

This enrichment means the embedding captures not just what the chunk says, but where it sits in the broader document context.

pgvector vs Dedicated Vector Databases

The most controversial architectural decision in this pipeline is using PostgreSQL with pgvector instead of a dedicated vector database like Pinecone, Weaviate, or Qdrant.

The reasoning is pragmatic:

Operational simplicity. Every team already runs PostgreSQL. Adding pgvector is an extension install, not a new service to provision, monitor, secure, and pay for. Your existing backup, replication, and failover strategies just work.

Transactional consistency. With pgvector, your vector embeddings and relational metadata live in the same database. When you update a document, you can update its chunks, embeddings, and metadata in a single transaction. Dedicated vector databases require you to coordinate updates across two systems.

Filtering performance. RAG queries almost always include metadata filters (by document type, date range, access permissions). In PostgreSQL, these are standard WHERE clauses on indexed columns. In dedicated vector databases, metadata filtering is bolted on and often limited.

Scale reality check. pgvector with HNSW indexing handles millions of vectors with sub-100ms query times. Unless you are building a search engine over billions of documents, pgvector is sufficient. Most enterprise RAG deployments have thousands to low millions of chunks.

The tradeoff is that pgvector does not match the raw throughput of purpose-built vector databases at extreme scale. But for the 95% of RAG deployments that are not at extreme scale, the operational simplicity is worth far more.

class VectorStore:
    def __init__(self, connection_pool: AsyncConnectionPool) -> None:
        self._pool = connection_pool

    async def upsert_chunks(
        self,
        chunks: list[EmbeddedChunk],
    ) -> int:
        async with self._pool.connection() as conn:
            async with conn.transaction():
                for chunk in chunks:
                    await conn.execute(
                        """
                        INSERT INTO chunks (
                            chunk_id, document_id, content,
                            embedding, metadata, updated_at
                        ) VALUES (%s, %s, %s, %s, %s, NOW())
                        ON CONFLICT (chunk_id)
                        DO UPDATE SET
                            content = EXCLUDED.content,
                            embedding = EXCLUDED.embedding,
                            metadata = EXCLUDED.metadata,
                            updated_at = NOW()
                        """,
                        (
                            chunk.chunk_id,
                            chunk.document_id,
                            chunk.content,
                            chunk.embedding,
                            Json(chunk.metadata),
                        ),
                    )
        return len(chunks)

Note the upsert pattern. This is idempotent by design: re-running the pipeline with the same documents produces the same result without duplicates. Standard data engineering practice.

Hybrid Search: Vector + Keyword With RRF

Pure vector search has a well-known weakness: it struggles with exact matches. If a user asks "What is policy HR-2024-047?" a vector search might return chunks about HR policies in general rather than the specific policy number.

The solution is hybrid search, combining vector similarity with keyword matching (BM25) and fusing the results using Reciprocal Rank Fusion (RRF).

async def hybrid_search(
    query: str,
    query_embedding: list[float],
    top_k: int = 10,
    vector_weight: float = 0.6,
    keyword_weight: float = 0.4,
) -> list[SearchResult]:
    vector_results = await vector_search(
        query_embedding, top_k=top_k * 2
    )
    keyword_results = await keyword_search(
        query, top_k=top_k * 2
    )
    fused = reciprocal_rank_fusion(
        result_lists=[vector_results, keyword_results],
        weights=[vector_weight, keyword_weight],
        k=60,
    )
    return fused[:top_k]

RRF works by converting ranked positions into scores: score = weight / (k + rank). This normalizes across the two search methods regardless of their native scoring scales.

PostgreSQL makes this particularly clean because both search types live in the same database. The vector search uses pgvector's <=> operator (cosine distance), and the keyword search uses PostgreSQL's built-in tsvector full-text search. No external search service required.

Dynamic Weight Adjustment

The pipeline includes query classification that adjusts the vector/keyword weights based on query characteristics:

• Queries containing identifiers, codes, or exact phrases get higher keyword weight (0.3/0.7)
• Conceptual or semantic queries get higher vector weight (0.7/0.3)
• Default balanced weight (0.6/0.4) for ambiguous queries

Citation Validation: The Trust Layer

RAG without citation is just a chatbot that sounds confident. The pipeline implements citation validation to ensure every generated claim can be traced back to a specific source chunk.

The approach works in two stages:

1. Chunk attribution: The LLM is instructed to tag each statement with the chunk ID(s) it drew from
2. Post-generation validation: A validation step checks that each cited chunk actually supports the claim made

def validate_citations(
    response: GeneratedResponse,
    retrieved_chunks: list[Chunk],
) -> ValidationReport:
    chunk_map = {c.chunk_id: c for c in retrieved_chunks}
    validations = []
    for citation in response.citations:
        chunk = chunk_map.get(citation.chunk_id)
        if chunk is None:
            validations.append(CitationCheck(
                status="INVALID",
                reason="Referenced chunk not in retrieval set",
            ))
            continue
        similarity = compute_similarity(
            citation.claim, chunk.content
        )
        validations.append(CitationCheck(
            status="VALID" if similarity > 0.75 else "WEAK",
            similarity=similarity,
            source_document=chunk.metadata["source"],
        ))
    return ValidationReport(checks=validations)

Citations flagged as WEAK or INVALID are either removed from the response or flagged for the user. This is not perfect, but it catches the most egregious hallucinations where the model fabricates a source.

Embedding Freshness: The Forgotten Maintenance Problem

Once your RAG pipeline is in production, you face a maintenance problem that most tutorials ignore: embedding freshness.

Documents change. Policies get updated. Knowledge bases evolve. Your embeddings need to stay current. The pipeline implements a change detection and re-embedding workflow:

1. Content hashing: Each document and chunk has a content hash. When the source document changes, the hash changes.
2. Incremental re-embedding: Only chunks whose content hash differs from the stored version are re-embedded. This avoids the cost of re-embedding your entire corpus on every update.
3. Stale detection: A scheduled job identifies chunks whose source documents have not been re-crawled within a configurable freshness window and flags them for review.
4. Version history: Previous versions of chunks are soft-deleted, maintaining an audit trail of how the knowledge base evolved.

This is standard data pipeline thinking applied to embeddings: idempotent updates, change detection, incremental processing, and audibility.

Performance: What Actually Matters

After building and testing this pipeline, here is what actually moves the needle on RAG quality, in order of impact:

1. Chunking quality - Semantic-aware chunking with metadata enrichment improved retrieval relevance by roughly 35% compared to fixed-size chunking in our evaluations
2. Hybrid search - Adding BM25 keyword search alongside vector search improved exact-match queries from 45% to 89% accuracy
3. Chunk enrichment - Prepending section titles and document context to chunks before embedding improved retrieval by roughly 20%
4. Embedding model choice - Switching between embedding models (e.g., OpenAI text-embedding-3-large vs open-source alternatives) had a measurable but smaller impact than the above factors
5. LLM choice - The generation model had the least impact on overall system quality, provided the retrieval was good

This ordering reinforces the thesis: RAG is a data engineering problem. The data pipeline decisions (chunking, indexing, search strategy) matter more than the AI decisions (which embedding model, which LLM).

The Practical Takeaway

If your RAG system is underperforming, do not reach for a better model. Fix your data pipeline:

• Audit your chunking strategy. Are chunks semantically coherent? Do they carry sufficient context?
• Implement hybrid search. Pure vector search will always fail on exact-match queries.
• Use pgvector unless you have a specific scale requirement that demands a dedicated vector database. The operational simplicity pays for itself.
• Build citation validation into your pipeline. Trust is a feature, not a nice-to-have.
• Treat embedding freshness like you treat data freshness in any other pipeline. Stale embeddings are stale data.

The rag-knowledge-base-pipeline repository is a working implementation of these principles. Clone it, point it at your documents, and see the difference that proper data engineering makes in RAG quality.

RAG is not an AI problem. It is a data problem. And data problems are what data engineers solve.