RAG Systems Explained: How I Built a Production Knowledge Base

I built a RAG (Retrieval-Augmented Generation) system to power the knowledge base for this platform. Users can ask questions about AI tools, technologies, and best practices, and get answers grounded in our curated content rather than generic AI responses.

After three months of building, testing, and refining, I want to share how RAG actually works, the architecture decisions I made, and the mistakes I wish I had avoided.

Why RAG Instead of Fine-Tuning

Before building anything, I had to decide between fine-tuning a model and using RAG. Fine-tuning would teach a model our content and style, but it would be expensive to update every time we published new articles. RAG, on the other hand, retrieves relevant content in real time and uses it as context for generating answers.

For a knowledge base that grows weekly, RAG was the obvious choice. New content gets indexed automatically. Old content gets updated. The model does not need retraining. The retrieval step also means I can show users exactly which articles informed the answer.

The Architecture

My RAG system has four main components:

Document ingestion: Articles from our database are chunked into smaller pieces and converted to vector embeddings.

Vector storage: Embeddings are stored in a vector database for fast similarity search. I chose Supabase with pgvector because we already use Supabase for our main database.

Retrieval: When a user asks a question, it gets converted to an embedding, and the vector database returns the most similar chunks.

Generation: The retrieved chunks are passed to an LLM along with the user's question, and the LLM generates an answer grounded in the retrieved content.

Chunking: The Decision That Mattered Most

How you split documents into chunks has an enormous impact on retrieval quality. I learned this the hard way.

My first attempt used fixed-size chunks of 500 tokens with 50 token overlaps. The problem was that 500 tokens often split a meaningful paragraph in half. A user would ask about a specific tool's pricing, and the retrieval would return a chunk that contained the tool name but cut off before the pricing information.

I switched to semantic chunking, which splits documents at natural topic boundaries. This requires a secondary model call for each document during indexing, but the retrieval quality improvement was dramatic. Relevant answers went from about 60% accuracy to 85%.

For technical content with code examples, I also implemented special handling that keeps code blocks intact rather than splitting them across chunks. This was critical because code snippets are often exactly what users are looking for.

The Embedding Model Choice

Embeddings are mathematical representations of text that capture semantic meaning. The model you choose affects both quality and cost.

I tested three options: OpenAI's text-embedding-3-small, Cohere's embed-v3, and the open source BGE-large-en-v1.5 model.

OpenAI's model was the easiest to integrate and produced consistently good results. Cohere's model was slightly better at handling multi-language content. The BGE model was free to run locally but required more tuning.

I went with OpenAI's text-embedding-3-small. The quality was good enough, the cost is very low (about $0.02 per 1,000 documents), and the integration was straightforward.

Hybrid Search for Better Results

Pure semantic search sometimes misses exact matches. If a user searches for "Claude pricing," semantic search might return articles about Claude's features because they are semantically similar. But the user wants the pricing page.

I implemented hybrid search that combines semantic vector search with traditional keyword search. Using pgvector, I can run both searches in a single query and weight the results. The keyword search catches exact matches, and the semantic search catches conceptual matches.

This combination improved answer relevance noticeably, especially for queries containing specific product names, version numbers, or technical terms.

Re-Ranking for Precision

After the initial retrieval, I added a re-ranking step. A cross-encoder model scores each retrieved chunk based on how well it actually answers the user's question, not just how semantically similar it is.

The re-ranking step improved precision by about 15%. It is an additional API call per query, but the latency impact is minimal (about 200ms) and the quality improvement justifies it.

What I Would Do Differently

Looking back, I wish I had invested more time in the evaluation framework earlier. I spent weeks tuning parameters based on gut feeling. Once I built an automated evaluation pipeline with 100 test questions and human-labeled answers, I could measure the actual impact of each change.

I also wish I had started with a simpler architecture. My initial design had multiple retrieval paths, caching layers, and fallback mechanisms. The complexity made debugging difficult. The production system is much simpler and works better because of it.