Fine-Tuning vs RAG vs Prompt Engineering: Which Approach Works Best?

Every few weeks someone asks me which approach they should use to make an LLM work better for their specific use case. The honest answer is that it depends, but that is not helpful. So I am going to share what I have learned from actually implementing all three approaches in production, with real numbers and real trade-offs.

Quick Definitions

Before diving in, let me define what I mean by each approach:

Prompt engineering is crafting the input to an LLM to get better outputs. This includes system prompts, few-shot examples, chain-of-thought instructions, and output formatting constraints. You are not changing the model at all, just communicating with it more effectively.

RAG (Retrieval-Augmented Generation) is giving the model access to external information at inference time. You search a knowledge base, retrieve relevant documents, and include them in the prompt. The model generates responses grounded in those documents.

Fine-tuning is training the model further on your specific data. You provide examples of inputs and desired outputs, and the model adjusts its internal weights to better match your expectations. This changes the model itself.

When Prompt Engineering Is Enough

I always start with prompt engineering, and I recommend you do too. In my experience, about 60% of the problems people think require fine-tuning can be solved with better prompts.

I spent two weeks trying to fine-tune a model to format JSON output in a specific schema. Then I wrote a clearer system prompt with an explicit JSON schema example and a constraint instruction. The zero-shot prompt achieved 95% compliance on the format. The fine-tuned model achieved 97%. The two weeks of engineering effort were not worth a 2% improvement.

Prompt engineering works well when your task is straightforward, you can describe the desired output clearly, and the model already has relevant knowledge in its training data.

The cost is essentially zero beyond your time. There are no training runs, no infrastructure, no deployment complexity.

When RAG Is the Right Choice

I reach for RAG when the model needs access to information that is not in its training data, or when the information changes frequently.

For our platform's knowledge base, RAG was the obvious choice. We publish new content weekly. Fine-tuning every week would be expensive and impractical. RAG lets us index new content automatically and have it available immediately.

RAG also provides something fine-tuning cannot: source attribution. When our RAG system answers a question, it tells the user which article the answer came from. That builds trust and lets users verify the information.

The downsides of RAG are the infrastructure complexity and the retrieval quality dependency. If your retrieval returns irrelevant documents, the model's answers will be wrong. Investing in good chunking, embeddings, and retrieval logic is critical.

When Fine-Tuning Actually Makes Sense

After trying prompt engineering and RAG, fine-tuning is the right choice in specific situations.

I fine-tuned a model for our internal code review assistant. The task was to identify common security anti-patterns in JavaScript code. Prompt engineering got us to about 70% accuracy. RAG did not help much because the patterns are distributed across many codebases, not concentrated in specific documents. Fine-tuning on 500 annotated examples pushed accuracy to 92%.

Fine-tuning works when you have a narrow task where the model's general knowledge is insufficient, you have enough high-quality training data (usually 200+ examples minimum), and you need consistent performance that prompt engineering cannot achieve.

The costs are real. OpenAI charges for fine-tuning by the hour during training and by the token for inference with fine-tuned models. Training on 500 examples of our code review data cost about $12 in compute. Not bad, but not free either.

Decision Framework

Here is my practical decision tree:

Start with prompt engineering. Spend at least a few days refining your prompts before considering anything else.

If the model lacks specific knowledge, add RAG. Build a knowledge base and let the model search it at inference time.

If the model knows the domain but consistently makes specific types of errors or fails to follow your format, consider fine-tuning. Make sure you have enough training examples first.

If you need a combination, combine them. We use fine-tuned models with RAG in production. The fine-tuned model is better at following our specific format and tone. RAG gives it access to current information. Both approaches are complementary, not competing.

Cost Comparison

Based on my actual project costs:

• Prompt engineering only: $0 for setup, standard API costs for inference
• RAG: $50-200/month for vector database hosting, embedding costs, plus standard API costs
• Fine-tuning: $10-50 for training, 50% higher per-token inference costs, plus ongoing maintenance
• Combined RAG + fine-tuning: All of the above combined

My recommendation for most projects is prompt engineering first, then RAG if needed. Fine-tuning should be a last resort for narrow, high-value tasks where the other approaches have already been tried and proven insufficient.