Blog
Decision Making·15 min read

Prompt engineering vs fine-tuning: when to choose each

Cost, data volume, latency, and maintenance criteria to decide between iterating prompts, RAG, or training/adapting a model.

Code flowing through a prism with light beams

Fine-tuning is not a substitute for messy data or vague requirements. Prompt engineering and RAG cover a huge slice of cases with shorter iteration cycles. The choice should be economic and operational—not ideological.

The spectrum of LLM customization

You have four levels from simplest to hardest: (1) Prompt engineering—zero/few-shot, no extra training, (2) RAG—inject knowledge via retrieval, (3) Fine-tuning—train on domain-specific data, (4) Train from scratch—only for foundation labs. We focus on the first three.

When prompt engineering is enough

Good prompts go surprisingly far. Most apps don’t need fine-tuning. Prefer prompts when the base model already knows enough, you need fast iteration, output shape can be specified in instructions, and you have fewer than ~100 examples of desired behavior.

Prompt techniques

  • Zero-shot: clear instructions, no examples
  • Few-shot: 1–5 examples for format and tone
  • Chain-of-thought: ask the model to reason step by step
  • Self-consistency: sample multiple answers, pick the consensus
  • Role prompts: “you are an expert in X”

Prompt limitations

Finite context, token cost of large few-shot prompts, latency from long prompts, and niche knowledge the model never saw in pretraining.

When RAG is the right choice

RAG injects specific knowledge without retraining. Use it when facts change often, you need citations, data is too large for few-shot, and the task is primarily document QA.

RAG advantages

  • Update knowledge by reindexing
  • Source citations for transparency
  • No GPUs or labeled training sets required
  • Access control via document filters
  • Swap retriever or LLM independently

RAG limitations

Operational complexity, added latency (embed + retrieve + generate), quality depends on retrieval, and it doesn’t teach wholly new tasks—only provides context.

When to consider fine-tuning

Fine-tuning helps when the model must internalize behavior or domain style. Consider it with thousands of high-quality labels, very specific output format, inference cost from huge prompts, or privacy constraints on external APIs.

Types of fine-tuning

  • Full fine-tuning: all weights (expensive, overfitting risk)
  • LoRA: small adapter layers (far more efficient)
  • QLoRA: LoRA + quantization for smaller GPUs
  • Instruction tuning for domain-specific following
  • RLHF for human preference alignment

Problems fine-tuning fixes that prompts don’t

Examples: many-class text classification, rare domain entities, proprietary mini-languages, highly specific brand voice.

Decision criteria

Use this decision tree between prompt, RAG, and fine-tuning.

1. Do you have labeled data?

  • <50 examples → prompt (few-shot)
  • 50–500 → advanced prompts or RAG for factual knowledge
  • 500–5000 → fine-tuning may help—still validate prompts first
  • >5000 quality examples → fine-tuning often worthwhile

2. How often does knowledge change?

  • Weekly/monthly → RAG
  • Rarely (yearly) → fine-tuning can work
  • Never → any approach can fit

3. What does inference cost today?

Huge prompts at high QPS can cost tens of thousands per month—a smaller fine-tuned model can pay back the investment.

4. Do you need explainability?

  • Citations required → RAG
  • Only the final answer matters → prompt or fine-tuning

5. Do you have ML expertise and infra?

  • No ML team → prompt or RAG
  • Experienced ML team → fine-tuning is viable

Hybrid combinations

Production systems combine techniques: fine-tuned base for domain tone, RAG for fresh facts, prompts for per-endpoint behavior.

Typical hybrid architecture

Example: fine-tuned model for medical terminology and style, RAG for up-to-date policies, prompts for response format and guardrails.

Fine-tuning + RAG together

  • Fine-tune the task (classify, extract, summarize)
  • RAG supplies current facts
  • Prompts tune final formatting

Costs: a realistic comparison

Support chatbot example: ~50k queries/month.

Option 1: Prompting GPT-4

  • Roughly $0.09/query with large context
  • ~$4.5k/month at 50k queries
  • No training setup

Option 2: RAG + smaller model

  • Lower per-token LLM cost + embeddings + vector DB (~$70/mo tier)
  • Often ~$200–400/month at this scale
  • Engineering time to build the pipeline

Option 3: Fine-tuned smaller model

  • One-time training cost, then cheaper inference
  • Months of data prep and evaluation

Incremental roadmap

Don’t start with fine-tuning. Start simple and evolve.

Phase 1: Validation (weeks 1–4)

  • Prompt the base model
  • Iterate with few-shot
  • Confirm product value
  • Collect real queries

Phase 2: Knowledge (months 2–3)

  • Add RAG if you need proprietary facts
  • Tune chunking, retrieval, prompts
  • Measure latency, cost, accuracy

Phase 3: Specialization (months 4–6)

  • Consider fine-tuning with >1k quality labels if RAG costs or accuracy plateau
  • Prefer LoRA/QLoRA first
  • A/B against prompt+RAG before committing

Conclusion

Pick the simplest tool that solves the problem. Iteration speed and maintenance: generally prompt > RAG > fine-tuning. Fine-tune only with volume, data, expertise, and clear ROI.

The model is one part of the system—sometimes the bottleneck is data quality, unclear requirements, or UX. Fix those before defaulting to fine-tuning.