# Fine-tuning or RAG Optional The ways to adapt a language model to a task, prompting, retrieval, fine-tuning and distillation, what each changes, what data each needs, and how to choose between them. > [!NOTE] > > **Before reading** > > [LLM applications and evaluation]() ยท [Training data management]() ## Why it matters When a model does not perform well enough out of the box, teams often reach for fine-tuning first. Fine-tuning is the right tool for some problems and the wrong one for the most common one, missing knowledge. Choosing well saves months and keeps answers current and governed. ## How it works Four techniques change different things. Prompting changes the instructions and examples given with each request. It is the cheapest, fastest to change, and should be exhausted first: clear instructions, the output format, a few good examples. Retrieval-augmented generation (RAG) changes what information the model sees for each request, by retrieving relevant documents and adding them to the prompt. It is the answer to missing or changing knowledge: product details, policies, recent events. Knowledge is updated by updating documents, access rules apply per user, and answers can cite their sources. Fine-tuning changes the model's weights by training it further on examples of inputs and desired outputs. It is good at teaching behaviour: a consistent style or tone, a strict output format, a specialised classification, domain language the model handles poorly. It is poor at teaching facts: knowledge trained into weights cannot be cited, is hard to update or delete, and is mixed with everything else the model knows. Parameter-efficient methods, such as LoRA, train a small set of extra weights instead of the whole model, which makes fine-tuning much cheaper. Fine-tuning can also cause catastrophic forgetting, where the model loses general abilities it had, so evaluation covers more than the target task. Distillation trains a smaller, cheaper model to imitate a larger one on a specific task, using the large model's outputs, or its output probabilities where available, as training data. It reduces cost and latency once a task works well with an expensive model. ``` flowchart TB Q{What is wrong
with the outputs?} -->|instructions unclear,
format inconsistent| PR[Improve the prompt] Q -->|missing or outdated
knowledge| RAG[Add or improve
retrieval] Q -->|behaviour, style, format
still wrong after prompting| FT[Fine-tune on
examples] Q -->|works, but too
slow or costly| DS[Distil to a
smaller model] ``` The techniques combine: a fine-tuned model can still use retrieval for facts, and a distilled model can be used in a RAG application. The data requirements differ, and are where the data platform matters. Prompting needs a handful of good examples. RAG needs well-managed documents, chunks, embeddings and access metadata. Fine-tuning needs hundreds to thousands of high-quality example pairs, curated, deduplicated, checked for personal data and rights, and versioned like any training set; its quality matters more than its size. Distillation needs a large set of realistic inputs and the large model's outputs, and the provider's terms must allow using outputs for training. Whatever is chosen, the evaluation set decides. Each technique is compared on the same evaluation, including cost and latency, before anything is adopted. ## Key terms - Prompting Guiding a model with instructions and examples in each request. - Retrieval-augmented generation (RAG) Adding retrieved documents to each request. - Fine-tuning Further training a model on task examples to change its behaviour. - LoRA A parameter-efficient fine-tuning method that trains small added weight matrices. - Distillation Training a smaller model to reproduce a larger model's outputs on a task. - Catastrophic forgetting A fine-tuned model losing general abilities it had before. ## Example The shop wants product descriptions written in its own friendly, concise tone, from product attributes. - Prompting with tone guidelines and three examples gets close, but the style drifts across categories, and evaluation by the copy team on 200 held-out products rates 62% of drafts as on-brand. - Facts are not the problem: attributes are given in the prompt, so RAG would not help. - The team fine-tunes a mid-sized open-weight model with LoRA on 1,800 approved descriptions, after removing duplicates and any customer names from reviews quoted in them. On the same 200 held-out products, not among the training descriptions and rated blind by the copy team, 88% of drafts are rated on-brand, at a fifth of the cost per description of the large hosted model. - The training set is tagged and recorded with the fine-tuned model version in the registry. ## Pitfalls - Fine-tuning to add knowledge. Use retrieval; facts in weights cannot be cited, updated or deleted. - Skipping prompting. Many problems disappear with better instructions. - Small, noisy fine-tuning sets. Quality of examples matters more than quantity. - No baseline. Without the same evaluation before and after, improvement is a belief. - Ignoring terms of use. Model and provider licences restrict fine-tuning and use of outputs for training. ## Learn more - [Hu et al., LoRA: Low-Rank Adaptation of Large Language Models]() - [Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network]() ## Related [Embeddings, vector search and RAG]() [Responsible AI and AI governance]() [Unstructured and multimodal data]()