RAG — the diagram you will be asked to draw

Retrieval-Augmented Generation is the answer to a specific problem: the model does not know about your documents, and you cannot fit them in the context window. So you find the relevant few passages and put those in the prompt.

That is it. RAG is not a model, a product, or a framework. It is a pattern: retrieve, then generate.

Two phases, and students conflate them

The single most common mistake in an interview is describing RAG as one flow. It is two, and they happen at completely different times.

Phase 1 — Indexing. Done once, ahead of time, offline.

documents → split into chunks → embed each chunk → store vector + original text

Phase 2 — Querying. Done every time a user asks something.

user question → embed the question → nearest-neighbour search → top-k chunks
             → build a prompt: instructions + retrieved chunks + question
             → LLM → answer (+ citations)

Why this distinction matters: the expensive, slow work happens in phase 1, and phase 2 is fast. If you describe it as one pipeline, the natural follow-up is "so you re-embed all the documents on every question?" — and now you are explaining something you did not mean.

The connecting insight is that both phases must use the same embedding model. The question's vector and the chunks' vectors have to live in the same space to be comparable. Knowing this is a strong signal; it is the kind of thing you only notice when you have built the thing.

Every arrow, defended

Step	What it does	The trade-off you should name
Chunking	splits documents	precision vs enough context (lesson 3)
Embedding	text → vector	model choice locks in your whole index
Vector store	fast nearest-neighbour	approximate — may miss the true best match
top-k	how many chunks to retrieve	too few misses the answer; too many buries it in noise and costs more
Prompt assembly	instructions + context + question	the instruction that says "answer only from the context" is doing real work
Generation	LLM writes the answer	it can still ignore your context, so citations matter

On k: more context is not better. Passing twenty chunks when three are relevant means the model has to find the signal, and it will sometimes latch onto a plausible irrelevant passage. It also costs more and is slower. A small k with good retrieval beats a large k with sloppy retrieval every time.

The three places it actually breaks

This is question shape 3 from lesson 1, and having a real answer here puts you ahead of most candidates.

1. Retrieval brings back nothing useful. The user asked something your documents do not cover, or asked it in wording that does not embed near anything relevant. The model now receives irrelevant context and a question, and — being a next-token predictor — will usually answer anyway. This is the worst failure mode, because the output looks exactly like a good answer.

The mitigation: a similarity threshold. If the best match scores below it, refuse: "I don't have information about that." Most tutorial projects have no threshold, which is why they answer everything with equal confidence.

2. The answer spans several chunks. The deadline is in one document and the fee is in another. If k is small, you get one and not the other, and the answer is confidently half-right. Real systems address this with re-ranking, query rewriting, or retrieving more and then filtering — all of which add complexity, which is why naming the problem is worth more than claiming to have solved it.

3. The model ignores the context. You gave it the passage; it answered from its training data instead, or blended the two. This is why answers should carry citations back to the chunk they came from — not as a nice-to-have, but as the only practical way to tell whether the answer came from your documents.

"RAG stops hallucination" — say something better

It does not. It reduces hallucination by putting real information in front of the model, and it makes hallucination detectable by giving you something to check the answer against.

But the failure modes above are all hallucination that RAG did not prevent — and case 1 is hallucination that RAG arguably caused, by feeding the model irrelevant context and an instruction to be helpful.

The precise version, which is a genuinely good interview answer:

RAG grounds the answer in retrieved text, so the model is far less likely to invent facts it was never given — and because every answer can cite its source, a wrong answer is checkable rather than invisible. What it does not do is guarantee correctness: if retrieval returns the wrong passages the model will still answer confidently, which is why I would set a similarity threshold and refuse below it.

Fine-tuning vs RAG

A frequent follow-up. Short version:

RAG adds knowledge. Use it for facts that change, are private, or are too numerous to train on. Update by re-indexing a document — minutes.
Fine-tuning adjusts behaviour. Use it for tone, format, or a task shape the model handles poorly. Update by retraining — expensive, and it needs a curated dataset.

The one-liner: RAG for what the model should know, fine-tuning for how it should behave. For a fresher project it is almost always RAG, and saying "fine-tuning would have been the wrong tool because my requirement was current information, not a different style" is a correct answer to "why did you choose that?".