How an LLM actually works

There is a version of this answer that is too shallow ("it predicts the next word") and a version that is too deep ("multi-head self-attention projects queries, keys and values..."). The first sounds like you read a tweet. The second, from a fresher, usually invites a follow-up you cannot survive. This lesson is the middle.

The one mechanism

A large language model does exactly one thing: given a sequence of tokens, it produces a probability distribution over what the next token should be.

A token is a chunk of text — usually a word or a piece of one. "unbelievable" might be three tokens (un, believ, able). The model has a fixed vocabulary of perhaps a hundred thousand tokens, and its output is a score for every single one of them.

To generate a sentence, that step runs in a loop:

prompt:  "The capital of France is"
step 1 → distribution over 100k tokens; " Paris" has the highest probability → pick it
step 2 → input is now "The capital of France is Paris"; next likely token is "."
step 3 → "." → then an end-of-text token → stop

That is the whole generation loop. Everything else — chat, code, summarising, translation — is this loop with different text in front of it.

Where the knowledge lives

During training the model saw an enormous amount of text and adjusted billions of internal numbers (parameters) to get better at that one prediction task. What it learned is baked into those numbers. There is no database inside it, no lookup table of facts, no copy of its training text it can consult.

This single fact answers a family of interview questions:

• "Does it know today's date?" No. It knows what was in its training data, which stopped at some point. Anything after that it cannot know unless you tell it in the prompt.
• "Is it learning from my chat?" Not while it runs. The parameters are fixed at inference time. Your conversation influences the input, not the model.
• "Why does it make things up?" Because it was trained to produce likely text, not true text. A confident, well-formed, false sentence scores well on likelihood. This is what "hallucination" is — not a bug in the code, but the objective working as specified.

That last one is worth being precise about, because it is asked constantly and answered badly. Hallucination is not the model lying or malfunctioning. It is the model doing exactly what it was trained to do, in a case where fluent and true come apart. A model asked about a citation it never saw will produce a plausible-looking citation, because plausible-looking is the target.

Why the same prompt gives different answers

The model outputs a probability distribution, and something has to choose from it. Temperature controls how that choice is made. At temperature 0 you take the highest-probability token every time and the output is (nearly) deterministic. Raise it and lower-probability tokens get a real chance, which reads as more varied — and more prone to drifting off.

A useful thing to say in an interview: "For anything where correctness matters I keep temperature low; the variety that helps creative writing is exactly what you don't want in a factual answer."

The context window

The model can only see a fixed number of tokens at once — its context window. Everything it knows about your specific question must fit in there: your prompt, the conversation so far, and any documents you passed in.

This constraint is the entire reason the next two lessons exist. You cannot paste a 400-page manual into the window, so you need a way to find the four paragraphs that matter and pass only those. That is retrieval, and it is what RAG is.

Attention, honestly

If asked, you should know the word. The architecture is called a transformer, and its key mechanism is self-attention: when processing a token, the model weighs how much every other token in the context should influence it. That is how it resolves "the trophy didn't fit in the suitcase because it was too big" — attention lets it draw more strongly from trophy than from suitcase.

That is enough. If a panel wants matrix shapes, they are interviewing for a research role and they know you are a fresher.

The four-sentence answer

Learn this shape, then say it in your own words:

An LLM predicts the next token in a sequence, over and over, and everything it does is that loop. What it knows is stored in its parameters from training, not in a database, which is why it has a knowledge cutoff and why it invents plausible-sounding details when it has not seen the answer. It can only see a fixed context window, so anything specific to my question has to be put in front of it. That constraint is why retrieval exists.

That answer earns the follow-up "so how does retrieval work?" — which is the next lesson, and exactly where you want the conversation to go.