Skip to content
AI Engineer Guide.

concept · concepts · intermediate · concept 1 of 28

LLM

0 of 28 complete

Why it matters

An LLM is the component that turns an application's intent into language. It is also the component that behaves differently on Tuesday. Understanding what the model actually is — a conditional distribution over tokens, not a database lookup and not a reasoning engine by construction — explains most of the engineering work around it: retrieval, evaluation, schema validation, guardrails, and cost control.

The practical consequence is that an LLM application is a probabilistic system. You cannot read the source of an answer. You can only constrain the inputs, sample deterministically where possible, validate the output, and measure the result. Teams that treat the model as an oracle ship brittle products; teams that treat it as a capable but unreliable function ship systems around it.

Mental model

Think of a very well-read autocomplete that has read a large fraction of the public internet. Given a prefix, it produces a probability for every token in its vocabulary, and generation walks that distribution one token at a time. The "knowledge" is compressed into weights during training; the "reasoning" is whatever computation the forward pass performs over the tokens you supplied.

The analogy breaks in two places. First, post-training changes the distribution substantially: an instruction-tuned model is not just a raw text continuer, it has been shaped to answer, refuse and format. Second, the model has no memory between calls. Any continuity you observe comes from an application re-sending history or storing state outside the model.

How it works

  1. 1Tokenise the prompt into vocabulary IDs, including special tokens that mark roles and boundaries.
  2. 2Embed the IDs, add positional information, and pass the sequence through transformer layers.
  3. 3At the final position, project the hidden state onto the vocabulary to produce logits.
  4. 4Turn logits into probabilities, apply a decoding rule, and select one token.
  5. 5Append the token and repeat until an end condition is met, then detokenise.

Pretraining optimises the same objective at every position: predict the next token. The model never sees a supervised label for "what a good answer looks like" during this stage. That is why pretraining produces a general text model and why it will happily continue a prompt with more prompt-like text rather than answer it.

Post-training supplies the assistant behaviour. Instruction tuning trains on demonstrations of instructions and responses. Preference optimisation — RLHF or a direct method such as DPO — shifts the distribution toward responses people prefer. The result is a model that follows requests far more reliably, at the cost of a narrower behavioural range and new failure modes such as over-refusal.

At inference time the model is a pure function of its input tokens and its weights. Temperature and truncation decide how the distribution is sampled. The context window bounds how many tokens can be considered at once. The KV cache makes generation affordable by avoiding recomputation, but it is an implementation detail of serving, not something the model "remembers" semantically.

Quick facts

Input unit
Tokens, not words
State
Stateless between calls
Output
A distribution over the vocabulary
Training
Pretraining, then post-training
Main knobs
Context, sampling, tools, schema
Guarantee
None on factuality

The objective, in one line

Training maximises the likelihood of the observed text under an autoregressive factorisation:

pθ(x1,…,xn)=∏t=1npθ(xt∣x<t)p_\theta(x_1,\dots,x_n) = \prod_{t=1}^{n} p_\theta(x_t \mid x_{<t})

Every capability the product relies on — summarising, extracting, classifying, calling a tool — is expressed as a conditional continuation of this form. The vocabulary is fixed at training time, which is why tokenisation and the context window leak into every cost and latency estimate.

The generation loop

Every provider SDK wraps roughly this loop. It is shown as plain pseudocode because the important part is the shape, not a vendor signature.

# Illustrative: the loop that every hosted API hides behind a streaming response.
def generate(prompt_ids: list[int], model, max_new_tokens: int = 256) -> list[int]:
    ids = list(prompt_ids)
    for _ in range(max_new_tokens):
        logits = model.forward(ids)[-1]              # one score per vocabulary item
        next_id = sample(softmax(logits / temperature))
        if next_id == model.eos_id:
            break
        ids.append(next_id)
    return ids[len(prompt_ids):]

Two properties of this loop matter operationally. Cost and latency scale with the number of output tokens, because each one requires a full forward pass. And the model cannot revise what it has already emitted, so early mistakes propagate — which is why structured output, verification and retries exist as separate layers.

Trade-offs

ChoiceGood forCostFails when
Frontier hosted modelHard reasoning, broad coverage, fastest startHighest per-token price, vendor dependencyThe task is narrow, high-volume, or latency-critical
Smaller hosted modelClassification, extraction, routingLower price, weaker on multi-step tasksThe task needs judgement or long context
Open-weights model, self-hostedData control, fine-tuning, predictable volumeGPU capacity and operationsTraffic is spiky or the team has no serving experience
Fine-tuned small modelStable, repetitive format-heavy tasksTraining data and maintenanceRequirements change faster than you can retrain
Trade-off mapInteractive

Good for

Hard reasoning, broad coverage, fastest start

Cost

Highest per-token price, vendor dependency

Fails when

The task is narrow, high-volume, or latency-critical

  1. The task is narrow, high-volume, or latency-critical
  2. The task needs judgement or long context
  3. Traffic is spiky or the team has no serving experience
  4. Requirements change faster than you can retrain

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Frontier hosted model is good for Hard reasoning, broad coverage, fastest start, costs Highest per-token price, vendor dependency, and fails when The task is narrow, high-volume, or latency-critical. Smaller hosted model is good for Classification, extraction, routing, costs Lower price, weaker on multi-step tasks, and fails when The task needs judgement or long context. Open-weights model, self-hosted is good for Data control, fine-tuning, predictable volume, costs GPU capacity and operations, and fails when Traffic is spiky or the team has no serving experience. Fine-tuned small model is good for Stable, repetitive format-heavy tasks, costs Training data and maintenance, and fails when Requirements change faster than you can retrain

Failure modes

Confident fabrication. The model produces a fluent claim with no source. Debug by separating retrieval failure from generation failure: if the supporting passage is absent from the prompt, no prompt change will fix it. Add an abstain path and check it against questions whose answers are known to be missing.

Silent context loss. The prompt exceeds the window or the output hits the token cap, and the answer is truncated. Debug by logging the counted prompt tokens, the output token count and the finish reason for every call. A finish reason of "length" is a bug in the budget, not in the model.

Non-reproducible behaviour. The same input yields different output after a provider upgrade, or even across identical calls. Debug by pinning model versions where the provider allows it, fixing sampling parameters, and treating any prompt change as a release with an eval run.

Stale knowledge. The model answers from training data when the current answer differs. Debug by checking whether the question is time-sensitive; if it is, route it to retrieval or a tool and require the answer to cite the fresh source.

Failure modes, one at a timeInteractive

Symptom

The model produces a fluent claim with no source.

Debug step

Separating retrieval failure from generation failure: if the supporting passage is absent from the prompt, no prompt change will fix it. Add an abstain path and check it against questions whose answers are known to be missing.

Figure. Each failure with its symptom and the check that would confirm it.

What to notice. A failure without a debug step is a rumour. Pick the symptom you are seeing; the check is the next action.

Text version

Failure modes: Confident fabrication: The model produces a fluent claim with no source. Debug by separating retrieval failure from generation failure: if the supporting passage is absent from the prompt, no prompt change will fix it. Add an abstain path and check it against questions whose answers are known to be missing. | Silent context loss: The prompt exceeds the window or the output hits the token cap, and the answer is truncated. Debug by logging the counted prompt tokens, the output token count and the finish reason for every call. A finish reason of "length" is a bug in the budget, not in the model. | Non-reproducible behaviour: The same input yields different output after a provider upgrade, or even across identical calls. Debug by pinning model versions where the provider allows it, fixing sampling parameters, and treating any prompt change as a release with an eval run. | Stale knowledge: The model answers from training data when the current answer differs. Debug by checking whether the question is time-sensitive; if it is, route it to retrieval or a tool and require the answer to cite the fresh source.

In production

Treat the model call as an external dependency with a cost, a latency distribution and a failure rate. Log the model identifier and version, token counts, sampling parameters, finish reason, and a hash of the rendered prompt. Monitor tokens per request and per user, p95 latency, error and retry rates, and refusal rate; each moves before a user complaint does. Keep a small evaluation set that runs on every prompt or model change, because "it looks better" is not evidence.

Common misconceptions

People think: The model looks facts up in a database inside its weights.

Actually: Weights encode statistical patterns from training. There is no lookup table and no provenance, which is why citation and retrieval have to be added by the application.

People think: A larger context window means the model remembers more.

Actually: A larger window allows more input tokens. Whether the model uses them well depends on position, relevance and the task, and long inputs can degrade accuracy.

People think: Temperature zero makes output deterministic.

Actually: It makes sampling greedy, but batching, hardware nondeterminism and provider-side changes can still produce variation. Reproducibility is a systems property, not a parameter.

Interview angle

Interviewers use "what is an LLM" as a filter for whether you understand the system boundary. A strong answer names the training objective, distinguishes pretraining from post-training, states that inference is stateless, and connects those facts to the engineering consequences: token budgets, retrieval, evaluation and validation. A weak answer lists product features. Practise with the LLM fundamentals questions and the prompting bank.

Sources