How Does an LLM Work? A Developer's Explanation

A large language model works by converting text into numbers, running those numbers through a network trained to predict what comes next, and repeating that prediction one piece at a time until it produces a full response. That's the mechanical answer. The more useful answer, if you're building software on top of one, is understanding why it behaves the way it does: why it sometimes hallucinates, why longer context costs more, why the same prompt can give different answers, and what's actually happening between your API call and the text you get back.

This article walks through the pipeline step by step, from raw text to generated output, with enough technical grounding to make architectural decisions in your own applications.

Step 1: Text becomes tokens

Models don't read words. They read tokens — chunks of text that might be a whole word, part of a word, or a punctuation mark. "Tokenization" is short for "programmable" no wait, let's not editorialize. A tokenizer splits "programmable" into pieces like program + mable, and common words like "the" usually stay as a single token.

This matters practically because:

• API pricing and context limits are measured in tokens, not characters or words.
• Roughly, 1 token ≈ 4 characters of English text, but this varies by language and content type (code and non-English text often tokenize less efficiently).
• A "128k context window" means 128,000 tokens total, shared between your input and the model's output.

Step 2: Tokens become vectors

Each token is mapped to a vector — a list of numbers, often several thousand dimensions long. This mapping is called an embedding, and it's learned during training so that tokens with similar meanings end up close together in that vector space. "Dog" and "puppy" sit nearer each other than "dog" and "spreadsheet."

At this point, the model isn't looking at words anymore. It's doing matrix math on high-dimensional numerical representations of meaning.

Step 3: Attention decides what matters

This is the core mechanism behind modern LLMs, introduced in the 2017 "Attention Is All You Need" paper and still the backbone of GPT, Claude, and every other major model family: the transformer architecture.

The key idea is self-attention: for every token in the input, the model calculates how relevant every other token is to it. When processing the word "it" in "the trophy didn't fit in the suitcase because it was too big," attention lets the model figure out that "it" refers to "trophy," not "suitcase," by weighing the relationships between all the tokens in the sentence simultaneously.

This happens across many layers (dozens in large models) and many parallel "attention heads" per layer, each specializing in different kinds of relationships — syntax, coreference, topic, tone. The output of all this stacked attention and feed-forward computation is, for each position in the sequence, a refined vector that encodes rich contextual meaning.

Step 4: Predicting the next token

At the final layer, the model converts its internal representation into a probability distribution over its entire vocabulary — tens of thousands of possible next tokens, each with a likelihood score. It picks one (more on how below), appends it to the sequence, and repeats the entire process to pick the next token.

This is why generation is sequential and why streaming responses feel natural: the model genuinely is producing one token at a time, not writing the full answer internally and revealing it gradually.

Two settings control how tokens get picked from that probability distribution:

• Temperature: lower values make the model pick high-probability tokens more consistently (more deterministic, more repetitive); higher values increase randomness and creativity.
• Top-p / top-k sampling: restricts the pool of candidate tokens to the most likely ones before sampling, avoiding low-probability, incoherent choices.

This sampling step is also the direct cause of non-determinism — ask the same question twice at nonzero temperature and you can get different phrasing, even though the underlying weights never changed.

Training vs. inference: two very different phases

Everything above describes inference — using an already-trained model to generate output. Training is a separate, much more expensive phase where the model's weights (the billions of numbers controlling every calculation above) are adjusted by:

1. Pretraining — predicting the next token across a massive text corpus, adjusting weights via backpropagation whenever the prediction is wrong.
2. Fine-tuning — further training on curated examples of good responses.
3. RLHF/RLAIF — using human or AI feedback to reward outputs that are helpful and penalize ones that are unsafe or low quality.

When you call an LLM API, none of this happens. The weights are frozen. You're only running inference — the forward pass through a fixed network. This is why the model has no persistent memory between separate API calls unless you explicitly resend the conversation history.

Why this matters when you're building with LLMs

Understanding this pipeline explains several things developers run into constantly:

• Context windows aren't free memory. Every previous message you resend gets re-tokenized and re-processed through the entire attention mechanism, which is why long conversations cost more and can hit limits.
• Hallucination isn't a bug in retrieval — it's a property of prediction. The model isn't looking anything up; it's generating the statistically likely next token based on patterns learned during training. If those patterns don't include the actual fact, it produces something plausible-sounding instead.
• Tool use extends what the model can do, not what it knows. When a model calls a function or API, it's still just predicting tokens — the tokens happen to form a structured request that your application executes and feeds back in.

If you're integrating an LLM into a product, the practical layer that matters is the API surface: authentication, streaming, structured tool calls, and usage tracking. SubToAPI turns your existing Claude access into that kind of clean HTTPS API — issue an sub_live_... key, send requests to /docs/messages, and get streaming responses via /docs/streaming or tool calls via /docs/tools without managing model infrastructure yourself. The /docs/quickstart guide covers the first request end to end, and you can start on a free trial at /signup.

questions

Does an LLM understand what it's writing, or just predict text? Mechanically, it predicts the next token based on learned statistical patterns. Whether that constitutes "understanding" is debated, but functionally, the model has no separate reasoning process outside of this prediction loop — sophisticated behavior emerges from it, not alongside it.

Why does the same prompt sometimes give different answers? Sampling settings like temperature introduce controlled randomness when picking the next token from a probability distribution. At temperature 0, output becomes far more consistent, though not always perfectly identical due to low-level numerical factors.

Why do LLMs have a maximum context length? Self-attention compares every token to every other token, so compute and memory costs grow quickly as input length increases. Context limits reflect the practical ceiling of what the model architecture and serving infrastructure can process efficiently.