Big Picture 9: Inside the Model, a Full Trace, and Limitations

This document zooms into Big Picture 4. First, how one token is made. Then one real request followed from start to finish. Then how the model was trained, and why its limitations follow from that training.

How one token is made

Example question: "When will it rain in Mumbai?". The model does not understand text, it needs numbers.

• 1 Text: the input. Text is not understood directly, it has to become numbers.
• 2 Tokenization: text is split into sub-word pieces and each token gets an ID. In English one token is roughly 4 chars, and Hinglish takes more tokens.
• 3 Embeddings: each token ID becomes a vector, a list of thousands of numbers. Tokens with similar meanings end up close together. Position information is added too.
• 4 Transformer layers: in self-attention each token looks at all earlier tokens (for example linking "when" to "rain"). The feed-forward part applies facts and patterns learned in training. This pair repeats across many layers. Anthropic does not publish Claude's exact architecture, this is the general working of any modern Transformer.
• 5 Next-token probabilities: after the last layer a probability is computed over the whole vocabulary (hundreds of thousands of tokens). In the example Mumbai gets 32%, Monsoon 21%. These numbers are illustrative only.
• 6 Choose + repeat: one token is chosen, added to the text, and step 4 runs again for the new token. This is autoregressive generation. It stops on a stop token, max_tokens, a stop sequence or a tool call.
• Thinking (extended / adaptive): not a separate brain. It is the same next-token generation, with reasoning tokens written first in a thinking block. In adaptive mode the model decides how much to think, can think again after a tool result, and thinking tokens are billed like output.
• Tool call inside the model: the tool's name, description and JSON schema are in the context. Through training the model writes a tool_use block when needed. Generation stops there, outside code runs the tool, and the result comes back as tokens.
• Images and PDFs: images are cut into 28x28 pixel patches and each patch is one visual token. For a PDF both the text and the image of each page go in. The output is always text, Claude does not generate images.
• KV cache to Prompt caching: every token gets Key and Value vectors that are kept during generation (the KV cache). Prompt caching takes the same idea across requests, so a cache read is about 10x cheaper and faster. Change even one byte of the prefix and the cache breaks.
• Context window limit: every new token attends to all previous tokens, so a longer context needs more compute and memory. Current models have a limit of 1M tokens. In very long contexts the middle can get less attention, so put important documents at the top and the question at the end.
• Same question, different answer: the model samples from the probabilities instead of always picking the top token. For reliable output use a clear prompt, structured outputs, examples and evals. The temperature parameter has been removed in the new API version.

One real request, traced end to end

A user on claude.ai with web search on asks: "Will it rain in Pune tomorrow? Should I take an umbrella?". Follow it through the earlier stages.

• 1 App builds the context: the claude.ai system prompt, the web_search tool, memory (city = Pune) and the new message. System prompt and tools are stable, so the cache breakpoint goes right after them.
• 2 API gateway: auth OK, rate limit OK, cache hit (about 3,000 tokens read cheaply), safety OK, then route to the model.
• 3 Model thinks: this is tomorrow's weather, it cannot be in the training data (knowledge cutoff), so it should search.
• 4 Model asks for a tool: server_tool_use web_search with the query "Pune weather tomorrow". Web search is a server tool, so Anthropic runs it itself within the same call. For a client tool, stop_reason would be tool_use and your code would run it.
• 5 Tool result comes back: web_search_tool_result with titles, snippets and URLs enters the context as tokens, and the model generates again from Big Picture 4 with fresh data.
• 6 Final answer: text with citations, something like "Yes, about a 70% chance tomorrow... take an umbrella.". Nothing else is needed, so stop_reason is end_turn and the app streams the text.
• 7 Bill and next turn: cached tokens, new input tokens, output tokens (thinking + text) and one web search. The next turn carries the whole history including the search results, but the old part is cheaper thanks to the cache.

How the model is built, and why it has limits

Model behaviour has its roots in training, so understanding limitations starts with understanding training.

• 1 Pre-training: next-token prediction on internet text, books and code. The model learns language, grammar, facts, coding and reasoning patterns. Data stops at a date, which creates the knowledge cutoff. The result is a base model that just continues text.
• 2 Supervised fine-tuning: from good question-answer examples the model learns to talk like an assistant, follow instructions, use formats and write tool_use.
• 3 RLHF: humans compare two answers, a reward model is built from that, and reinforcement learning teaches the model to give the answer people prefer.
• 4 Constitutional AI / RLAIF: based on written principles (the constitution) the model critiques and revises its own answers, and RL is done with AI feedback. This teaches it to stay harmless and honest, and when to say no.
• 5 Agentic + character training: RL on tools, coding environments and long tasks, plus character (curious, direct, avoiding sycophancy).
• 6 Evals, then Deploy: benchmarks, red-teaming, the ASL checks of the RSP and the system card, then release in the API and apps.

Each limitation below has a cause and a fix. Note: the fix is almost always something outside the model.

• Hallucination: the model writes the most probable next token instead of looking things up, so wrong but confident text can still look probable. Fix: RAG / documents, citations, permission to say "I do not know", web search and a verification step.
• Knowledge cutoff: the data ends at a date. Fix: web search / fetch, RAG and tools for live data.
• Stateless: weights do not change during a request and every call is separate. Fix: send the history every time, and use memory files, Projects, CLAUDE.md and compaction.
• Exact math / counting: numbers are tokens too, so the calculation is predicted, not computed. Fix: code execution, calculator tools and thinking.
• Context limit: the compute and memory cost of attention. Fix: caching, compaction, context editing, subagents and RAG.
• Different output every run: sampling. Fix: structured outputs, clear prompts + examples, and evals.
• Prompt injection: to the model, both instructions and data are just tokens. Fix: tool results are data, least privilege, permission modes, human approval and a sandbox.
• Does nothing by itself: the model only writes text / tool_use, and the harness runs the tools, so control stays with you.