Big Picture 13: Security, Choosing an Approach, and Evals

This document covers three interview-favourite topics: defense in depth, the decision tree for picking an approach and a model, and the eval loop that decides with numbers.

Security, defense in depth

Many layers from the outside in. If one layer fails, the next one protects you. At the centre are your data, code and systems. This is the full picture of the Safety rail from Big Picture 8.

• Organization (outermost): SSO, roles, audit logs, managed settings, ZDR, SOC 2 / ISO 27001 / ISO 42001.
• Platform: API keys / OAuth / WIF, workspaces, rate and spend limits, usage policy.
• Model: Constitutional AI training, safety classifiers, refusal + fallback.
• Prompt / context: system prompt rules, treating tool results as data (so prompt injection does not work), only trusted MCP servers and plugins.
• Harness: permission modes, allow / deny rules, PreToolUse hooks, human approval on risky actions.
• Execution (innermost): sandbox / VM / container, network allowlist, secrets in a vault, least-privilege tokens.
• Your data, code and systems: the very centre, the thing to protect.

Which approach, which model

The interview favourite: "Which approach and which model would you choose, and why?". Build the answer with a decision tree.

• Can one call do it? Classify, summarize, extract, Q and A, translate.
• Yes: Messages API, a single call, plus structured output and caching. Many items and not in a hurry: Batch API.
• Steps known in advance? A fixed pipeline such as extract, check, write.
• Yes: Workflow (your code): prompt chaining, routing, parallel calls. Predictable and cheap. If not, the model decides the steps, which makes it an agent.
• Which tools are needed? Only your functions, or files / bash / code too.
• Only your own: API + Tool Runner (or a manual loop if you want full control).
• Who runs the infra? Your own server, or Anthropic.
• You run it: Claude Agent SDK, Claude Code's loop and built-in tools in your process.
• Anthropic: Managed Agents, a hosted loop and sandbox, sessions, cron, vaults.
• Shortcut: developer + codebase, go to Claude Code. Non-developer, claude.ai (Projects, connectors). Before building an agent, ask 4 questions: is the task complex, is it valuable, can Claude do it, can mistakes be caught?

Picking the model and effort

• Which model: Opus 5.5 is the default, for complex coding and agents. Fable 5.1 is for the hardest, multi-hour work where cost matters less. Sonnet 5.5 is for everyday coding, enterprise chatbots and speed. Haiku 5.5 is for high-volume classify / route / extract work and sub-agents.
• Tune effort: low for chat, classification and sub-agents. Medium for cheap but good. High for intelligence-sensitive work. xhigh for coding and agentic work. Max when correctness matters more than cost.
• The rule: first try a capable model with low effort. If it is still too expensive, try a smaller model. Compare both with evals, on cost per completed task.

Evals and the improvement loop

After changing a prompt, model or tool, decide with numbers, not feelings. The eval loop has seven steps.

• 1 Success criteria: specific and measurable. For example accuracy 90%, latency < 3s, cost < $0.01 per request, polite tone.
• 2 Test set: real user questions, edge cases and tricky / adversarial ones. Keep train and test separate.
• 3 Grader: code (exact match, regex, tests), LLM-as-judge (rubric) or human (the most accurate).
• 4 Baseline: the score, cost and latency of the current prompt or model.
• 5 One change: prompt, model, effort, tool description or RAG settings. One at a time.
• 6 Rerun + compare: did the score go up? What happened to cost and latency? Check on the test set too, to see you have not overfit.
• 7 Ship + monitor: production failures and thumbs-downs become new test cases (back to step 2). Run the same eval on every new model or feature.
• For agents: look at the trajectory too. Right tools? How many steps? How many tokens? Any wrong action?
• LLM-as-judge tips: a clear rubric (1 to 5), examples, check against human-graded cases, and a reason with each score.
• Tools: the Console Evaluate tab, a Python script, claude plugin eval, OpenTelemetry.