RAG, Embeddings and Evals
In RAG (Retrieval-Augmented Generation), you first find the relevant pieces of your data, then put them in the prompt and have Claude produce the answer. This makes the answer fresh, aware of private data and grounded.
28.1 RAG pipeline
- 1.Documents: PDF, wiki, tickets
- 2.Chunking: small pieces (with overlap)
- 3.Embeddings: text → vector (Voyage AI etc.)
- 4.Vector DB: pgvector, Pinecone, Azure AI Search
- 5.User query: the question
- 6.Retrieve: vector + BM25 (hybrid)
- 7.Rerank: top-K best chunks
- 8.Claude: chunks + query → answer + citations
The top row runs once (indexing), the bottom row on every query
28.2 Key points
| Topic | Detail |
|---|---|
| Embeddings | Anthropic doesn't offer its own embedding model; it recommends Voyage AI (any embedding model will work) |
| Contextual Retrieval | An Anthropic technique: prepend 1-2 lines of context from Claude to each chunk ("this chunk is from the revenue section of the Q2 report"), then embed + BM25. Retrieval failures drop a lot, and even more with reranking |
| Hybrid search | Both semantic (vector) + keyword (BM25); BM25 is essential for exact IDs/error codes |
| Long context vs RAG | If the knowledge base is small (~a few hundred thousand tokens), put all of it in the prompt + caching. If it's large, use RAG |
| Agentic search | Give Claude a search tool and let it write queries and iterate on its own (this is how Claude Code does grep/glob) |
28.3 Evals: measure, don't guess
An eval is a test suite that tells you whether a prompt/model change improved or hurt quality. Every serious LLM app needs evals.
| Grading method | When | Example |
|---|---|---|
| Code-based (exact / regex) | The answer is fixed | Classification label match |
| LLM-as-judge | Open-ended text | Give Claude a rubric and get a 1-5 score |
| Human grading | Most accurate, expensive | Final quality check |
| Execution-based | Code / agents | Do the tests for the generated code pass? |
- Define success criteria first: specific, measurable (accuracy, latency, cost, tone).
- Take test cases from real user data and add edge cases and adversarial cases.
- Keep a train / validation / test split so the prompt doesn't overfit.
- In the Console you can run test cases from the Evaluate tab and compare prompt versions.