Claude Seekho

Big Picture 3: The Anthropic API Gateway

Before a request reaches the model, it passes through the Anthropic API gateway. Five things happen here in a row, and some of them can stop the request.

Gateway checks in order
  1. 1.Auth: API key / OAuth / Workload Identity, tied to a workspace
  2. 2.Rate limits: token bucket on RPM, ITPM, OTPM; over the limit gives 429 + retry-after
  3. 3.Prompt cache lookup: prefix match, hit is cheaper and faster
  4. 4.Safety checks: classifiers and usage policy
  5. 5.Routing: model, region (inference_geo), batch / fast mode, then into the model

What each check does

  • Auth: checks who you are using an API key, OAuth or Workload Identity. That identity maps to a workspace, which is where limits and billing attach.
  • Rate limits: a token bucket counts requests per minute (RPM), input tokens per minute (ITPM) and output tokens per minute (OTPM). The limit depends on your tier. Go over and you get a 429, with retry-after telling you how long to wait.
  • Prompt cache lookup: the gateway checks whether the prefix of your request is already cached. A hit makes the request cheaper and faster, which is why Big Picture 2 keeps the stable part on top.
  • Safety checks: the request is screened by safety classifiers and against the usage policy. A request can be stopped here.
  • Routing: the request goes to the right model in the right region (the region can be chosen with inference_geo). Batch or fast mode is decided here too. After that it enters the model.