AGNotes & writing
Blog post

3 min read

The happy path is the least interesting thing to test

Real AI providers make it hard to test rate limits, malformed tool calls, and broken streams. So I built Mock AI API, a local stand-in that fails on purpose.

Most AI integrations I've seen have one well-tested path: send a prompt, get a nice answer, render it. That path matters, but it's also the one least likely to break in production.

What actually breaks is everything around it. The provider returns a 429 in the middle of a busy hour. The model calls a tool with arguments that aren't valid JSON. A stream starts, sends half a response, then fails. Your retry logic, your error messages, and your UI's half-rendered state all get exercised at once, usually for the first time, in front of a user.

The problem is that you can't easily make a real provider do any of that on demand. And even the happy-path tests are awkward against a real API: they're slow, they cost money, they need a key in CI, and the model doesn't say the same thing twice.

So I built Mock AI API.

A fake provider that speaks the real protocol

It's a local server that speaks OpenAI-compatible APIs on port 8011 and Anthropic-compatible APIs on port 8012. You point the official SDK at it, and your code doesn't know the difference.

pipx install mock-ai-api
mock-ai-api

The responses are deterministic. Same request, same answer, every time. There are no keys, no accounts, and nothing leaves your machine.

Picking a scenario with the model name

The design decision I like most is how you choose what happens. You don't configure the server or call a setup endpoint. You add a suffix to the model name:

model="gpt-4.1:mock-text"           # a normal reply
model="gpt-4.1:mock-tool-call"      # one function tool call
model="gpt-4.1:mock-bad-tool-json"  # a tool call with broken JSON arguments
model="gpt-4.1:mock-429"            # a rate-limit error

The model name is already something every code path passes through, so you can trigger a specific failure from a single test without touching the rest of your setup. It also makes tests easy to read. The scenario is right there in the request.

Testing a rate limit in four lines

Here's a test that checks your code actually surfaces a rate limit, against the real OpenAI SDK:

from openai import OpenAI, RateLimitError
import pytest

client = OpenAI(
    api_key="mock-key",
    base_url="http://127.0.0.1:8011/v1",
    max_retries=0,
)

def test_surfaces_rate_limit():
    with pytest.raises(RateLimitError):
        client.chat.completions.create(
            model="gpt-4.1:mock-429",
            messages=[{"role": "user", "content": "Hello"}],
        )

The same pattern covers mock-400, mock-401, mock-500 and mock-503. Each returns an error shaped like the real provider's, so the SDK raises the same exception type it would in production.

The failures that are hard to fake by hand

Status codes are the easy part. You could mock those with any HTTP library. The scenarios I actually wanted were the ones that are painful to hand-write:

  • mock-bad-tool-json returns a tool call whose arguments are cut off mid-string: {"path":"tmp/mock-file.txt","contents":"unterminated". If your agent does json.loads(arguments) without a try, you'll find out now instead of later.
  • mock-tool-stream streams tool-call deltas the way the real API does, so you can test the code that stitches partial arguments back together.
  • mock-stream-error (Anthropic) starts a stream, then emits an error event after it has begun. That's the case where your UI has already rendered half a message and has to decide what to do with it.
  • mock-delay adds a fixed delay to streaming, for testing loading states and timeouts.

Keeping it honest

A mock is only useful if it behaves like the thing it replaces. If it drifts, your tests pass and production still breaks.

So the test suite has two layers. The API tests call each app directly and check the response shapes. Then an SDK compatibility suite starts real local servers and drives them with specific versions of the official openai and anthropic Python packages. If an SDK can't parse what the mock returns, those tests fail.

Every scenario also lives in one registry. The model catalog, the docs table, and the behavior all come from the same place, so adding a scenario is a single entry plus the code that implements it, not four files to keep in sync.

What it isn't

It's not an inference server, and it's not trying to imitate a model's intelligence. Responses are fixed, state lives in memory, there's no auth, and the API surface is intentionally incomplete. It covers what you need to test integration code and stops there.

It's MIT licensed and on PyPI. The source is at github.com/armanghev/mock-ai-api. If there's a failure mode you keep hitting in production that it can't simulate yet, open an issue. That's exactly the kind of scenario it should have.