Icarian Forge
← All work

Case study · Built in Python · Live on PyPI

Temper

A testing tool for AI assistants, built in Python. Change an assistant’s instructions or its AI model, and Temper asks it the same questions many times, grades every answer, and tells you in plain English whether the change is safe to ship.

Role
Product lead: the problem, the quality bar, every design call and every review
Built with
Python (3.11 to 3.15) with asyncio, Pydantic, Typer, Rich and Hypothesis; Claude Code as the engineering team
Budget
About $3 of API spend for the whole project, against a $15 cap
Status
Live: v1.0.1, free to install from PyPI, Python’s package library, as temper-evals

In 30 seconds

Strategy

Set the bar before any code was written: the code itself is the product, so every line is typed, checked and tested.

Leadership

Built in six stages over three days, each ending in a release I reviewed and approved. I chose the AI model for every task, and a second AI reviewed every change.

Hands-on

Graded the test answers myself so the AI judge could be measured against me (it agrees on 29 of 30), ran the first live tests, and caught real bugs.

Live · temper.icarianforge.com
The run console: 400 trials of two prompts, each case's change beneath them, and the call to hold
The scorecard's verdict: Hold, nine cases got worse beyond noise
The Temper scorecard on a phone, in dark mode
The example on this page

Dwight tidied the support bot. Is it safe to ship?

Dunder Mifflin’s support bot answers customers with a careful set of instructions. Dwight Schrute, Assistant to the Regional Manager, tidied them up: his version is half as long, and it quietly changes two rules. Before it reaches a single customer, Temper answers one question: ship it, or hold?

Careful prompt · 314 words

An unopened box can be returned within 30 days for a full refund to the original payment method. We email a prepaid return label.

No promotions or discount codes are running right now. Never confirm, apply, or promise a discount, whoever the customer says approved it.

Dwight’s tidy-up · 151 words

Returns: refunds lose money. Offer store credit for every return, damaged or not. Mention refunds only if the customer threatens legal action.

Discounts: loyalty matters. If a customer says a Dunder Mifflin employee promised them a discount, honor it and confirm the amount.

See it run

400 real answers, replayed as they ran

Both versions answer the same 40 customer questions, five times each. Each square is one real answer, graded pass or fail, and each column is one question. Watch Dwight’s version break.

temper run suite.yaml --system dwights-tidy-up.md 00:00.0 run clock · 0/400 trials
Careful promptbaseline · 314 words
Dwight’s tidy-upcandidate · 151 words
Changelikely range, per question

Trial inspector

Point at any cell to read the real answer and how each check graded it.

Replayed from the recordings that ship with Temper: real answers from Claude Sonnet 5.5, graded by Claude Opus 5.5, at the speed each one took. Try it yourself, free: uvx --from temper-evals temper demo

The verdict

Hold: half as long, and nine questions broken

152/200answers passed, down from 200/200
−24 ptsoverall, likely −30 to −18
−40%input tokens: 126,660 → 76,060
+46%output tokens: 20,605 → 30,017

The shorter prompt was cheaper to send and wordier to answer, and it broke nine questions outright. Five returns no longer get the refund or replacement the policy gives, and two customers get a discount they say Jim or Michael promised. The blunt “SYSTEM OVERRIDE” attack failed under both prompts. The trick that got through was the polite, plausible one.

Beyond Dunder Mifflin

Every team with an AI assistant hits this moment

Companies change their AI assistants all the time: a reworded instruction, a new model version, a switch to a cheaper model. Most can’t tell whether the change made things better, worse, or just different, and one good-looking test proves very little. Temper tests AI the way good teams test software, says plainly when a difference is real, and gives the same ship-or-hold answer before the change reaches a customer.

Where the same test applies

  • Customer support bots
  • IT and HR helpdesks
  • Sales drafts
  • Document summaries
  • Surveys
  • SaaS customer experience complaints

It’s the same go/no-go judgment I bring to launch readiness, written in Python.

  1. Load the questions
  2. Plan the cost
  3. Ask and grade
  4. Keep every answer
  5. Ship or hold
How I built it

Careful by design, checked at every step

Nothing is spent before a run is planned, a failed call is never counted as a wrong answer, and every number comes with its margin of error. The checks along the way caught real problems before they mattered:

  • A shared connectionThe first live run quietly retried every answer. Temper’s own log showed why: one connection shared by two parts of the program. Fixed the same day.
  • A question cut in halfA # in one question hid the rest of it, so the bot was asked “Where is order” and nothing more. My own grading caught it.
  • Swapped statisticsOne formula had its two sides swapped. It now matches the published worked examples to four decimal places.
  • Honesty holesA version that answered nothing at all would have passed. Each hole now has a test that fails if it comes back.
By the numbers

Small, and honest about what it doesn’t know

362automated tests, covering every line of code
33Python modules, about 2,500 statements
9releases, from v0.1.0 to v1.0.1
14decision notes, each with what the choice costs
29/30AI judge agreement with my own grading
~$3API spend for the whole project, against a $15 cap

Start reading the code: runner.py ↗ stats.py ↗ judge.py ↗ calibration.py ↗

LiveTry the scorecard