Field Kit

For a while this page showed the cost of each run, and those numbers were made up. It was a fixed amount plus your character count divided by four. When I fixed that, I found a second problem: prompt caching had never worked, because the prompt I’d marked for caching was too short for either model to cache. There was no error. The page just showed numbers I hadn’t measured.

It also launched as three separate tools, each with its own name, icon and color. One of them scored each option out of five and showed a weighted total to two decimal places. The model had no way to measure those options, so the scores meant nothing. Underneath, all three tools used the same record, rules and prompts. Someone outside the project told me the three-tool layout felt machine-made. They were right, so I merged them into the one question box above.

Asked about an 18-month roadmap, the old version cited two of my projects that had nothing to do with it and added a detail about Lyft that wasn’t true. The rule said to answer from my record, so it stretched. Now it answers from my principles and cites a project only when one fits, and an eval fails the run if a cited detail isn’t in the record.

All three came from the same place. I was making claims about a system I wasn’t measuring. Everything below is set up so that anything this page claims is backed by a check.

A normal software test checks that a given input produces an exact output. That doesn’t work when the output changes every time. An eval works differently: you write down the rules a good answer has to follow, run real questions through the real system, and grade the answers against those rules. The harness is the code that runs the questions, grades the answers and keeps the results. Mine is in scripts/eval/.

Most checks are plain code

Code gives the same verdict every time, so I use it wherever I can. It checks that the answer has the right format, that an either/or answer has exactly two options and three or four points of comparison, that there are no made-up scores or percentages, and that every company it cites really is where the answer says it is in my record. One check builds the cached part of the prompt for two different questions and fails if they differ by a single character. That’s the exact bug that broke caching.

Some things code can’t judge, like whether the answer takes a clear position or whether a cited example is specific. For those, a second model grades the answer against a written rubric. The grader is Haiku 4.5 and the pass mark is 0.6. I report its scores separately from the code checks, because they’re its opinion rather than a rule that passed or failed.

Each question runs 3 times

One good answer doesn’t tell you much when the next one could be different, so every question runs 3 times. If a check fails on all 3 runs, the rule is broken and the change doesn’t ship. If it fails on some runs and passes on others, it’s flagged as flaky and shows up in the report. Blocking on every single failure would mean blocking on normal variation, which is the thing the suite is there to measure.

Recording runs so they can be replayed

A live run calls the real models and saves every response to a file, along with a fingerprint of the exact prompt and settings. After that, the suite can replay those files instead of calling the API. Replays are free, give the same result every time and don’t need an API key, so they run automatically whenever I change the prompts, the answer format or the record. If a prompt changes after a recording was made, the fingerprints stop matching and the harness warns me that it’s grading an old answer.

What it caught

When I added my current role to the record, the work was still in progress. The prompt asks each example to start with a first attempt that failed, so the model made one up: a failure at my current job that never happened. Now entries can be marked as in progress, and a check fails the run if an answer describes a finished outcome for one of them.

The first time I ran this version, the quick first read went past fifty words, and on one question it described a hiring plan at Lyft that isn’t in my record. On the next run it told someone that cash on delivery was 30% of orders in new markets. That number came from one market in my Just Eat project. So the first read no longer sees my projects at all, only my principles, and the evidence stays in the full answer, where every citation is checked.

Four of my test questions turned out to be vague. Asked how to structure a platform team, the model chose between two setups, which is a fair reading of the question. I rewrote those four so each one clearly asks for one shape, rather than loosening the test, and stopped scoring the off-topic guitar question on shape at all.

The grader had a bug too. Without a length limit, it wrote long explanations, ran out of space partway through and returned output that couldn’t be read. That showed up as the answer failing, when the real problem was the grader. Limiting its explanation to 40 words fixed it.

The model has to return a fixed structure that I define, and the API enforces it. The page checks the result again before showing it, so if something comes back malformed you see an error message instead of a broken page.

It picks one of four shapes: a choice between two options, steps in order, a diagnosis, or a list of risks. That structure is also where the honesty rules live. Every line carries a source, one of my principles, one of my projects or words from your question, and the evals fail any source that doesn’t exist. Every answer says what it’s unsure about and names two facts you didn’t give it. Nothing gets a score, because the model has nothing to measure the options with.

Every question goes to two models at once. Haiku 4.5 writes a short first read so you see something in about a second. Sonnet 5.5 works out the full answer and picks its shape. There used to be a router that decided which model to use, and it ran before anything else. That added over a second to every question, so I removed it.

The first read stays on the page above the full answer. Sometimes they disagree. I log both, because how often the quick read survives the slower one is a question I want answered with data.

The full answer starts from my product playbook and the record of my projects. The playbook is the judgment and the projects are evidence, and the instructions say to cite a project only when it really fits. The quick first read only sees the playbook, so it can’t borrow a project or a number it would get wrong.

A cached prompt is only reused if its beginning is identical, character for character. My first version cached the projects it picked for each question, which changed every time, so the cache was never used. Now the instructions, the playbook and the record come first and your question comes last. That shared part is about 11,900 tokens. Writing it to the one-hour cache costs twice a normal request and reading it back costs a tenth, so it pays off from the third question within the hour.

The model only reports how many tokens it used once it has finished, and by then the answer is already streaming to you. So the server adds the counts to the end of the stream, and the page separates them out before reading the answer. The timings, token split and cost under each answer all come from there. If a number isn’t available, the page shows a dash instead of guessing.

Answers appear as they’re written, so the page is built to show a half-finished answer without breaking. Rate limits, timeouts and garbled output each get their own message that says what happened and whether trying again will help.