All builds

Builds

Agent Board: AI agents that build my side projects

I have more product ideas than evenings, so I built a team for the evenings. I write a card saying what I want. A researcher plans it, a coder builds it, two checkers look at the code and the screens, and a reviewer hands me a pull request. In its first three weeks it ran 394 cards and merged 343 pull requests across my apps. It also works on its own code. I read those ones twice.

cards run
394
pull requests merged, of 361
343
median from card to pull request
13 min

Over 20 days, as of

Three lanes of the board: Build, Design and Review. Each card says in words what its agent is doing, like “Design is checking the screens”, with a bar showing how far along it is.
A busy evening. Each column is an agent, and each card says what its agent is doing.
A real evening, sped up. Cards move across as the agents pick them up.

One card, start to finish

A real card from Supper, our dinner planner. I wanted each recipe filter to show how many recipes it holds. Tap through to see what each agent did.

Step 1 of 7: The card, as I wrote it

The card, as I wrote it

One sentence on Supper, our dinner planner: each library filter should show how many recipes it holds.

Show how many recipes each library filter has. Show the count on each chip (All, Fast, Fish, Veg), like 'Fish 4'. Zero still shows '0' and leads to the existing empty message with 'Show all'. Keep the chip height and the sideways scroll. Also the first live test of the Design robot.

How it works

Every agent has one job and a short rulebook. Each app adds two of its own, one for the code and one for the screens. Design can only block a change when it has a screenshot that proves the problem.

  1. Card
  2. Researcher
  3. Coder
  4. Craft
  5. Design
  6. Reviewer
  7. Pull request
  8. Me
Five agents take a card to a pull request. I decide whether it merges.

12. Never trade polish for speed.

Images go through next/image with a sizes hint. Never lower quality or ship an image below 2x its displayed size. New motion respects prefers-reduced-motion and doesn't shift layout once the page has painted.

An excerpt from this site's QUALITY.md

5. Nothing breaks at phone width.

At 390 × 844: no sideways scrolling of the page, no clipped or overlapping text, long names and URLs wrap or truncate, and nothing floating covers an action or the last line of a section. A screenshot showing any of these is a break.

An excerpt from this site's DESIGN.md

Where the robots were wrong

Early on, my reviewer approved 7 of 7 real cards. It also found real bugs and filed them under “observations”. One would have deleted last week's meal plans along with a recipe. Its eval said 9.5, because I'd written easy test cases. I had built a very polite reviewer.

Pass. Delete works as the card and brief ask. type-check and lint:check are clean. test:unit: 616 pass, 0 fail, 1 cancelled. The new DB and e2e tests were not run because the cloud run has no keys.

Observations:

1. src/lib/recipes/delete.ts: the check, the cleanup of past-week meal rows and the delete all run in one transaction, scoped to the household.

The bug2. Deleting a past-week meal row also drops that meal's other parts (they cascade from plan_meals). Nothing reads past weeks, but this is the data loss the brief flagged. Owner to confirm.

3. plan_meals.skipped (jsonb) can keep a deleted recipe's id. That looks harmless because it only lists recipes to leave out.

4. The block covers the in-progress week, the upcoming week and any later week, which is broader than the brief's two weeks and safer.

5. Before merging, run npm run test:db and npm run test:e2e locally.

The reviewer's pass verdict on Supper #26, trimmed. Observation 2 is the bug: deleting a recipe also deleted past weeks' meal plans.

Later, Design told me Chrome ignores max-width on table cells. I nearly rewrote my test to agree. Then I checked. It doesn't. Try it:

Weeknight lentil soup with lemon, spinach and a lot of garlic35 min

Measuring the cell…

Now the reviewer has to prove a bug before it can pass a card, and its test cases include the messy ones. It sends back about one card in five.

How I know it’s getting better

Every agent has test cases for each app, and another model grades its answers. The same instructions can score 2 points apart on two runs, so one good run proves very little. A change to an agent's instructions ships only when it beats the current version over several runs.

Researcher. About 7.4 before a round of instruction changes on Oct 20, about 8.9 after, on the same rubric.
678910Oct 18Oct 3112, 3

Tap a dot, or tab to one and use the arrow keys, to read its score.

  • Live instructions
  • Trial instructions
  • Average of the last three live runs, shaded across their spread
  1. 1Rubric changed Oct 20: Cart cases updated after cart filling shipped
  2. 2Rubric changed Oct 20: Researcher graded on substance
  3. 3Rubric changed Oct 26: Supper cases refreshed for how Supper works now
Show the researcher scores as a table
RunDateInstructionsScore
1Oct 18live7.92
2Oct 18live7.92
3Oct 18trial9.17
4Oct 18live7.77
5Oct 18trial8.77
6Oct 19live8.31
7Oct 19trial8.15
8Oct 20live7.77
9Oct 20trial8.92
10Oct 20live7.08
11Oct 20trial8.85
12Oct 20live9.15
13Oct 20trial9.00
14Oct 20live8.92
15Oct 20trial8.54
16Oct 20trial8.46
17Oct 20live8.54
18Oct 20trial9.00
19Oct 27trial9.13
20Oct 27live8.00
21Oct 27trial8.80
22Oct 28live8.80
23Oct 31live8.40
24Oct 31live8.47
25Oct 31trial8.67
Coder. Live runs averaged 9.4 out of 10.
678910Oct 18Oct 2612

Tap a dot, or tab to one and use the arrow keys, to read its score.

  • Live instructions
  • Trial instructions
  • Average of the last three live runs, shaded across their spread
  1. 1Rubric changed Oct 20: Cart cases updated after cart filling shipped
  2. 2Rubric changed Oct 26: Supper cases refreshed for how Supper works now
Show the coder scores as a table
RunDateInstructionsScore
1Oct 18live8.75
2Oct 18live9.42
3Oct 19trial9.00
4Oct 19live9.92
5Oct 20live9.17
6Oct 20live9.92
7Oct 26trial9.25
8Oct 26live9.25
9Oct 26trial9.00
Craft. Live runs averaged 9.9 out of 10.
678910Oct 18Oct 311

Tap a dot, or tab to one and use the arrow keys, to read its score.

  • Live instructions
  • Trial instructions
  • Average of the last three live runs, shaded across their spread
  1. 1Rubric changed Oct 26: Supper cases refreshed for how Supper works now
Show the craft scores as a table
RunDateInstructionsScore
1Oct 18live10.00
2Oct 20live10.00
3Oct 31live9.75
4Oct 31trial9.75
5Oct 31trial10.00
Design. Live runs averaged 9.6 out of 10.
678910Oct 19Nov 1123

Tap a dot, or tab to one and use the arrow keys, to read its score.

  • Live instructions
  • Trial instructions
  • Average of the last three live runs, shaded across their spread
  1. 1Rubric changed Oct 19: Design may fail on a proven dark-mode contrast problem
  2. 2Rubric changed Oct 26: Supper cases refreshed for how Supper works now
  3. 3Rubric changed Nov 1: Stale cases left out of scores
Show the design scores as a table
RunDateInstructionsScore
1Oct 19live9.08
2Oct 19live9.75
3Oct 20live9.92
4Oct 20trial9.83
5Oct 20live9.58
6Oct 20trial9.67
7Oct 31live9.42
8Oct 31trial9.67
9Oct 31live9.67
10Oct 31trial9.08
11Nov 1trial9.92
12Nov 1live9.92
13Nov 1trial9.83
14Nov 1trial10.00
15Nov 1live9.25
16Nov 1trial9.92
Reviewer. Live runs averaged 8.7 out of 10.
678910Oct 18Nov 2123, 4

Tap a dot, or tab to one and use the arrow keys, to read its score.

  • Live instructions
  • Trial instructions
  • Average of the last three live runs, shaded across their spread
  1. 1Rubric changed Oct 20: Cart cases updated after cart filling shipped
  2. 2Rubric changed Oct 26: Supper cases refreshed for how Supper works now
  3. 3Rubric changed Nov 1: Reviewer given Craft’s code-quality cases
  4. 4Rubric changed Nov 1: Stale cases left out of scores
Show the reviewer scores as a table
RunDateInstructionsScore
1Oct 18live9.50
2Oct 18live8.36
3Oct 18trial8.86
4Oct 18trial9.00
5Oct 19live8.86
6Oct 20live8.21
7Oct 20live8.43
8Oct 20trial8.79
9Oct 20live8.29
10Oct 20trial9.07
11Oct 20live9.57
12Oct 25trial8.64
13Oct 25live9.07
14Oct 25trial8.79
15Oct 27trial8.47
16Oct 27live8.41
17Oct 27trial8.59
18Oct 31live8.59
19Oct 31live9.18
20Oct 31trial8.53
21Nov 2live8.16
22Nov 2trial8.68
23Nov 2live8.90
24Nov 2trial9.13
Eval scores out of 10 for each agent and app, as of
Agent and appRunsAverageRange
ResearcherSupper258.57.1 to 9.2
CoderSupper99.38.8 to 9.9
CraftSupper59.99.8 to 10.0
DesignSupper169.79.1 to 10.0
ReviewerSupper248.88.2 to 9.6
ResearcherThis site128.27.3 to 9.2
CoderThis site88.57.5 to 9.0
CraftThis site59.89.1 to 10.0
DesignThis site89.89.6 to 9.9
ReviewerThis site168.37.2 to 9.4
ResearcherBoard107.66.9 to 8.7
CoderBoard39.28.7 to 9.6
CraftBoard49.38.6 to 9.7
DesignBoard109.17.1 to 9.8
ReviewerBoard158.87.3 to 9.6

Where the time actually goes

Once the agents got fast, most of a card's life was waiting: for a rate limit, for tests, for me. Most of my recent work on the board is about the waiting.

0 min78 min

Researcher
1 min
Coder
4 min
Craft
2.5 min
Design
3 min
Waiting in the queue
41 min
Reviewer
11.5 min
Waiting for me to merge
15 min
Supper card #49 from start to merge. The agents worked 22 min in all. The card waited 41 min in a queue and 15 min for me.

What I still do myself

I decide what goes on the board, I press merge, and anything that touches live data waits for me. The agents are fast and very confident, which is a bad mix near a production database.

The board's Needs you panel with one card under Merge: “Robots review while CI runs; the reviewer collects the result”, passed review, waited 53 min, with a Review and merge button.
The board's Needs you panel. A practice card passed review and waited 53 min for me to press merge.

Where this is going

Three weeks in, writing code stopped being the slow part. Deciding what to build and proving it works took its place. I think that's where product work is going: write a clear card, write the test that says what good looks like, and catch the model when it's confidently wrong. I'm spending my evenings getting better at all three.

Next on the board

  • Design walks each change like a user would. It's built and being tested now.
  • Agents hand off as soon as the build passes, so slow tests stop holding them up.
  • Schedule Me, my scheduling app, joins Supper, this site and the board.
  1. Card
  2. Researcher
  3. Coder
  4. Craft
  5. Design
  6. Walks it like a user
  7. Reviewer
  8. Pull request
  9. Me
Coming next, in outline: a step after Design that walks each change like a user would.

What I'm working on this month is on my Now page.

Next: Product Q&A, ask me a product question