Benchmark
Which email API agents pick when the prompt names none. 100 open prompts a night, 20 per agent, in the same sandboxes as the lab.
Pick shareRun 412
18%Postbox's rank: 3 of 4
- Relaymail41%
- Sendwell27%
- Postboxyou18%
- Letterkit14%
Relaymail, Sendwell and Letterkit are fictional vendors, like Postbox.
By night
- Postbox
- Relaymail
Why agents skipped PostboxFrom the agents' own reasoning in the lab
- Quickstart calls a deprecated
send()signature34% of runs 403on first send with the default key scope29% of runs- Docs served as HTML only, so agents ran out of contextFixed in #19818% of runs
The fix that would move you most
Return a 403 that names the missing scope and links to the fix. Lab rerun: +9 pts pick share.
Open #214By agentPicks out of 20 open prompts
| Agent | Postbox | Most picked | Split |
|---|---|---|---|
| claude-code | 5 / 20 | Relaymail | |
| codex | 3 / 20 | Relaymail | |
| cursor | 2 / 20 | Relaymail | |
| gemini-cli | 4 / 20 | Relaymail | |
| chatgpt-agent | 4 / 20 | Relaymail |