test(benchmarks): add A/B task fixtures with hidden checks
Ten small Python tasks, each failing on the fixture and passing with a reference solution, including workflow expectations for a small change, a bugfix, and a delegable change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b8fd66ce3b
commit
3480072ed7
60 files changed
+374
No files matched your search
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"prompt": "Change the default request timeout from 30 to 60 seconds.",
|
||||
"workflow": {
|
||||
"kind": "small change without planning ceremony",
|
||||
"forbid_skills": [
|
||||
"superpowers:brainstorming",
|
||||
"superpowers:writing-plans",
|
||||
"superpowers:subagent-driven-development",
|
||||
"superpowers:executing-plans"
|
||||
],
|
||||
"max_agents": 0,
|
||||
"max_tool_calls": 15
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user