test(benchmarks): add concept-search tasks

Three tasks whose target code shares no vocabulary with the prompt, run in a
repository with generated distractor modules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Julian lechnerandClaude Opus 5.5 committed 2026-10-01 13:27:56 +02:00
1 parent a8f6f01f2f
commit 00a44f50b0
19 files changed
+106

No files matched your search

@@ -0,0 +1,9 @@
{
"prompt": "Customers whose free trial has ended can still download their monthly statements. Block that by raising PermissionError, and keep it working for paying customers and for customers whose trial is still running.",
"generate": {
"filler_modules": 250
},
"workflow": {
"kind": "concept search in a large repository"
}
}