test(benchmarks): add concept-search tasks
Three tasks whose target code shares no vocabulary with the prompt, run in a repository with generated distractor modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
a8f6f01f2f
commit
00a44f50b0
19 files changed
+106
No files matched your search
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"prompt": "Customers whose free trial has ended can still download their monthly statements. Block that by raising PermissionError, and keep it working for paying customers and for customers whose trial is still running.",
|
||||
"generate": {
|
||||
"filler_modules": 250
|
||||
},
|
||||
"workflow": {
|
||||
"kind": "concept search in a large repository"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user