feat(benchmarks): add setup versus baseline A/B runner

Run tasks with and without the installed setup and compare pass rate, rework,
duration, tokens, cost, tool calls, and workflow checks; exclude runs that end
with a client or API error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Julian lechnerandClaude Opus 5.5 committed 2026-10-01 12:45:41 +02:00
1 parent 3480072ed7
commit 89bee27dd2
3 files changed
+278

No files matched your search

+4
View File
@@ -0,0 +1,4 @@
$ErrorActionPreference = 'Stop'
$root = Split-Path (Split-Path $PSScriptRoot -Parent) -Parent
& python (Join-Path $root 'scripts/benchmarks/ab.py') @args
exit $LASTEXITCODE