From 6d4ac56901a76a973af49d518a91d3ae65f89dc1 Mon Sep 17 00:00:00 2001 From: Julian lechner Date: Thu, 1 Oct 2026 13:27:59 +0200 Subject: [PATCH] docs(scripts): document the semantic search comparison Co-Authored-By: Claude Opus 5.5 --- docs/scripts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/scripts.md b/docs/scripts.md index 4a9a995..4f23b47 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -23,7 +23,7 @@ Run scripts from the repository root. PowerShell and Bash commands have equivale | `commands/doctor.ps1`, `commands/doctor.sh` | Check generated output and installed configuration. | | `commands/prompt-budget.ps1`, `commands/prompt-budget.sh` | Track permanent context, skill metadata, the rule catalog, and agent metadata; warn above 10,000 permanent-context tokens, fail when any token estimate exceeds its baseline by 10%, or permanent context exceeds 15,000 tokens. | | `commands/context-report.ps1`, `commands/context-report.sh` | Report the context installed Claude and Codex actually load, from Claude `/context`, `/skill-doctor`, and its init event, and Codex `debug prompt-input`: categories, active plugins, listed skills, installed skills missing from the listing, and skills listed without description context. `--check` fails on missing or description-less skills. No model request is made. | -| `commands/benchmark.ps1`, `commands/benchmark.sh` | Run the ten tasks under `scripts/benchmarks/tasks/` with and without the installed setup and compare pass rate, rework turns, duration, tokens, tool calls, cost, and workflow checks (no planning ceremony for a small change, targeted verification for a bugfix, bounded delegation for a larger change). Starts real model sessions; use `--dry-run` to list runs. The Codex baseline needs one `codex login` with `CODEX_HOME` set to `--codex-baseline-home`. | +| `commands/benchmark.ps1`, `commands/benchmark.sh` | Run the tasks under `scripts/benchmarks/tasks/` with and without the installed setup; `--arms setup setup-no-search --tasks 11-* 12-* 13-* --warm-index` compares Claude with and without semantic search on concept-search tasks in a large generated repository, and compare pass rate, rework turns, duration, tokens, tool calls, cost, and workflow checks (no planning ceremony for a small change, targeted verification for a bugfix, bounded delegation for a larger change). Starts real model sessions; use `--dry-run` to list runs. The Codex baseline needs one `codex login` with `CODEX_HOME` set to `--codex-baseline-home`. | | `commands/repository-context.ps1`, `commands/repository-context.sh` | Refresh the local repository-context cache. | | `commands/session-state.ps1`, `commands/session-state.sh` | Read or update the local agent session state. | | `commands/telemetry.ps1`, `commands/telemetry.sh` | Append a local telemetry event. |