From c31af49b70f3b045e72d1c16ecee2635d7cecdc1 Mon Sep 17 00:00:00 2001 From: Julian lechner Date: Thu, 8 Oct 2026 12:15:15 +0200 Subject: [PATCH] docs(qmd): make semantic search mandatory and document QMD local search Prompt budget stays within +10% of baseline (instruction 1485 -> 1331, skill metadata 3163 -> 3017, rule catalog claude 28818 -> 30517); baseline files unchanged. Co-Authored-By: Claude Opus 5.5 --- README.md | 19 ++++---- docs/hooks.md | 1 + docs/plugins.md | 7 +-- docs/scripts.md | 12 ++--- docs/skills.md | 2 +- shared/rules/general.md | 3 +- .../skills/research/semantic-search/SKILL.md | 48 +++++-------------- 7 files changed, 34 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index cf4ee2d..5be148d 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ Bash: ## Quickstart With the [requirements](#requirements) installed, clone the repository, build it, then install it. -Stop if any command fails. Installation asks for the shell, client, Flashbang, status line, Qwen retrieval, and extension choices. +Stop if any command fails. Installation asks for the shell, client, Flashbang, status line, local QMD search models, and extension choices. PowerShell: @@ -96,11 +96,10 @@ Install and update also rebuild internally; the explicit build above lets you ch - Separate install, update, and uninstall scripts: install refuses to run against an already-installed destination, update refuses to run against one that is not installed yet, and uninstall refuses to run against one that was never installed. -- User-level extension installation for Ponytail, i-have-adhd, Superpowers, Context7, Caveman, QMD, MCPorter, +- User-level extension installation for Ponytail, i-have-adhd, Superpowers, Context7, Caveman, MCPorter, Humanizer, Impeccable, and Anthropic Frontend Design, with an interactive extension toggle during both install and update. Only extensions installed and recorded by this setup can be removed through deselection. -- Optional local semantic retrieval for both clients using Qwen3-Embedding-0.6B embeddings and Qwen3-Reranker-0.6B - reranking, disabled by default and removable through the install/update choice. +- Optional local search for both clients through QMD with Qwen3-Embedding-0.6B, Qwen3-Reranker-0.6B, and QMD Query Expansion 1.7B (Yes/No/Auto; Auto benchmarks GPU, then CPU). ## Requirements @@ -108,8 +107,8 @@ Install and update also rebuild internally; the explicit build above lets you ch - Bash 4.3 or newer, with standard utilities such as `awk`, `sed`, and `sha256sum`. - Python 3.11 or newer for configuration validation and task-state hooks/helpers (`python` for PowerShell scripts, `python3` for Bash scripts). - `jq` for Claude's Bash status line; without it, that status line produces no output. -- Python `venv` and `pip` when the optional semantic retrieval layer is enabled. -- Node.js 22 or newer and `npm` when QMD or MCPorter is selected. +- Node.js 22 or newer and `npm` when local QMD search is enabled. +- Node.js 22 or newer and `npm` when MCPorter is selected. Core Bash setup scripts do not require PowerShell. The credential commands use Windows Credential Manager and require PowerShell, including when invoked through their Bash wrappers. The PowerShell scripts remain compatible with @@ -232,11 +231,11 @@ Dry runs still rebuild `generated/`, but do not modify installation destinations Selecting Codex also installs shared skills to `.agents/skills`. An interactive extension list includes plugins and Codex skills; unchecked extensions are skipped on installation. Normal install and update ask whether to enable -Flashbang, apply the custom status line, and enable Qwen3 semantic retrieval. Choices are saved for Quickupdate. +Flashbang, apply the custom status line, and enable local QMD search. Choices are saved for Quickupdate. The shared skill catalog includes development skills and focused writing, image, site structure, psychology, and conversion skills. All retained skills are included in both generated clients. The status line defaults to enabled to preserve existing behavior. Disabling it removes the custom client setting and, -for Claude, the setup-owned status line script. Disabling retrieval on update removes its setup-owned runtime and index. +for Claude, the setup-owned status line script. Disabling local QMD search on update removes only its setup-owned package, models, and repository collections. Dry runs preview changes without updating extensions or retrieval. See [Plugins](docs/plugins.md) for client-specific behavior. ## Update Script @@ -263,7 +262,7 @@ setup; pre-existing or manually installed extensions are retained. Quickupdate r ## Uninstall Script Remove everything this setup previously installed for the selected client(s): managed files tracked in each -destination's manifest, extensions and skills recorded in the extension ledger, an enabled semantic retrieval setup, +destination's manifest, extensions and skills recorded in the extension ledger, an enabled local QMD search setup, and the saved Quickupdate selection (once no destination remains installed for any client). PowerShell: @@ -289,7 +288,7 @@ touched. Codex skill backups are kept under `.agents/.ai-config-skill-backups/`. deleted once it is fully empty; backups already on disk are always kept. Extensions and skills recorded in the ledger are removed the same way `update` removes a deselected extension: pre-existing or manually installed ones outside the ledger are left alone, and a skill file modified since it was -installed is kept with a warning. An enabled semantic retrieval setup is disabled and its setup-owned runtime and +installed is kept with a warning. An enabled local QMD search setup is disabled and its setup-owned runtime and index removed for the selected client(s). The saved Quickupdate selection (`~/.my-ai-configuration/selection.json`) is only removed once uninstalling leaves no client installed at all, so uninstalling just one client out of a `Both` installation keeps the saved choices for the client that remains. diff --git a/docs/hooks.md b/docs/hooks.md index c2323ae..8ee53b9 100644 --- a/docs/hooks.md +++ b/docs/hooks.md @@ -9,6 +9,7 @@ The adapters register these events: | Codex and Claude | `Stop` | Run the finish notification (`flashbang`). | | Claude | `PreCompact` | Record the compaction timestamp under the installed configuration root and point to existing workspace task state. | | Claude | `SessionStart` | Print an absolute pointer if the workspace's `.ai-session/state.json` exists. | +| Codex and Claude | SessionStart | Warm the QMD index and search daemon in the background (only while local QMD search is enabled). | The main agent alone maintains canonical task state. Hooks never write or summarize it, print its contents, or infer completed verification. They use the hook event's `cwd`, falling back to the process working directory when no workspace is supplied. Malformed event data does not fall back to another project. An absent state file is a successful no-op. Legacy state under the installed configuration root is not automatically adopted; confirm its task before migrating it. The scripts do not grant permissions, bypass approvals, or replace client permission systems. diff --git a/docs/plugins.md b/docs/plugins.md index b510bbf..f5d0a7d 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -2,7 +2,7 @@ The user-level plugin manifest is `adapters/plugins.tsv`. It maps each logical extension to the client-specific plugin, skill, or MCP installation method. -The current baseline includes Ponytail, i-have-adhd, Superpowers, Context7, Caveman, QMD, MCPorter, Humanizer, Impeccable, and Anthropic Frontend Design. Both scripts operate at user scope and show an extension selector unless running a dry run. Selections are collected before configuration changes. Both rebuild generated packages first. +The current baseline includes Ponytail, i-have-adhd, Superpowers, Context7, Caveman, MCPorter, Humanizer, Impeccable, and Anthropic Frontend Design. Both scripts operate at user scope and show an extension selector unless running a dry run. Selections are collected before configuration changes. Both rebuild generated packages first. Selecting Context7 with MCPorter registers the server in MCPorter. Selecting Context7 without MCPorter installs its native client plugin; that explicit selection permits the native MCP route under the global instructions. @@ -16,11 +16,8 @@ Codex receives Caveman through its native plugin adapter. Humanizer, Impeccable, On Windows, installation and update repair the known Caveman 1.0.0 hook launcher in setup-owned Codex installations so it resolves the plugin path under PowerShell as well as Command Prompt. Custom commands and foreign installations are preserved. Review the changed hook definitions in Codex `/hooks`; setup does not grant hook trust automatically. -[QMD](https://github.com/tobi/qmd) requires Node.js 22 or newer and `npm`. Selecting it installs `@tobilu/qmd` globally. Claude receives QMD's official `qmd@qmd` plugin; Codex can use the `qmd` CLI. Deselecting QMD removes setup-owned integrations and uninstalls the global package when the Codex-owned QMD record is removed. -Normal installation and update ask whether to enable Flashbang, apply the custom status line, and enable the optional Qwen3-Embedding-0.6B semantic retrieval layer. Retrieval uses a per-user virtual environment and index and is invoked through the installed `server.py search` CLI, disabled by default. While it is enabled, both clients receive the `semantic-search` skill, which the general rules point to for concept questions when its cost check favors it; disabling retrieval removes the skill. Codex may write only the retrieval `data/` directory from its sandbox (`sandbox_workspace_write.writable_roots`); the sandbox forbids the daemon's pipes, so each Codex search loads the models in-process and takes about half a minute. Quickupdate reuses the saved shell, client, status line, Flashbang, retrieval, and extension choices without opening menus. Disabling the status line removes its custom setting and setup-owned Claude script; disabling retrieval removes its setup-owned runtime and index. Only successful runs save selections; dry runs do not update extensions or retrieval. Install and update do not modify the Codex or Claude CLI installations. - -The `server.py search`/`rebuild` CLI transparently reuses a background daemon (spawned automatically on first use per `--data-dir` and code version, so an updated `server.py` never reuses an older daemon) that keeps the embedding and reranking models loaded in memory, so only the first query after a cold start or an idle period pays the model-loading cost; every call falls back to loading the models directly in-process if the daemon cannot be reached or started. The daemon exits on its own after 30 minutes of inactivity; `server.py stop` ends it at once. `server.py status` reports indexable, indexed, and stale files and whether the models are loaded, without loading models or starting the daemon, so agents can judge what the next search costs. Search requires a Git repository and indexes tracked files plus untracked files that Git does not ignore. Indexing covers common source, markup, style, SQL, XAML, project, and configuration file types, uses bfloat16 on GPUs that support it, and skips files larger than 512 KB and, per repository, stops after 2000 indexable files by default; set `SEMANTIC_RETRIEVAL_MAX_FILES` to raise that cap for larger monorepos. When the cap is reached, the CLI prints an explicit warning to stderr rather than silently omitting files. +Normal installation and update ask whether to enable Flashbang, apply the custom status line, and enable optional local QMD search (Yes/No/Auto, disabled by default). Local search runs [QMD](https://github.com/tobi/qmd) (`@tobilu/qmd@2.8.3`, Node.js 22 or newer, `npm`) with Qwen3-Embedding-0.6B, Qwen3-Reranker-0.6B, and QMD Query Expansion 1.7B. Auto benchmarks GPU first, then CPU, and enables search only when median search latency is at most 5 s and embedding runs at least 20 chunks/s. While it is enabled, both clients receive the `semantic-search` skill, which the general rules require for questions whose identifiers are unknown; disabling local search removes the skill and its instruction line. The skill calls `qmd-search.mjs`, which reuses a background daemon on loopback 127.0.0.1 protected by a per-start token; the daemon keeps the models loaded and exits after 30 idle minutes (`QMD_IDLE_MINUTES`). A SessionStart hook warms the index and daemon in the background. The setup saves and restores a user's own QMD `models:` block, and removal touches only the setup-owned package, models, and repository collections; foreign installs and collections survive. Updating migrates from the earlier Python stack by removing `~/.my-ai-configuration/semantic-retrieval/` and old QMD extension ledger records. Quickupdate reuses the saved shell, client, status line, Flashbang, local search, and extension choices without opening menus. Disabling the status line removes its custom setting and setup-owned Claude script. Only successful runs save selections; dry runs do not update extensions or local search. Install and update do not modify the Codex or Claude CLI installations. Plugin installation requires the corresponding client CLI, network access, and any authentication required by the plugin. Context7 may open an OAuth flow. Codex or plugin hooks may require a trust review in `/hooks`. The scripts never install plugins during a build or doctor run and never install them when `--dry-run` or `-DryRun` is used. diff --git a/docs/scripts.md b/docs/scripts.md index 4b478e5..dc6c403 100644 --- a/docs/scripts.md +++ b/docs/scripts.md @@ -40,7 +40,7 @@ To undo an added instruction, pass the same text to `remove-instruction` with `- Credential commands currently require PowerShell and Windows Credential Manager, including their Bash wrappers. Use `~/.my-ai-configuration/bin/with-credential.ps1` or `.sh` to run a tool with a stored key available only in that child process. The key must be a valid environment-variable name; its value is never written to configuration. Use `remove-credentials` with `-Client`/`--client` and `-Key`/`--key` to delete only that credential. The shared `with-credential` helpers remain installed for other keys. -The A/B benchmark measures whole sessions. Qwen Auto selection uses `shared/retrieval/benchmark.py`; run it through `scripts/lib/semantic-retrieval.py benchmark --root . --home `. This warm-model check uses synthetic, repeated short documents. Its result does not measure model startup, first indexing, real code chunks, CLI latency, or memory use; do not use it as an end-to-end performance estimate. +The A/B benchmark measures whole sessions. Auto selection of local QMD search uses `shared/qmd/qmd-benchmark.mjs`, run through `scripts/lib/qmd.py`. It measures median search latency and embedding throughput on GPU, then CPU, and is not an end-to-end performance estimate. ## Checks and tests @@ -63,8 +63,8 @@ Gitea Actions runs the repository checks from `.gitea/workflows/ci.yml`. | `tests/test-quick-update.py` | Validate saved Quickupdate choices. | | `tests/test-remove-customizations.py` | Validate targeted instruction and credential removal without accessing the real credential store. | | `tests/test-quick-update.ps1`, `tests/test-quick-update.sh` | Run the Quickupdate test through the relevant shell. | -| `tests/test-retrieval-server.py` | Validate retrieval indexing, ranking, and optional real-model execution. | -| `tests/test-semantic-retrieval.py` | Validate retrieval installation and removal. | +| `tests/test-qmd-*.mjs` | Validate the QMD library, search, daemon, warm-up, and benchmark scripts. | +| `tests/test-qmd-setup.py` | Validate QMD installation, migration, and removal. | | `tests/test-session-state.py` | Validate session-state helpers. | | `tests/test-shell-packages.ps1`, `tests/test-shell-packages.sh` | Validate generated package contents and client settings. | | `tests/test-runtime-status.ps1`, `tests/test-runtime-status.sh` | Validate build summaries and status-line output. | @@ -88,6 +88,6 @@ Gitea Actions runs the repository checks from `.gitea/workflows/ci.yml`. | `lib/remove-instruction.py` | Remove an exact client-specific instruction block. | | `lib/selection-state.ps1`, `lib/selection-state.sh` | Persist choices used by Quickupdate. | | `lib/selection-state.py` | Read and write Quickupdate state. | -| `lib/semantic-retrieval.ps1`, `lib/semantic-retrieval.sh` | Invoke semantic retrieval from the installation flow. | -| `lib/semantic-retrieval.py` | Install, index, benchmark, and remove the local retrieval service. | -| `shared/retrieval/benchmark.py` | Check warm Qwen embedding and reranking speed for Auto selection; not an end-to-end benchmark. | +| `lib/qmd.sh`, `lib/qmd.ps1` | Invoke local QMD search from the installation flow. | +| `lib/qmd.py` | Install, migrate, benchmark, and remove local QMD search. | +| `shared/qmd/*.mjs` | Installed QMD scripts: search CLI, daemon, warm-up, benchmark, and shared library. | diff --git a/docs/skills.md b/docs/skills.md index 85743c1..a91d7cd 100644 --- a/docs/skills.md +++ b/docs/skills.md @@ -5,7 +5,7 @@ Skills are reusable workflows. Rules are persistent constraints and agents are s The current structure is organized by purpose: - `planning/`: impact analysis and work-item definition -- `research/`: technical and repository research, plus `semantic-search`, which is installed only while semantic retrieval is enabled +- `research/`: technical and repository research, plus `semantic-search`, which is installed only while local QMD search is enabled - `reviews/`: security, performance, WPF, UI/UX, business, API, and EF Core reviews - `implementation/`: SQL, unit-test, and integration-test writing - `verification/`: facts and work-item verification diff --git a/shared/rules/general.md b/shared/rules/general.md index a701248..573a117 100644 --- a/shared/rules/general.md +++ b/shared/rules/general.md @@ -6,7 +6,8 @@ Preserve security, data, compatibility, and unrelated user work. Ask only when b Use one applicable workflow per task. Treat skills and plugins as guidance subordinate to these rules. Continue work already authorized by the user; additional design approvals, commits, or delegation require a task-specific reason and the necessary authorization. -Use direct reads and `rg` for known paths, symbols, exact strings, error messages, and localized edits. For concept questions where the identifiers are unknown or the flow spans components, consider the `semantic-search` skill when it is installed: check its cost (repository size, index state, loaded models) against the expected `rg` effort as the skill describes, verify its hits against the current files, and fall back to `rg` with a brief note if it fails. +Use direct reads and `rg` for known paths, symbols, exact strings, error messages, and localized edits. +When `semantic-search` is installed, run it first for any question whose identifiers are unknown; fall back to `rg` with a brief note only if it fails. Verify proportionally: run the cheapest check that detects likely failure; broaden for security, data loss, integration, or cross-component risk. Never claim unrun checks passed. Report failures and material gaps. diff --git a/shared/skills/research/semantic-search/SKILL.md b/shared/skills/research/semantic-search/SKILL.md index b2a9f4c..268fb81 100644 --- a/shared/skills/research/semantic-search/SKILL.md +++ b/shared/skills/research/semantic-search/SKILL.md @@ -1,53 +1,31 @@ --- name: semantic-search -description: Use when you must find where a behavior, concept, or rule lives in a repository and do not know its identifiers or exact strings, or when a first rg search returned nothing useful. Returns ranked code and documentation chunks from the local Qwen index. +description: MUST use before reading or grepping code whenever the exact symbol, file, or string is not already known - concept questions, unfamiliar areas, "where/how is X done", and bug reports that describe behavior. Use rg only for known identifiers, paths, or exact strings. --- # Semantic Search -Search the current Git repository by meaning with the locally installed Qwen3 embedding and reranking models. +Search the current Git repository, plus the user's registered QMD collections, by meaning. QMD runs locally with Qwen3-Embedding-0.6B, Qwen3-Reranker-0.6B, and QMD Query Expansion 1.7B; a background daemon keeps them loaded. -## When to use +## Rule -- Concept questions: "where are expired trials blocked", "how is the cache invalidated", "which code retries failed uploads". -- Unfamiliar repositories where names and conventions are unknown. -- After an `rg` search for guessed terms returned nothing relevant or too many unrelated files. - -Use `rg` or direct reads instead for known file paths, symbol names, exact strings, and error messages. - -## Decide by cost - -Check the cost of the next search first: run the command under Run with `status` in place of `search` and without `--query` and `--top-k`. It loads no models, takes about a second, and prints `{indexable_files, indexed_files, stale_files, models_loaded, in_process}`. Weigh that against the `rg` rounds you expect: - -- Fewer than about 200 indexable files, or one or two guessed terms likely to hit: use `rg`; a search costs more than it saves. -- `stale_files` in the thousands, including a first index (`indexed_files` 0): the search embeds them first, which can take minutes. Pay that only for a large unfamiliar repository or several concept questions in this session. -- `models_loaded` false: the search loads the models first, which takes tens of seconds. With `in_process` true, as inside the Codex sandbox, every search pays that again; search only when `rg` would need several rounds, and batch related questions into one precise query. -- Index fresh and `models_loaded` true: a search takes seconds; use it for any concept question. +Run this search first whenever you do not already know the exact identifier, path, or string. Do not start with `rg` guesses for a concept. `rg` comes after the search, for callers and tests of verified hits. ## Run -PowerShell: +The same command works in Bash and PowerShell: -```powershell -python "$HOME/.my-ai-configuration/semantic-retrieval/server.py" search --root . --data-dir "$HOME/.my-ai-configuration/semantic-retrieval/data" --model-cache "$HOME/.my-ai-configuration/semantic-retrieval/model-cache" --query "" --top-k 5 -``` + node "$HOME/.my-ai-configuration/qmd/qmd-search.mjs" --query "" --top-k 5 -Bash: - -```bash -python3 "$HOME/.my-ai-configuration/semantic-retrieval/server.py" search --root . --data-dir "$HOME/.my-ai-configuration/semantic-retrieval/data" --model-cache "$HOME/.my-ai-configuration/semantic-retrieval/model-cache" --query "" --top-k 5 -``` - -The output is a JSON list of `{path, symbol, text, score}`, best match first. +Output: JSON list of `{path, line, score, snippet}`, best first. Repository paths are relative to the repository root; hits from other collections are `qmd:///`. ## Use the results -1. Phrase the query as a full question about behavior, not as a keyword list. -2. Open the top hits and verify them against the current files before acting; the index can lag behind uncommitted edits by one refresh. -3. Follow the verified hit with `rg` for its symbols to find callers and tests. +1. Phrase the query as a full question about behavior, not a keyword list. +2. Open the top hits and verify them against the current files before acting. +3. Follow verified hits with `rg` for their symbols to find callers and tests. +4. If no hit is relevant, run one rephrased search before switching to `rg`. -## Cost and limits +## Failure -- Searches refresh only changed files. Outside a sandbox a background daemon keeps the models loaded for 30 idle minutes; inside the Codex sandbox there is no daemon. -- Only Git-tracked and unignored files up to 512 KB are indexed, by default at most 2000 files per repository. -- If the command fails or the setup is missing, continue with `rg` and state the limitation briefly. +If the command exits non-zero, continue with `rg` and state the limitation in one sentence.