fix(retrieval): run search as local CLI

README.md

@@ -217,7 +217,7 @@ other local settings are not generally merged. Dry runs still rebuild `generated
 Selecting Codex also installs shared skills to `.agents/skills`. An interactive extension list includes plugins and
 Codex skills; unchecked extensions are skipped on installation. Normal runs also ask whether to update
 the selected client CLIs, enable Flashbang, and enable Qwen3 semantic retrieval. These choices are saved for Quickupdate.
-Disabling retrieval on a later update removes its setup-owned runtime, index, and MCP registrations. Dry runs preview
+Disabling retrieval on a later update removes its setup-owned runtime and index. Dry runs preview
 changes without updating CLIs, extensions, or retrieval. See [Plugins](docs/plugins.md) for client-specific behavior.
 
 ## Update

docs/plugins.md

@@ -14,7 +14,7 @@ Codex receives Caveman through its native plugin adapter. Humanizer, Impeccable,
 
 [QMD](https://github.com/tobi/qmd) requires Node.js 22 or newer and `npm`. Selecting it installs `@tobilu/qmd` globally. Claude receives QMD's official `qmd@qmd` plugin; Codex registers `qmd mcp` as its MCP server. Deselecting QMD removes the client integration owned by this setup, but leaves the global `qmd` executable installed.
 
-Normal installation and update also ask whether to update the selected client CLIs, enable Flashbang, and enable the optional Qwen3-Embedding-0.6B semantic retrieval layer. Retrieval is shared by Codex and Claude through a local MCP server, uses a per-user virtual environment and index, and is disabled by default. Quickupdate reuses the saved shell, client, CLI update, Flashbang, retrieval, and extension choices without opening menus. Disabling retrieval removes its setup-owned runtime, index, and MCP registrations. Only successful runs save selections; dry runs do not update CLIs, extensions, or retrieval.
+Normal installation and update also ask whether to update the selected client CLIs, enable Flashbang, and enable the optional Qwen3-Embedding-0.6B semantic retrieval layer. Retrieval uses a per-user virtual environment and index and is invoked through the installed `server.py search` CLI, disabled by default. Quickupdate reuses the saved shell, client, CLI update, Flashbang, retrieval, and extension choices without opening menus. Disabling retrieval removes its setup-owned runtime and index. Only successful runs save selections; dry runs do not update CLIs, extensions, or retrieval.
 
 Plugin installation requires the corresponding client CLI, network access, and any authentication required by the plugin. Context7 may open an OAuth flow. Codex or plugin hooks may require a trust review in `/hooks`. The scripts never install plugins during a build or doctor run and never install them when `--dry-run` or `-DryRun` is used.
 

scripts/lib/semantic-retrieval.py

@@ -8,30 +8,14 @@ import subprocess
 import sys
 
 
-NAME = 'my-ai-qwen3-retrieval'
 FILES = ('server.py', 'requirements.txt')
 CUDA_INDEX = 'https://download.pytorch.org/whl/cu126'
-TOOL_TIMEOUT_SECONDS = 1800
 
 
 def digest(path):
     return hashlib.sha256(path.read_bytes()).hexdigest()
 
 
-def command(client, arguments, home, dry_run):
-    if dry_run:
-        print('DRYRUN ' + ' '.join(arguments))
-        return True
-    executable = shutil.which(client)
-    if executable is None:
-        raise ValueError(f'{client} CLI is required for semantic retrieval.')
-    result = subprocess.run([executable, *arguments[1:]], stdin=subprocess.DEVNULL, capture_output=True, text=True,
-                            timeout=180, env=dict(os.environ, HOME=str(home), USERPROFILE=str(home), CODEX_HOME=str(home / '.codex'), CLAUDE_CONFIG_DIR=str(home / '.claude')))
-    if result.returncode:
-        raise ValueError(f'Semantic retrieval command failed: {client} {" ".join(arguments[1:])}\n{result.stdout}{result.stderr}')
-    return True
-
-
 def python_path(target):
     return target / ('Scripts/python.exe' if os.name == 'nt' else 'bin/python')
 
@@ -57,26 +41,6 @@ def ensure_runtime(target, dry_run):
     return python
 
 
-def set_codex_tool_timeout(path, seconds):
-    lines = path.read_text(encoding='utf-8').splitlines(keepends=True)
-    section = f'[mcp_servers.{NAME}]'
-    start = next((number for number, line in enumerate(lines) if line.strip() == section), None)
-    if start is None:
-        raise ValueError('Codex semantic retrieval configuration is missing.')
-    end = next((number for number in range(start + 1, len(lines)) if lines[number].lstrip().startswith('[')), len(lines))
-    for number in range(start + 1, end):
-        if lines[number].lstrip().startswith('tool_timeout_sec'):
-            newline = '\r\n' if lines[number].endswith('\r\n') else '\n'
-            lines[number] = f'tool_timeout_sec = {seconds}{newline}'
-            break
-    else:
-        while end > start + 1 and not lines[end - 1].strip():
-            end -= 1
-        newline = '\r\n' if lines and lines[0].endswith('\r\n') else '\n'
-        lines.insert(end, f'tool_timeout_sec = {seconds}{newline}')
-    path.write_text(''.join(lines), encoding='utf-8')
-
-
 def load_state(path):
     if not path.exists():
         return {'version': 1, 'clients': [], 'files': {}}
@@ -110,8 +74,6 @@ def sync(root, home, clients, enabled, dry_run, update, summary=False):
             if not summary:
                 print('DRYRUN disable semantic retrieval: ' + (', '.join(owned) if owned else 'nothing owned'))
             return
-        for client in owned:
-            command(client, [client, 'mcp', 'remove', NAME] + (['--scope', 'user'] if client == 'claude' else []), home, False)
         state['clients'] = [client for client in state['clients'] if client not in clients]
         if state['clients']:
             save_state(state_path, state, False)
@@ -135,23 +97,9 @@ def sync(root, home, clients, enabled, dry_run, update, summary=False):
         shutil.copy2(source / name, target / name)
         state['files'][name] = digest(target / name)
     python = ensure_runtime(target, False)
-    existing_clients = set(state['clients'])
     state['clients'] = list(dict.fromkeys(state['clients']))
     save_state(state_path, state, False)
-    arguments = ['--', str(python), str(target / 'server.py'), '--root', '.', '--data-dir', str(target / 'data'), '--model-cache', str(target / 'model-cache')]
     for client in clients:
-        command_args = [client, 'mcp', 'add']
-        if client == 'claude':
-            command_args += [NAME, '--scope', 'user', '-e', f'HF_HOME={target / "model-cache"}']
-        else:
-            command_args += ['--env', f'HF_HOME={target / "model-cache"}']
-            command_args += [NAME]
-        command_args += arguments
-        if update and client in existing_clients:
-            command(client, [client, 'mcp', 'remove', NAME] + (['--scope', 'user'] if client == 'claude' else []), home, False)
-        command(client, command_args, home, False)
-        if client == 'codex':
-            set_codex_tool_timeout(home / '.codex' / 'config.toml', TOOL_TIMEOUT_SECONDS)
         if client not in state['clients']:
             state['clients'].append(client)
             save_state(state_path, state, False)

scripts/test-retrieval-server.py

@@ -27,6 +27,15 @@ def embedding():
 
 
 class RetrievalServerTests(unittest.TestCase):
+    def test_runtime_python_uses_adjacent_virtualenv(self):
+        with TemporaryDirectory() as temporary:
+            script = Path(temporary) / 'semantic-retrieval' / 'server.py'
+            runtime = script.parent / '.venv' / 'Scripts' / 'python.exe'
+            runtime.parent.mkdir(parents=True)
+            runtime.touch()
+
+            self.assertEqual(MODULE.runtime_python(script, 'C:/Python/python.exe', 'nt'), runtime.resolve())
+
     def test_preferred_device_uses_cuda_with_cpu_fallback(self):
         with patch.dict('sys.modules', torch=SimpleNamespace(cuda=SimpleNamespace(is_available=lambda: True))):
             self.assertEqual(MODULE.preferred_device(), 'cuda')
@@ -89,7 +98,7 @@ class RetrievalServerTests(unittest.TestCase):
                     encoded.clear()
                     with ThreadPoolExecutor(max_workers=1) as executor:
                         self.assertEqual(executor.submit(indexes[0].search, 'query', 5).result()[0]['text'], 'alpha')
-                    self.assertEqual(encoded, ['query'])
+                    self.assertEqual(encoded, [])
                     timestamp = file.stat()
                     file.write_text('bravo', encoding='utf-8')
                     os.utime(file, ns=(timestamp.st_atime_ns, timestamp.st_mtime_ns))

scripts/test-semantic-retrieval.py

@@ -16,15 +16,13 @@ SPEC.loader.exec_module(MODULE)
 
 
 class SemanticRetrievalTests(unittest.TestCase):
-    def test_codex_registration_sets_tool_timeout(self):
+    def test_sync_does_not_register_mcp(self):
         with tempfile.TemporaryDirectory() as directory:
-            config = Path(directory) / 'config.toml'
-            config.write_text('[mcp_servers.my-ai-qwen3-retrieval]\ncommand = "python"\n\n[mcp_servers.my-ai-qwen3-retrieval.env]\nHF_HOME = "cache"\n', encoding='utf-8')
-
-            MODULE.set_codex_tool_timeout(config, 1800)
-
-            self.assertEqual(config.read_text(encoding='utf-8'),
-                             '[mcp_servers.my-ai-qwen3-retrieval]\ncommand = "python"\ntool_timeout_sec = 1800\n\n[mcp_servers.my-ai-qwen3-retrieval.env]\nHF_HOME = "cache"\n')
+            home = Path(directory) / 'home'
+            output = io.StringIO()
+            with redirect_stdout(output):
+                MODULE.sync(ROOT, home, ['codex', 'claude'], True, True, False)
+            self.assertNotIn('mcp', output.getvalue().lower())
 
     def test_runtime_prefers_cuda_packages_for_nvidia_gpu(self):
         with tempfile.TemporaryDirectory() as directory:
@@ -60,24 +58,7 @@ class SemanticRetrievalTests(unittest.TestCase):
                 self.assertEqual(output.getvalue(), '')
                 self.assertFalse(home.exists())
 
-    def test_claude_registration_places_name_before_environment(self):
-        with tempfile.TemporaryDirectory() as directory:
-            home = Path(directory) / 'home'
-            calls = []
-            original = MODULE.command
-            original_runtime = MODULE.ensure_runtime
-            MODULE.command = lambda client, arguments, home, dry_run: calls.append((client, arguments))
-            MODULE.ensure_runtime = lambda target, dry_run: target / '.venv' / 'bin' / 'python'
-            try:
-                MODULE.sync(ROOT, home, ['claude'], True, False, False)
-            finally:
-                MODULE.command = original
-                MODULE.ensure_runtime = original_runtime
-            add = calls[0][1]
-            self.assertEqual(add[:5], ['claude', 'mcp', 'add', MODULE.NAME, '--scope'])
-            self.assertIn('-e', add)
-
-    def test_disable_removes_owned_runtime_and_registration(self):
+    def test_disable_removes_owned_runtime_without_cli_registration(self):
         with tempfile.TemporaryDirectory() as directory:
             home = Path(directory) / 'home'
             target = home / '.my-ai-configuration/semantic-retrieval'
@@ -87,14 +68,7 @@ class SemanticRetrievalTests(unittest.TestCase):
             state_path = target.parent / 'semantic-retrieval.json'
             state_path.write_text(json.dumps({'version': 1, 'clients': ['codex'], 'files': {
                 name: hashlib.sha256((target / name).read_bytes()).hexdigest() for name in MODULE.FILES}}))
-            calls = []
-            original = MODULE.command
-            MODULE.command = lambda client, arguments, home, dry_run: calls.append((client, arguments))
-            try:
             MODULE.sync(ROOT, home, ['codex'], False, False, False)
-            finally:
-                MODULE.command = original
-            self.assertEqual(calls[0][1], ['codex', 'mcp', 'remove', MODULE.NAME])
             self.assertFalse(target.exists())
             self.assertFalse(state_path.exists())
 

shared/retrieval/requirements.txt

@@ -1,2 +1 @@
-mcp>=1.9,<2
 sentence-transformers>=5.4,<6

shared/retrieval/server.py

@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-"""Local MCP semantic search backed by Qwen3 embedding and reranking."""
+"""Local CLI semantic search backed by Qwen3 embedding and reranking."""
 
 from __future__ import annotations
 
@@ -8,10 +8,12 @@ import ast
 import hashlib
 import heapq
 import json
+import multiprocessing
 import os
 from pathlib import Path
 import re
 import sqlite3
+import sys
 import threading
 
 
@@ -24,12 +26,27 @@ DENSE_CANDIDATES = 50
 LEXICAL_CANDIDATES = 50
 RERANK_CANDIDATES = 50
 FINAL_RESULTS = 5
+CLI_TIMEOUT_SECONDS = 300
 RRF_K = 60
 INDEX_VERSION = 2
 RETRIEVAL_INSTRUCTION = "Given a codebase question, retrieve relevant code and documentation that answer the question."
 TEXT_EXTENSIONS = {".c", ".cpp", ".cs", ".go", ".java", ".js", ".json", ".md", ".py", ".ps1", ".rs", ".sh", ".toml", ".ts", ".tsx", ".txt", ".yaml", ".yml"}
 
 
+def runtime_python(script=__file__, executable=sys.executable, system=os.name):
+    relative = Path('.venv') / ('Scripts/python.exe' if system == 'nt' else 'bin/python')
+    candidate = (Path(script).resolve().parent / relative)
+    if candidate.is_file() and candidate != Path(executable).resolve():
+        return candidate
+    return None
+
+
+def use_runtime_python():
+    runtime = runtime_python()
+    if runtime is not None:
+        os.execv(str(runtime), [str(runtime), str(Path(__file__).resolve()), *sys.argv[1:]])
+
+
 def sections(text: str, path: str):
     lines = text.splitlines(keepends=True)
     boundaries = {0: "<module>", len(lines): "<module>"}
@@ -118,6 +135,14 @@ def document_text(path: str, symbol: str, content: str):
     return f"Path: {path}\nSymbol: {symbol}\n\n{content}"
 
 
+def run_action(action: str, root: Path, data_dir: Path, model_cache: Path, query: str | None, top_k: int, output):
+    model_cache.mkdir(parents=True, exist_ok=True)
+    os.environ.setdefault("HF_HOME", str(model_cache.resolve()))
+    index = Index(root, data_dir)
+    result = index.search(query, top_k) if action == "search" else {"chunks": index.rebuild()}
+    output.put((True, result))
+
+
 def preferred_device():
     import torch
     return "cuda" if torch.cuda.is_available() else "cpu"
@@ -144,6 +169,7 @@ class Index:
         self.db.execute("create index if not exists chunks_path on chunks (path)")
         self.db.execute("create virtual table if not exists lexical using fts5(document)")
         self.db.execute("create table if not exists files (path text primary key, digest text)")
+        self.db.execute("create table if not exists query_cache (cache_key text primary key, result text not null)")
         self.db.commit()
         self.lock = threading.RLock()
         self.model = None
@@ -234,6 +260,13 @@ class Index:
             for relative in previous:
                 self.delete_path(relative)
                 self.db.execute("delete from files where path = ?", (relative,))
+            if changed or previous:
+                self.db.execute("delete from query_cache")
+
+    def cache_key(self, query: str, top_k: int):
+        files = self.db.execute("select path, digest from files order by path").fetchall()
+        state = json.dumps(files, separators=(",", ":"))
+        return hashlib.sha256(f"{top_k}\0{query}\0{state}".encode("utf-8")).hexdigest()
 
     def search(self, query: str, top_k: int = FINAL_RESULTS):
         if not query.strip():
@@ -242,6 +275,14 @@ class Index:
             self.refresh()
             if not self.db.execute("select 1 from chunks limit 1").fetchone():
                 return []
+            key = self.cache_key(query, top_k)
+            cached = self.db.execute("select result from query_cache where cache_key = ?", (key,)).fetchone()
+            if cached:
+                try:
+                    return json.loads(cached[0])
+                except (TypeError, json.JSONDecodeError):
+                    with self.db:
+                        self.db.execute("delete from query_cache where cache_key = ?", (key,))
             import numpy as np
             vector = self.encoder().encode([query], prompt_name="query", normalize_embeddings=True)[0]
             if len(vector) != EMBEDDING_DIMENSIONS:
@@ -260,34 +301,42 @@ class Index:
                     for identifier, path, symbol, text in self.db.execute(
                         f"select rowid, path, symbol, text from chunks where rowid in ({placeholders})", identifiers)}
             candidates = [rows[identifier] for identifier in identifiers]
-            return self.rerank(query, candidates, max(1, min(top_k, FINAL_RESULTS)))
+            result = self.rerank(query, candidates, max(1, min(top_k, FINAL_RESULTS)))
+            with self.db:
+                self.db.execute("insert or replace into query_cache values (?, ?)",
+                                (key, json.dumps(result, ensure_ascii=False)))
+            return result
 
 
 def main():
     parser = argparse.ArgumentParser()
+    parser.add_argument("action", choices=("search", "rebuild"))
+    parser.add_argument("--query")
+    parser.add_argument("--top-k", type=int, default=FINAL_RESULTS)
     parser.add_argument("--root", default=os.getcwd(), type=Path)
     parser.add_argument("--data-dir", required=True, type=Path)
     parser.add_argument("--model-cache", required=True, type=Path)
     args = parser.parse_args()
-    args.model_cache.mkdir(parents=True, exist_ok=True)
-    os.environ.setdefault("HF_HOME", str(args.model_cache.resolve()))
-    from mcp.server.fastmcp import FastMCP
-
-    index = Index(args.root, args.data_dir)
-    mcp = FastMCP("my-ai-qwen3-retrieval")
-
-    @mcp.tool()
-    def semantic_search(query: str, top_k: int = FINAL_RESULTS) -> list[dict]:
-        """Search repository concepts, behavior, and implementation patterns first with Qwen3 embedding and reranking. Verify returned passages against current files; use direct reads for known paths."""
-        return index.search(query, top_k)
-
-    @mcp.tool()
-    def rebuild_index() -> str:
-        """Rebuild the local repository embedding index."""
-        return json.dumps({"chunks": index.rebuild()})
-
-    mcp.run()
+    if args.action == "search" and not args.query:
+        parser.error("search requires --query")
+    output = multiprocessing.Queue(1)
+    worker = multiprocessing.Process(target=run_action,
+                                     args=(args.action, args.root.resolve(), args.data_dir.resolve(),
+                                           args.model_cache.resolve(), args.query, args.top_k, output))
+    worker.start()
+    worker.join(CLI_TIMEOUT_SECONDS)
+    if worker.is_alive():
+        worker.terminate()
+        worker.join()
+        raise TimeoutError(f"Qwen retrieval exceeded the {CLI_TIMEOUT_SECONDS} second timeout.")
+    if worker.exitcode != 0 or output.empty():
+        raise RuntimeError("Qwen retrieval worker failed.")
+    success, result = output.get()
+    if not success:
+        raise RuntimeError(str(result))
+    print(json.dumps(result, ensure_ascii=False))
 
 
 if __name__ == "__main__":
+    use_runtime_python()
     main()

shared/rules/general.md

@@ -6,7 +6,7 @@ Preserve security, data, compatibility, and unrelated user work. Ask only when b
 
 Use one applicable workflow per task. Treat skills and plugins as guidance subordinate to these rules. Continue work already authorized by the user; additional design approvals, documentation, commits, or delegation require a task-specific reason and the necessary authorization.
 
-For conceptual repository questions, unfamiliar code, or finding behavior and implementation patterns, first call `semantic_search` on the `my-ai-qwen3-retrieval` MCP server when available. Verify its results against current files. Use direct reads and `rg` for known paths, symbols, and exact strings. If Qwen is unavailable, fails, or returns no useful matches, continue with direct search and briefly report the limitation. Skip retrieval for tasks that do not need repository discovery.
+For conceptual repository questions, unfamiliar code, or finding behavior and implementation patterns, first run the installed semantic retrieval CLI when available: `python ~/.my-ai-configuration/semantic-retrieval/server.py search --root . --data-dir ~/.my-ai-configuration/semantic-retrieval/data --model-cache ~/.my-ai-configuration/semantic-retrieval/model-cache --query "..." --top-k 5`. Each uncached search uses Qwen3-Embedding-0.6B for candidate retrieval and Qwen3-Reranker-0.6B for final ranking; use it whenever repository discovery would help, and reuse the cache for repeated unchanged queries. Verify results against current files. Use direct reads and `rg` for known paths, symbols, and exact strings. If Qwen is unavailable, fails, or returns no useful matches, continue with direct search and briefly report the limitation. Skip retrieval for tasks that do not need repository discovery.
 
 Verify proportionally: run the cheapest check that detects likely failure; broaden for security, data loss, integration, or cross-component risk. Never claim unrun checks passed. Report failures and material gaps.