Evaluation · benchmarks

Empirical results across four benchmark dimensions.

skylakegrep ships multiple reproducible benchmark families. Each measures a different question; quoting the right number for the right claim matters. The repository's parity-benchmarks.html holds the full raw tables, methodology, and per-task data; this page is the curated summary.

0a. Wrong-path quick-answer (NEW in 0.5.7)

The newest UX gate. Measures how fast the parallel proactive umbrella streams a first answer when the user is in an unrelated cwd and the answer lives in a sibling folder. Setup: cd into an empty directory, SKYGREP_PROACTIVE_DIRS=/tmp/oss-bench, query "do I have any package configuration files". The user runs the real skygrep CLI; we measure wall clock to first stream block AND wall clock to script exit.

Concrete output captured in the v0.5.7 release notes (full notes →): the streaming markers 🔍 / ▾ / 🌐 / 🌊 / ⚡ arrive with route + quality labels (filename_extend, ~100 ms-1 s; pure filename glob, no semantic understanding / cross-folder lazy, embed budget 5 seeds, σ-validated cosine) so the user can judge the answer's quality against the route that produced it. Cascade has a 30 s hard timeout; cross-folder lazy has an 8 s hard timeout.

0b. Cold-start lazy auto-trigger (NEW in 0.5.3)

The newest tier — measures whether the --lazy auto-trigger that fires on a never-indexed project actually beats plain ripgrep cold-start on vocabulary-mismatch queries. Run via benchmarks/release-0.5.3-rg-vs-lazy.py — 10 hand-labelled Django queries phrased as natural language ("where is the migration runner that applies pending schema changes to the database"), every query starts from a freshly cleaned SQLite DB, and the script invokes the real skygrep search CLI (not the python API) so the measured numbers are exactly what a user would see. Compares --no-lazy (pure rg cold-start) vs default (auto-trigger).

Specific hits: Q1 → django/urls/resolvers.py; Q3 → migration.py + executor.py; Q4 → backends.py; Q7 → base.py. 0.5.3 release notes call out the misses (Q2, Q5, Q6, Q8, Q9, Q10 still don't hit because qwen 2.5:3b can't reliably pick the right dir for some oracle phrasings — tracked as a 0.6 candidate for a larger router model).

1. End-to-end Claude Code agent

Real Claude Code sub-agents answering hand-labelled code-search questions in two prompted conditions — rg-only (skygrep forbidden) vs skygrep-on (rg/grep/find forbidden) — across 21 questions and 1 multi-turn session in three repos.

The cleanest, most consistent signal is tool-call reduction: −37.6 % single-turn, −82 % multi-turn. Each agent tool call costs an LLM round-trip + network RTT + serialization + context-window growth, so reducing them shortens the agent loop even when total tokens are equal. Token cost across the 20-task aggregate is roughly flat (+6.5 %); we do not claim skygrep saves the LLM bill.

Best-case task in the original aggregate: a vocabulary-mismatch question (NL phrasing didn't match any code identifier) — skygrep finished in 1 tool call vs rg-only's 25 (25× fewer), at one eighth the wall time. Worst-case: a token-friendly question whose vocabulary (auth / session / token) overlapped directly with code-path tokens, so rg's straightforward scan was already efficient — skygrep was 40 % more tool calls. The lexical pre-gate addresses this via a conservative four-condition gate that detects exactly these queries and short-circuits to rg internally in ~50 ms. Methodology and aggregate published in the release notes; see the release notes for the routing fix.

2. Public OSS recall (Django · React · Tokio)

Historical receipt: this section preserves an earlier three-repository run that used moving repository tips, approximate chars / 4 token counts, and a deliberately broad term-OR rg dump. Its 30/30 recall and 60×–770× range are not the current general-performance claim. The current pinned, quality-gated protocol is General Benchmark v2.

This table records 30 hand-labelled questions across three popular open-source codebases under the legacy protocol. The current runner benchmarks/public_oss_bench.py now uses exact public pins and emits a new receipt rather than promising byte-identical reproduction of these historical numbers.

Historical reading: in that run, rg's 100 % was a recall-ceiling baseline — it returned 20 M+ legacy-estimated tokens of term-OR scan output per query, so the answer was in the dump but the agent had to read the whole haystack. Under that historical protocol, skygrep ranked the right file in the top 10 for 30 / 30 cases while emitting 60×–770× less estimated context. React reached 10 / 10 after the Option-C substrate upgrade (bge-m3 embedder + content-agnostic non-canonical-path filter); the original failure modes on react-007 and react-010 and the resolution are documented in parity-benchmarks.html as the engineering record, not erased.

3. Worked example — django-001 (archived calculation)

Historical worked example from one of the 30 tasks aggregated into the public-OSS table above. The values remain visible as the original receipt; rerunning the current pinned protocol produces a new, separately identified measurement.

Query: "Where does Django turn an incoming URL into the view function that should handle it?"
Expected canonical: django/urls/resolvers.py (URLResolver.resolve())
Vocab mismatch: query says "URL into view", code identifier is resolve — the failure mode that grep-as-search collapses on.

Historical side A — rg term-OR scan

The rg-agent extracts up to 8 terms (TF-IDF-ish stopword filter), runs rg -i -F --max-count=20 -C2 per term, concatenates output. For this query the extractor produced:

['function', 'incoming', 'incom', 'django', 'handle', 'should', 'into', 'that']

The high-signal words URL, view, resolve did not survive the extractor — they were either stopword-filtered or pushed past the 8-term cap. That is the vocab-mismatch failure: the query's actual intent is URLResolver.resolve(), but rg searches for function, django, that instead.

Archived chars / 4 estimates derived from measured output volumes per rg invocation:

Under the legacy approximation, django alone is 1.4 million estimated tokens because the term matches in basically every file of the Django source tree. that is 840 K estimated tokens for the same reason — high-frequency words that the stopword filter let through.

Historical side B — skygrep --top 10 --json

$ skygrep "Where does Django turn an incoming URL into the view function that should handle it?" \
    --json --top 10
$ wc -c   # 10,430 chars ≈ 2,607 tokens

Top files returned include django/urls/resolvers.py and related canonical implementation files.

Reduction — visual comparison

The historical 1,395 × estimate is higher than that receipt's 60 × – 770 × range because vocab-mismatch queries are the worst case for rg (stopwords flood the output) and the best case for skygrep (the embedder bridges "URL into view" → resolve()).

Why the historical receipt used a 60 × – 770 × range

rg output scales with (repo LOC) × (term-frequency of high-signal terms). skygrep output is ~constant-per-K (top-10 ≈ 10 KB):

Tokio is the floor (small repo, focused vocabulary). Django and React both blow up because django / react saturate term-OR scans across mid-sized monorepos.

Inspect the legacy calculation (not an exact reproduction)

# rg side — counts bytes from term-OR scan
cd /tmp/oss-bench/django
for term in function incoming incom django handle should into that; do
  rg -i -F --max-count 20 -C 2 "$term" .
done | wc -c

# skygrep side — counts bytes from top-10 JSON
skygrep "Where does Django turn an incoming URL into the view function that should handle it?" \
  --json --top 10 | wc -c

# divide chars by 4 to approximate tokens

The original page did not pin the Django source commit, so a current clone is not expected to reproduce these values within a fixed tolerance. The commands illustrate the old chars / 4 calculation only; the current pinned runner produces a new, versioned receipt with exact tokenization. Per-task analysis for all 30 historical queries lives in parity-benchmarks.html.

4. Closed-loop agent benchmark (0.5.14, gated in 0.5.15)

The historical 0.5.14 fixture expanded the agent benchmark from one retrieval call to the full loop that coding agents actually run: path discovery, evidence gathering, and sufficiency scoring before the next reasoning step. It does not call remote Claude, GPT, or any cloud model. Instead it compares two deterministic local policies: a skygrep-first agent that uses JSON path-only probes, compact snippet passes, scoped reads, and --no-rerank first; and a raw rg-only agent that runs repeated term searches and line-window reads.

The task set covers 38 generic maintenance questions across this repo plus Django, React, and Tokio. The scoring model measures expected-path coverage, path precision, evidence-term coverage, sufficiency, completed tasks, tool-call count, raw retrieval elapsed time, estimated downstream agent elapsed time, and context tokens. Path precision and context volume measure how much irrelevant evidence the next LLM turn has to filter.

Historical framing: raw rg remains the recall ceiling, and the skygrep-first policy still missed three of the 38 tasks that broad lexical dumping covered. Many public tasks in this 0.5.14 fixture were path-only and carried no literal evidence terms, while the agent elapsed row is modeled rather than measured. Keep these numbers as a historical receipt, not a current general multiplier. General Benchmark v2 uses pinned public commits, non-empty source evidence, exact tokenization, paired quality gates, and confidence intervals. Reproduce the legacy receipt with benchmarks/universal_closed_loop_benchmark.py --repo self --repo django --repo react --repo tokio --summary-only. In 0.5.15, saved JSON reports can be checked by benchmarks/closed_loop_regression_gate.py, which fails a release when skygrep-first falls below configured coverage, sufficiency, context-reduction, or work-quality-per-minute thresholds.

4b. Hybrid agent-context parity (0.5.17)

0.5.17 re-runs the self repository-maintenance task set against a real ripgrep agent baseline after promoting bounded rg, path tokens, symbols, chunk text, source-type priors, and symbol anchors into one automatic --agent-context recall substrate. The purpose is narrower than the 0.5.14 closed-loop benchmark: it asks whether the first machine-readable evidence pass can recover every target that broad ripgrep finds, while still giving the next LLM much less context to read.

The release reading is: for this agent-facing self benchmark, skygrep now reaches the same hit-rate ceiling as real ripgrep while making recall automatic inside --agent-context. Agents should inspect agent_summary, why_ranked, and suggested_followup_probe before manually broadening to raw rg. Fresh indexing for this run took 192.281 s; the latency row above reports hot-query reuse after the index exists.

4c. Indexing and closed-loop hardening (0.6.0)

0.6.0 measures the new cross-file embedding batch path against the previous per-file request pattern on the same repository, then reruns the medium closed-loop agent policy after the confidence and benchmark-contract fixes. These numbers are separate receipts: the indexing row measures a fresh build; daemon and doctor rows measure hot operational paths; the closed-loop rows measure eight generic repository tasks.

Accuracy is not inferred from speed. Regression tests separately cover deterministic chunk/file ordering, explicit-version authority, stale watch cleanup, retrieval-only confidence, and answer-source filtering. A live release-policy answer cited only the current docs/RELEASING.md anchor after unrelated snapshots and planning notes were removed.

4d. Strict evidence and CI contract gate (0.7.0)

0.7.0 adds a deterministic, model-free six-task fixture that executes the real hybrid candidate-recall implementation and feeds its report through the closed-loop regression gate on every pull request. The raw-rg side models the repeated exact-search calls and returned context an agent would need for the same evidence contract. This fixture is a stable regression check, not a universal runtime claim.

Separate unit and HTTP integration tests enforce strict fresh/stale behavior, exit-code semantics, direct/daemon Top-K and provenance parity, daemon project-boundary rejection, and macOS path-alias coalescing without lexical score double-counting.

4e. General Benchmark v2

General Benchmark v2 replaces a single headline multiplier with a paired, quality-gated cross-repository protocol. It pins exact commits of Cobra, Django, React, Spring Framework, Tokio, and Vite and evaluates 60 public source-evidence tasks. Every task requires an accepted path plus literal evidence from that source; empty path-only evidence contracts are rejected.

The 2026-08-15 clean-source receipt is reportable. On the 53 / 60 unique tasks where both policies met quality ≥ 0.85, skygrep-first returned 17.982× less tool context at the median. The repository-aware 95% hierarchical-bootstrap interval is 5.202×–94.127×. Retrieval-context quality was 92.5% for skygrep versus 88.4% for rg-only; completion was 180 / 180 versus 159 / 180 observations.

rg-only lets --max-count act per file, so common terms return most of a large repository (median 13.8M tokens per Spring task). The ratio therefore measures savings against an unbounded dump, not against an agent that truncates tool output. 0.8.0 adds the realistic rg-agent baseline and the token-lean skygrep-slim policy; see General performance.

This remains a retrieval-workflow benchmark: context-token counts exclude hidden model reasoning, final-answer output, provider caching, billing, and price. See the full methodology and reproduction commands or inspect the immutable raw receipt. The older 4.05× six-task result remains useful as a deterministic CI regression receipt, not as a general multiplier.

5. Agent bounded-latency smoke matrix (0.5.16)

0.5.16 adds a focused release smoke matrix for the agent timeout contract: scoped path anchors, compact evidence snippets, router-timeout fallback, cascade-timeout fallback, setup-instruction discovery, deep known-file reads, broad implementation lookup, and an absent-concept negative control. The matrix is intentionally local and generic; it uses repository-maintenance questions, not private user paths or document names.

The intended reading is not "every machine will return in exactly these seconds." It is that agent calls now have an enforced bounded-response policy: useful compact evidence when available, fallback or an honest empty result when evidence is weak, and no unbounded foreground refresh before JSON returns.

6. skylakegrep self-test (regression guard)

Deterministic local benchmark over 30 repository-navigation tasks against this very repo. Compares a single skygrep search call against a simulated grep-agent. Token volumes are estimated as chars / 4. Every release is verified to keep 30 / 30 at top-k 10.

Vs real ripgrep (not the simulated grep-agent), this same task set shows ~17.7× total-token reduction at equal recall. See the benchmark protocol for definitions and limitations.

How to scope the historical receipts

The examples below are versioned engineering records, not current general performance claims. Cite them only with the named fixture and method. General Benchmark v2 does not publish a multiplier until its quality, sample, source, and confidence gates pass.

• Historical single-turn and multi-turn agent fixtures → benchmark 1: −37.6 % single-turn, −82 % multi-turn (real Claude Code sub-agents, 20 + 1 hand-labelled tasks).
• Historical 0.2 public-OSS parity fixture → benchmark 2: 30 / 30 (100 %) across Django + React + Tokio public OSS, vs rg's 30 / 30 baseline (which emitted 20 M+ legacy-estimated tokens per query).
• Historical django-001 worked example → benchmark 3: django-001 worked example with archived chars / 4 estimates — rg 3.6 M tokens vs skygrep 2.6 K tokens (≈ 1,395 ×) on a single vocab-mismatch query.
• Historical 30-task self-test → benchmark 6: ~17.7× total-token reduction at equal recall on the 30-task self-test.
• Historical 0.5.17 agent-parity fixture → benchmark 4b: 30 / 30 vs real rg, 22.28× less context, 19.61× lower estimated total tokens, 30 tool calls instead of 179, and much higher MRR.
• Historical 0.5.16 bounded-latency smoke matrix → benchmark 5: the 0.5.16 smoke matrix covers scoped path, evidence, timeout fallback, deep read, broad lookup, and negative controls.
• "the legacy 0.5.14 fixture reported smaller context and modeled closed-loop utility" → benchmark 4: 470× less context, 23.7× lower modeled agent elapsed, and 22.9× higher modeled work quality per minute. These are historical fixture figures, not a current general claim.

Don't combine these into a single number or detach them from their release and fixture labels; they answer different questions. The historical protocol is in parity-benchmarks.html's archived methodology section, while the current claim contract is General Benchmark v2.