Evaluation · benchmarks

Empirical results across four benchmark dimensions.

skylakegrep ships multiple reproducible benchmark families. Each measures a different question; quoting the right number for the right claim matters. The repository's parity-benchmarks.html holds the full raw tables, methodology, and per-task data; this page is the curated summary.

0a. Wrong-path quick-answer (NEW in 0.5.7)

The newest UX gate. Measures how fast the parallel proactive umbrella streams a first answer when the user is in an unrelated cwd and the answer lives in a sibling folder. Setup: cd into an empty directory, SKYGREP_PROACTIVE_DIRS=/tmp/oss-bench, query "do I have any package configuration files". The user runs the real skygrep CLI; we measure wall clock to first stream block AND wall clock to script exit.

Concrete output captured in the v0.5.7 release notes (full notes →): the streaming markers 🔍 / ▾ / 🌐 / 🌊 / ⚡ arrive with route + quality labels (filename_extend, ~100 ms-1 s; pure filename glob, no semantic understanding / cross-folder lazy, embed budget 5 seeds, σ-validated cosine) so the user can judge the answer's quality against the route that produced it. Cascade has a 30 s hard timeout; cross-folder lazy has an 8 s hard timeout.

0b. Cold-start lazy auto-trigger (NEW in 0.5.3)

The newest tier — measures whether the --lazy auto-trigger that fires on a never-indexed project actually beats plain ripgrep cold-start on vocabulary-mismatch queries. Run via benchmarks/release-0.5.3-rg-vs-lazy.py — 10 hand-labelled Django queries phrased as natural language ("where is the migration runner that applies pending schema changes to the database"), every query starts from a freshly cleaned SQLite DB, and the script invokes the real skygrep search CLI (not the python API) so the measured numbers are exactly what a user would see. Compares --no-lazy (pure rg cold-start) vs default (auto-trigger).

Specific hits: Q1 → django/urls/resolvers.py; Q3 → migration.py + executor.py; Q4 → backends.py; Q7 → base.py. 0.5.3 release notes call out the misses (Q2, Q5, Q6, Q8, Q9, Q10 still don't hit because qwen 2.5:3b can't reliably pick the right dir for some oracle phrasings — tracked as a 0.6 candidate for a larger router model).

1. End-to-end Claude Code agent

Real Claude Code sub-agents answering hand-labelled code-search questions in two prompted conditions — rg-only (skygrep forbidden) vs skygrep-on (rg/grep/find forbidden) — across 21 questions and 1 multi-turn session in three repos.

The cleanest, most consistent signal is tool-call reduction: −37.6 % single-turn, −82 % multi-turn. Each agent tool call costs an LLM round-trip + network RTT + serialization + context-window growth, so reducing them shortens the agent loop even when total tokens are equal. Token cost across the 20-task aggregate is roughly flat (+6.5 %); we do not claim skygrep saves the LLM bill.

Best-case task in the original aggregate: a vocabulary-mismatch question (NL phrasing didn't match any code identifier) — skygrep finished in 1 tool call vs rg-only's 25 (25× fewer), at one eighth the wall time. Worst-case: a token-friendly question whose vocabulary (auth / session / token) overlapped directly with code-path tokens, so rg's straightforward scan was already efficient — skygrep was 40 % more tool calls. The lexical pre-gate addresses this via a conservative four-condition gate that detects exactly these queries and short-circuits to rg internally in ~50 ms. Methodology and aggregate published in the release notes; see the release notes for the routing fix.

2. Public OSS recall (Django · React · Tokio)

30 hand-labelled questions across three popular open-source codebases. Anyone can clone the repos and rerun benchmarks/public_oss_bench.py to reproduce every number below.

Honest framing: rg's 100 % is a recall-ceiling baseline — it returns 20 M+ tokens of term-OR scan output per query, so the answer is in there but the agent has to read the whole haystack. skygrep returns the right file ranked top-10 in 30 / 30 cases while emitting 60×–770× less context. React reached 10 / 10 after the Option-C substrate upgrade (bge-m3 embedder + content-agnostic non-canonical-path filter); the original failure modes on react-007 and react-010 and the resolution are documented in parity-benchmarks.html as the engineering record, not erased.

3. Worked example — django-001 (one query, real numbers)

Reproduce locally to verify every number. One of the 30 tasks aggregated into the public-OSS recall table above, run against the actual Django source tree (524 K LOC).

Query: "Where does Django turn an incoming URL into the view function that should handle it?"
Expected canonical: django/urls/resolvers.py (URLResolver.resolve())
Vocab mismatch: query says "URL into view", code identifier is resolve — the failure mode that grep-as-search collapses on.

Side A — rg term-OR scan

The rg-agent extracts up to 8 terms (TF-IDF-ish stopword filter), runs rg -i -F --max-count=20 -C2 per term, concatenates output. For this query the extractor produced:

['function', 'incoming', 'incom', 'django', 'handle', 'should', 'into', 'that']

The high-signal words URL, view, resolve did not survive the extractor — they were either stopword-filtered or pushed past the 8-term cap. That is the vocab-mismatch failure: the query's actual intent is URLResolver.resolve(), but rg searches for function, django, that instead.

Real measured output volumes per rg invocation:

django alone is 1.4 million tokens because the term matches in basically every file of the Django source tree. that is 840 K tokens for the same reason — high-frequency words that the stopword filter let through.

Side B — skygrep --top 10 --json

$ skygrep "Where does Django turn an incoming URL into the view function that should handle it?" \
    --json --top 10
$ wc -c   # 10,430 chars ≈ 2,607 tokens

Top files returned include django/urls/resolvers.py and related canonical implementation files.

Reduction — visual comparison

1,395 × is higher than the 60 × – 770 × headline range because vocab-mismatch queries are the worst case for rg (stopwords flood the output) and the best case for skygrep (the embedder bridges "URL into view" → resolve()).

Why the headline is 60 × – 770 ×, not a single number

rg output scales with (repo LOC) × (term-frequency of high-signal terms). skygrep output is ~constant-per-K (top-10 ≈ 10 KB):

Tokio is the floor (small repo, focused vocabulary). Django and React both blow up because django / react saturate term-OR scans across mid-sized monorepos.

Reproduce yourself (3 commands)

# rg side — counts bytes from term-OR scan
cd /tmp/oss-bench/django
for term in function incoming incom django handle should into that; do
  rg -i -F --max-count 20 -C 2 "$term" .
done | wc -c

# skygrep side — counts bytes from top-10 JSON
skygrep "Where does Django turn an incoming URL into the view function that should handle it?" \
  --json --top 10 | wc -c

# divide chars by 4 to approximate tokens

Numbers will land within ± 5 % of the ones above (variance from your Django clone's commit and rg minor-version output formatting). Per-task analysis for all 30 queries lives in parity-benchmarks.html.

4. Closed-loop agent benchmark (0.5.14, gated in 0.5.15)

0.5.14 expands the agent benchmark from one retrieval call to the full loop that coding agents actually run: path discovery, evidence gathering, and sufficiency scoring before the next reasoning step. It does not call remote Claude, GPT, or any cloud model. Instead it compares two deterministic local policies: a skygrep-first agent that uses JSON path-only probes, compact snippet passes, scoped reads, and --no-rerank first; and a raw rg-only agent that runs repeated term searches and line-window reads.

The task set covers 38 generic maintenance questions across this repo plus Django, React, and Tokio. The scoring model measures expected-path coverage, path precision, evidence-term coverage, sufficiency, completed tasks, tool-call count, raw retrieval elapsed time, estimated downstream agent elapsed time, and context tokens. Path precision and context volume measure how much irrelevant evidence the next LLM turn has to filter.

Honest framing: raw rg remains the recall ceiling, and the skygrep-first policy still missed three of the 38 tasks that broad lexical dumping covered. The 0.5.14 win is closed-loop agent economy: much less context, lower estimated end-to-end agent time, and nearly the same sufficiency when the agent follows the path-first / evidence-second playbook. Reproduce with benchmarks/universal_closed_loop_benchmark.py --repo self --repo django --repo react --repo tokio --summary-only. In 0.5.15, saved JSON reports can be checked by benchmarks/closed_loop_regression_gate.py, which fails a release when skygrep-first falls below configured coverage, sufficiency, context-reduction, or work-quality-per-minute thresholds.

4b. Hybrid agent-context parity (0.5.17)

0.5.17 re-runs the self repository-maintenance task set against a real ripgrep agent baseline after promoting bounded rg, path tokens, symbols, chunk text, source-type priors, and symbol anchors into one automatic --agent-context recall substrate. The purpose is narrower than the 0.5.14 closed-loop benchmark: it asks whether the first machine-readable evidence pass can recover every target that broad ripgrep finds, while still giving the next LLM much less context to read.

The release reading is: for this agent-facing self benchmark, skygrep now reaches the same hit-rate ceiling as real ripgrep while making recall automatic inside --agent-context. Agents should inspect agent_summary, why_ranked, and suggested_followup_probe before manually broadening to raw rg. Fresh indexing for this run took 192.281 s; the latency row above reports hot-query reuse after the index exists.

5. Agent bounded-latency smoke matrix (0.5.16)

0.5.16 adds a focused release smoke matrix for the agent timeout contract: scoped path anchors, compact evidence snippets, router-timeout fallback, cascade-timeout fallback, setup-instruction discovery, deep known-file reads, broad implementation lookup, and an absent-concept negative control. The matrix is intentionally local and generic; it uses repository-maintenance questions, not private user paths or document names.

The intended reading is not "every machine will return in exactly these seconds." It is that agent calls now have an enforced bounded-response policy: useful compact evidence when available, fallback or an honest empty result when evidence is weak, and no unbounded foreground refresh before JSON returns.

6. skylakegrep self-test (regression guard)

Deterministic local benchmark over 30 repository-navigation tasks against this very repo. Compares a single skygrep search call against a simulated grep-agent. Token volumes are estimated as chars / 4. Every release is verified to keep 30 / 30 at top-k 10.

Vs real ripgrep (not the simulated grep-agent), this same task set shows ~17.7× total-token reduction at equal recall. See the benchmark protocol for definitions and limitations.

Which number to cite for which claim

• "skygrep cuts agent tool calls" → benchmark 1: −37.6 % single-turn, −82 % multi-turn (real Claude Code sub-agents, 20 + 1 hand-labelled tasks).
• "skygrep matches rg on hit-rate while emitting 60×–770× less context" → benchmark 2: 30 / 30 (100 %) across Django + React + Tokio public OSS, vs rg's 30 / 30 baseline (which dumps 20 M+ tokens per query for the agent to filter).
• "this is what one query actually looks like" → benchmark 3: django-001 worked example with real measured volumes — rg 3.6 M tokens vs skygrep 2.6 K tokens (≈ 1,395 ×) on a single vocab-mismatch query.
• "skygrep is more token-efficient than ripgrep when feeding LLM context" → benchmark 6: ~17.7× total-token reduction at equal recall on the 30-task self-test.
• "agent context now matches real rg hit-rate with far less context" → benchmark 4b: 30 / 30 vs real rg, 22.28× less context, 19.61× lower estimated total tokens, 30 tool calls instead of 179, and much higher MRR.
• "agent JSON calls are now bounded against slow foreground work" → benchmark 5: the 0.5.16 smoke matrix covers scoped path, evidence, timeout fallback, deep read, broad lookup, and negative controls.
• "skygrep gives agents much smaller context and faster closed-loop utility" → benchmark 4: 470× less context, 23.7× lower estimated agent elapsed, and 22.9× higher work quality per minute on the 0.5.14 closed-loop agent benchmark.

Don't combine these into a single number; they answer different questions. The honest framing is in parity-benchmarks.html's "Strongest claims" section.