Evaluation contract

General Performance Benchmark v2

Across pinned public repositories, how much retrieval context and tool work does a skygrep-first agent policy use relative to an rg-only policy, after both meet the same task-quality floor?

The older 4.05× result remains a deterministic six-task CI receipt. It is not a general multiplier. General Benchmark v2 separates correctness, context efficiency, and measured retrieval latency so none can hide a regression in another dimension.

Published result · 2026-08-15

The first release-scale receipt passed every General v2 gate. It was produced from clean source commit e47e7f7b100bd1fcf30f28ea509703a1d2d1f17a on Darwin arm64 with Python 3.13.7, local bge-m3 embeddings, batch size 64, exact tiktoken:cl100k_base counting, six fresh/reset indexes, and three trials per policy and task.

Reportable headline: 17.982× median reduction in returned tool-context tokens on the 53 quality-eligible paired tasks; repository-aware 95% hierarchical-bootstrap interval: 5.202×–94.127×.

The eligible-task P25 / median / P75 is 5.998× / 17.982× / 93.141×; the eligible ratio of token sums is 83.181×. The median is the headline so each unique task contributes one observation after its three trials are combined.

No universal speed or tool-call claim: the paired median rg elapsed / skygrep elapsed ratio was 0.561, so the median eligible skygrep task took longer even though the all-row aggregate elapsed ratio was 2.41 in skygrep's favor. The paired median tool-call ratio was 0.929, and skygrep used more calls overall. Large-repository outliers make aggregate totals useful capacity readings, not typical-task latency claims.

Read the immutable raw receipt · capacity receipts and provenance

Token-lean policies · 0.8.0

The rg-only baseline lets --max-count act per file, so common terms return most of a large repository (median 13.8M tokens per Spring task in the receipt above). Its ratios measure savings against an unbounded dump, not against an agent that truncates tool output. 0.8.0 adds two opt-in policies:

• skygrep-slim: one --agent-slim call, skygrep symbols on the next result files three at a time, path/filename probes feeding more outline batches, a progressive version of the symbol sweep, and only then the full skygrep-first policy (paying for both).
• rg-agent: per-term rg -c with agent-visible output capped at 250 lines, BM25-style ranking, the same outline reader, then reads capped at 2,000 lines.

A 60-task smoke run (1 trial, chars/4, stand-in embeddings so the semantic lane is noise) is a regression check, not a receipt:

On the 46 tasks both complete, skygrep-slim has a 1.31× lower median than rg-agent but a higher total (ratio 0.75×): the 5 tasks that fall back to the full policy dominate the sum. A bge-m3 receipt with three trials is a release follow-up. Compare policies with benchmarks/compare_policies.py.

Public matrix

Every task has a canonical path, optional accepted alternatives, at least two literal evidence terms, at least two quality terms, and a public ground-truth note. Validation fails if a commit, origin, tracked worktree, accepted path, or evidence term drifts. Each report also records the benchmark source commit and whether its tracked worktree was clean.

Quality before efficiency

• Both policies must complete the paired task and achieve quality ≥ 0.85.
• skygrep must remain within two percentage points of rg for aggregate quality and completion.
• A general claim needs at least 30 unique eligible tasks across three repositories and at least 80% eligible paired observations.
• Repeated trials stabilize measurement but do not increase the unique-task count.
• The benchmark source must be one full Git commit with a clean tracked worktree; all six parallel receipts must use that same commit.

The quality score is a deterministic, auditable retrieval-context proxy based on accepted paths, literal source facts, noise control, sufficiency, and stopping behavior. It does not claim that an unobserved downstream model wrote a correct final answer.

Only after these gates pass does the report calculate context-token and tool-call reduction, measured harness-wall and retrieval-tool elapsed ratios, cross-task and per-repository distributions after taking each task's median across trials, ratio-of-sums, and a 95% hierarchical-bootstrap interval over repositories and unique tasks. Harness wall time includes deterministic scoring and token counting; retrieval tool time is captured before token counting.

Run cadence

The scheduled GitHub workflow validates the protocol on a standard hosted runner; it does not mint a new performance receipt. The full workflow is a manual dispatch targeting a self-hosted runner labeled skygrep-benchmark. It first fresh-indexes Cobra and projects the other five workloads from exact chunk and capped-model-input counts, stopping before the large indexes if any projection exceeds the index timeout. Local reruns reuse completed indexes unless the index format, embedding model, or pinned source changes.

Reproduce

One command on a machine with Ollama + bge-m3 (Apple-silicon Metal or another accelerator) runs all four policies on a fresh clone, privacy-scans the receipt, and pushes it to a bench/receipt-YYYY-MM-DD branch for review:

curl -fsSL https://raw.githubusercontent.com/danielchen26/skylakegrep/master/scripts/run_general_receipt.sh | bash

Or step by step:

python -m pip install -e ".[benchmark]"
python benchmarks/validate_public_fixtures.py \
  --oss-root /tmp/skygrep-general-v2-repos --prepare
python benchmarks/universal_closed_loop_benchmark.py \
  --oss-root /tmp/skygrep-general-v2-repos --prepare \
  --refresh-index --reset-index --index-timeout 18000 --trials 3 \
  --tokenizer tiktoken --report /tmp/skygrep-general-v2.json \
  --summary-only
python benchmarks/closed_loop_regression_gate.py \
  /tmp/skygrep-general-v2.json --require-general-reportable

The first run builds six complete semantic indexes and can take hours on a CPU-only or memory-constrained machine. Index elapsed is recorded separately and never enters the retrieval-efficiency headline. Omit --refresh-index --reset-index on later query-only repetitions.

The manual accelerated workflow runs the six repositories after the capacity gate, then merges compact rows and recomputes one aggregate quality gate and confidence interval. Capacity, per-repository, and merged receipts are retained as artifacts. The weekly protocol workflow validates pins and benchmark logic without presenting protocol validation as a fresh performance result.

Claim boundary

This is a retrieval-workflow benchmark. Its quality gate scores retrieved source support rather than a model-generated final answer. Token counts cover tool context, not hidden model reasoning, final-answer output, cache billing, or provider pricing. A provider end-to-end claim additionally requires the same model, prompt, temperature, agent loop, and provider usage receipts in both conditions.

← Back to all benchmarks