EvolveBench: Evaluating Coding Agents on Real Repositories

The AlgoTune experiments taught me how much an evolutionary coding agent depends on its evaluator. It also left an important question unanswered. AlgoTune presents a clean function, a reference implementation, and a narrow objective. Real software rarely does. The relevant code is embedded in a repository, performance depends on surrounding behavior, and the evaluator itself can encode the wrong goal.

I wanted to test whether the same approach would survive that messier setting. We built EvolveBench around real functions in actively maintained libraries, including CPython, pandas, SymPy, NetworkX, and python-jsonschema. Instead of asking only whether an agent could optimize code, we asked whether we could construct an evaluation signal precise enough to guide that optimization without rewarding regressions or superficial shortcuts.

I helped turn 11 repository-level bottlenecks into repeatable optimization tasks. Each task needed a clearly bounded edit target, representative inputs, correctness checks, a stable performance measurement, and an aggregation rule that made results comparable. The targets covered algorithmic complexity, data structures, parser loops, cache policies, numerical routines, and library-specific regressions.

The most important design choice was to treat the evaluator as an experimental variable. Each task ships with a handwritten evaluator built from direct inspection of the target, an LLM-generated evaluator synthesized from the task description, and a config for an LLM-as-a-judge evaluator for objectives that resist a fixed metric, though I only ran the first two below. This separated two capabilities that are often conflated: improving a program and specifying what improvement should mean.

An evaluator has to do more than run a test suite. It must reject behaviorally invalid programs, expose a useful gradient among valid candidates, and measure the workload we actually care about. If any one of those pieces is weak, evolution can optimize the measurement instead of the program.

For pandas_rolling_rank, the generated evaluator used pandas as the oracle and built a two-stage test surface. The quick stage covered five representative cases, while the comprehensive stage expanded to 12, including NaNs, infinities, ascending and descending order, three tie-breaking methods, percentile mode, and several array/window sizes. The core loop looked like this, abridged for readability¹:

The staging matters. Small cases cheaply eliminate broken candidates before the full suite consumes the budget. Differential testing avoids maintaining hand-calculated outputs, while fixed random seeds make failures reproducible. Warming the Numba kernel before starting the timer measures steady-state throughput rather than compilation latency².

The generated evaluator was not simply longer than the handwritten one. It encoded a different search landscape: more workload variation, measured baseline targets at three scales, and a smooth sigmoid reward around each target. This gave the agent credit for partial performance gains while keeping correctness dominant.

EvolveBench scores AI-driven optimization systems on real repositories, not synthetic functions. The EvolveBench score is the harmonic mean of per-task speedups, following the aggregation convention used in AlgoTune. OpenEvolve is currently the only system benchmarked; each evaluator design it was run with counts as a separate config below. Submit a result for another system and it will be added here.

We ran the same 11 tasks with handwritten and LLM-generated evaluators. The EvolveBench score is the harmonic mean of per-task speedups, following the aggregation convention used in AlgoTune.

The aggregate result favored generated evaluators, but the task-level failures were more informative than the headline number. These numbers are post-run measurements of the selected programs, not the internal reward returned during evolution³. That separation is important: the evaluator guides search, while an independent benchmark tells us whether the resulting program improved.

Rolling rank is the clearest example. The handwritten run returned an implementation that was effectively unchanged and measured at 1.00×. The generated-evaluator run instead produced a specialized Numba kernel: for every output position it scans the small active window, counts values below and equal to the current value, and computes the requested tie rule. Although that kernel is O(nw) rather than the skiplist's O(n log w), the JIT-compiled implementation and a smaller constant factor made it faster in the benchmark's small-window regime.

Here is the actual change, abridged to the two compute methods that decide the algorithm:

The difflib run found a different kind of optimization. Its selected program cached cheap similarity bounds and skipped the expensive full ratio whenever the bound could not beat the current match. Across seven pathological input sizes, the generated-evaluator result reduced total time from 7.773 seconds to 2.196 seconds, a 3.54× speedup. The handwritten-evaluator result took 7.862 seconds, or 0.99×. The useful lesson was not that generation is inherently superior. It was that broader cases and a smoother signal can expose an optimization path that a carefully written but narrower harness fails to reward.

SymPy provides the counterexample. The task optimizes Min/Max local-zero discovery over partially ordered symbolic values. Here, domain structure mattered more than generic workload diversity: the handwritten evaluator guided the run toward a 2.40× result, while the generated-evaluator run remained at 0.99×. This kind of symbolic code has semantic traps, such as comparability, transitivity, and antichains, that are difficult to infer from an interface alone.

EvolveBench grew out of problems we had already observed in the AlgoTune experiments. Naive model ensembles overwrote one another's progress, manual parameter sweeps consumed substantial effort, and incremental evolution sometimes proposed near-duplicate candidates. We used those failures as requirements for three additions to OpenEvolve.

We replaced equal model alternation with Thompson sampling¹. The router samples one model per iteration from a distribution fitted to recent reward history, while a minimum round-robin phase prevents early lockout. We also tested island-based variants that let different models preserve separate search trajectories.

The result changed how I think about multi-model systems. Diversity is useful only when the system protects useful state and assigns work based on evidence. On connected components, removing migration let a Union-Find strategy mature without being replaced by incompatible breadth-first-search edits. No ensemble configuration won everywhere, so the routing policy remains part of the optimization problem.

We added a local embedding-based novelty check that compares a new program against the current island population before it consumes an expensive evaluation. This turns a pattern I previously found by manually reading trajectories into an explicit filter in the search loop:

class NoveltyChecker:
    def __init__(self, embedding_client, similarity_threshold: float = 0.95):
        self.embedding_client = embedding_client
        self.similarity_threshold = similarity_threshold

    async def check_novelty(self, candidate_code, island_programs):
        candidate_embedding = await self.embedding_client.generate_embedding_for_code(candidate_code)
        max_sim = max(
            cosine_similarity(candidate_embedding, prog.metadata["code_embedding"])
            for prog in island_programs
        )
        return NoveltyCheckResult(is_novel=max_sim <= self.similarity_threshold, max_similarity=max_sim)

A candidate above the 0.95 threshold is not simply discarded. OpenEvolve regenerates from the same parent up to a fixed number of attempts, raising the sampling temperature after each one and telling the model what its last attempt was too similar to, so a rejection becomes feedback rather than a dead end. Only after exhausting those attempts does it fall back to the least similar candidate produced. Embeddings run locally (all-MiniLM-L6-v2), so novelty checking adds no LLM cost.

We also automated the prompt and hyperparameter sweeps that had been launched manually in the AlgoTune experiments, as a fixed-budget phase that runs before the main evolution loop:

@dataclass
class WarmupConfig:
    prompt_optimization_budget: int = 5   # Phase 1: prompt variants
    hyperparameter_budget: int = 5        # Phase 2: configs to test
    iterations_per_config: int = 5        # Phase 2: iterations per config
    temperature_min: float = 0.2
    temperature_max: float = 0.9
    exploration_min: float = 0.1
    exploration_max: float = 0.4

Phase 1 runs pairwise feedback descent: generate a prompt variation, evaluate the original and the variant on a one-iteration mini-experiment, then ask the model what made the winner better and fold that into an improved prompt. Phase 2 tests up to hyperparameter_budget hand-designed presets (conservative, exploratory, balanced, and focused combinations of temperature and exploration/exploitation ratio), runs each for iterations_per_config iterations, and keeps whichever preset produced the largest measured improvement. At the reference 50-call budget, that is 5 calls for prompt optimization, 25 for hyperparameter trials, and 20 left for the main search.

The class implementing this is named BayesianWarmupOptimizer, but the name is aspirational rather than descriptive. Phase 2 has no surrogate model and no acquisition function. It picks the best of a small, fixed set of presets by direct measurement, which is closer to a greedy grid search than to Bayesian optimization. Fixing that gap, along with running the ablation above, is the concrete next step.

1. An evaluator is an executable specification. It determines which behavior the agent learns to preserve and which shortcuts it is allowed to exploit.
2. Real repositories expose hidden dependencies. A locally faster function is not an improvement if it breaks callers, changes numerical behavior, or moves work elsewhere.
3. Automating evaluator construction changes the bottleneck. It reduces manual setup, but shifts more responsibility to validation and adversarial testing of the generated objective.
4. Adaptive systems need evidence at every layer. Model routing, novelty filtering, and warmup tuning are useful only when their feedback is comparable and reproducible.