Kai: A Verifier-First Autonomous Codebase Engineer

I helped develop Kai from a coding-agent prototype into an always-on codebase engineer. It had to work across two very different tasks. In optimization mode, it evolved real code against developer-defined correctness and performance tests. In security mode, it searched for vulnerabilities, checked whether they were reachable and economically meaningful, produced an exploit, and proposed a patch.

My work sat at the intersection of agent infrastructure, evaluation, and execution. I brought lessons from OpenEvolve and EvolveBench into the optimization workflow, and I helped connect the specialist harnesses to the product surfaces and isolated environments needed for longer autonomous runs. That changed the question from "can the model suggest a patch?" to "can the system execute, verify, measure, and report that patch without asking a human to reconstruct what happened?"

We designed Kai around one coordinating agent and two specialist harnesses. The agent maintains conversation and workspace context, while the security and optimization harnesses own their evaluation contracts. Both return normalized reports containing status, findings, artifacts, and metrics.

The key invariant was simple: if Kai could not verify a security finding or confirm that an optimization remained correct, the result stayed unverified. I found this separation useful because generation and judgment fail differently. A model can propose many plausible changes, but the harness decides which ones are allowed to affect the final answer.

Kai Evolve applies the verifier-first idea to performance work. A language model proposes a code change, but that change contributes to the search only after it passes the task's correctness tests and produces a measured runtime improvement. Developer-defined fitness functions let the same evolution engine target different repositories without coupling those objectives to the agent core.

We reused Thompson sampling¹ from our OpenEvolve work to route proposals among three models according to recent performance rather than fixed alternation. In the reported run, GPT-5.2 Codex received 36.2% of selections, Claude Sonnet 4.5 received 34.6%, and Grok 4.1 Fast received 29.2%. The relatively even allocation showed that no single model dominated every task.

The same verifier-first idea carries over to security work, where findings are especially vulnerable to confident false positives. We treated verification as a program-analysis pipeline rather than another model response. Candidate bugs moved through focused static analysis, reachability tracing, state-transition reasoning, an assessment of real economic impact, and finally a runnable exploit test.

1. Research the protocol. Map the architecture, trust boundaries, design patterns, and relevant threat model.
2. Generate candidate findings. Inspect reentrancy, arithmetic, access control, oracle behavior, and economic invariants.
3. Verify independently. Trace whether the vulnerable state is reachable and whether the claimed impact follows.
4. Produce executable evidence. Write a Foundry test that triggers the exploit before reporting the finding as verified.

The public security_scan workflow packages discovery, verification, exploit construction, and patching into one sequence. We attach the finding, exploit, patch diff, and verification status to the report, which makes it possible to inspect the evidence instead of trusting a confidence score.

Wildcat Protocol. We identified a withdrawal-batch rounding interaction in which half-up division was composed with floor rounding. Once the scale factor crossed a threshold, the normalized amount paid could exceed available liquidity and consume reserved funds. A Foundry test reproduced the corrupted accounting path. The historical bounty attached to the finding was $20,252.

Noya Protocol. We found that a multi-hop oracle passed the original asset into every hop instead of the intermediate token produced by the previous hop. A controlled A→B→C→D test showed that the composed price no longer matched the expected 24× result, creating a route to misprice deposits and withdrawals.

A useful autonomous engineer needs to execute code, preserve workspace state, and survive failures without putting the host environment at risk. I helped connect Kai to a provider-agnostic execution layer with hosted E2B micro-VMs as the default backend, ephemeral Vercel sandboxes as an alternative, and a local mode for development.

The execution layer also supported long-running work. A /teleport command copied the workspace and conversation state into a sandbox and resumed the session in a browser, while /hand-off dispatched an autonomous sub-agent in the background. These interfaces kept audits and optimization searches from blocking the main session and made the execution boundary visible to the user.

I also worked on making Kai usable where engineering teams already communicate. The project exposed a terminal REPL and local web UI, plus adapters for Slack, Discord, Telegram, and email. A plugin system with pre-call, post-call, result-transformation, session-start, and session-end hooks let custom policies and logging wrap every tool call without forking the agent core.

We evaluated the security and optimization workflows separately because they measure different capabilities. EVMBench Detect asks whether the agent recovers known high-severity vulnerabilities from historical audits. GSO asks whether a correct patch reaches at least 95% of the runtime improvement achieved by an expert developer.

Against other systems evaluated on the same 120-finding EVMBench release, our Detect Recall led by a wide margin:

The GSO leaderboard from the same release shows a similar gap, though our row is not a like-for-like entry: it covers 30 compatible tasks rather than the official 102-task set, at a different compute budget.

For me, the useful result was not only the headline score. Both benchmarks rewarded the same system behavior: preserve an explicit contract, reject unsupported claims, and return evidence that another engineer can inspect. The verifier made the agent more useful by reducing the amount of trust required from the user.

1. Generation is the cheap part. The harder engineering problem is deciding whether a patch or finding deserves to leave the sandbox.
2. Verification should produce artifacts. A test, exploit, traceback, benchmark, or patch diff is more useful than an unsupported confidence score.
3. Execution boundaries are part of agent design. Sandboxes, handoffs, timeouts, and preserved state determine which long-running tasks are practical.
4. Specialist harnesses need a shared contract. Normalized status, findings, artifacts, and metrics let one coordinating agent work across very different domains.
5. Adaptive routing needs measured feedback. Multiple models help only when the system can compare recent outcomes and preserve productive search trajectories.