PlugBO: A Modular Framework for Agentic Bayesian Optimization

Bayesian optimization (BO) is the standard method for sample-efficient search of expensive, derivative-free black boxes. We use it for hyperparameter tuning, controller design, and experimental science. A probabilistic surrogate, typically a Gaussian process, maintains a posterior over the unknown objective. An acquisition function then turns that posterior into the next query. Usually we declare the surrogate family, acquisition function, search bounds, objectives, and constraints before the first evaluation, and we keep them fixed after that.

The fixed-policy assumption gives us clean algorithms, but it also rules out the adjustments that practitioners typically make during a real search. We might tighten a bound after identifying a regime, reclassify an outcome as a constraint, or update a previously misspecified belief. In recent years, large language models (LLMs) have given us a way to encode the prior knowledge that classical BO misses, including natural-language documentation and qualitative judgments about plausible regions.

Because of this potential, a number of recent approaches combine LLMs and BO. They generally take one of two paths. In some methods, such as OPRO and LLAMBO, the LLM itself acts as the optimizer, proposing candidates from a textual history without maintaining an explicit posterior. Other approaches insert the LLM into a single component of an otherwise standard loop. These preserve calibration, but they leave the outer policy unchanged. CAKE is an example of this second category. It lets the GP kernel evolve as observations arrive, but the overall optimization loop remains fixed.

Meta recently introduced agentic BO, a new framework where the LLM agent sits at the center of the optimization process, orchestrating all decisions while a Bayesian backend is responsible for maintaining and updating uncertainty. In their setup, Sara is the agent driving the search, while lenz, a BoTorch CLI, manages the trial log, posterior, and acquisition steps.

Building on the agentic BO framework, I developed an implementation that generalizes the backend into a flexible plugin system. In PlugBO, the surrogate, region, prior, and sampler are modular slots. Existing and new BO modules fill these slots via lenz commands, which the agent can activate, query, or modify at runtime. The agent, Sara, continues to operate with only bash and read access. This post explains the architecture, experimental setup, and main insights while I was working on this project.

Motivation

To make this gap precise, consider an expensive black-box objective \(f:\mathcal{X}\to\mathbb{R}\). After \(t\) evaluations we have data \(\mathcal{D}_t=\{(\mathbf{x}_i,y_i)\}_{i=1}^{t}\). A GP posterior \(p(f\mid\mathcal{D}_t)\) supplies a mean and a variance at every unevaluated point. An acquisition function \(\alpha(\mathbf{x}\mid\mathcal{D}_t)\) turns that posterior into a score, and we take the next query as

where expected improvement is one classical choice of \(\alpha\). The active region \(\mathcal{B}\) is usually the entire domain.

At each step \(t\) we can collect the live choices into a configuration

that holds the surrogate, acquisition, active region, objectives, and constraints. Standard BO is then a mapping \(\Pi_0\) that we choose before the first evaluation, from data to the next configuration and query:

The outputs of \(\Pi_0\) may vary over time, but we choose the mapping itself in advance. TuRBO expands or contracts a trust region according to a prescribed schedule. πBO reweights the acquisition by a user belief that decays with \(t\). In both cases a quantity inside \(c_t\) adapts, but the rule that governs the adaptation does not.

Agentic BO takes a different route. We replace \(\Pi_0\) with a meta-level loop. Between two expensive evaluations the agent can probe the backend, reconfigure any piece of \(c_t\), request or override proposals, or commit a point to the oracle. Only the last of these spends black-box budget. That split follows a broader tool-use idea: language models can delegate precise computation rather than approximate it internally.

Surrogate adaptation and policy adaptation are easy to conflate. Surrogate adaptation changes \(M_t\), that is, how we model the observations. Policy adaptation changes how we use that model, including which objective we optimize and which evaluation we commit. PlugBO exposes both as slots so that we can pin one layer in a comparison and vary the other.

Method

Agentic BO framework

We keep the two-part split from Meta’s paper. lenz is a stateless CLI over a persistent state.json Frame. The Frame stores the search space, a mutable shelf of live policy, a trial log, an event log, and a plugins blob. Changing a bound, swapping the acquisition, or occupying a slot updates the shelf. Trials are never discarded.

Sara runs the search. She receives bash and read, a system prompt that puts her in charge of the optimization decisions, and a command reference for lenz. Both tools are sandboxed to the run directory. The prompt tells her not to accept surrogate proposals blindly. Before each submission she has to state what she currently believes, what the next evaluation should reveal, and why the selected action follows from the context or the observations.

A run proceeds as follows. Sara reads context.md, calls lenz verbs, evaluates configurations through --eval or ./oracle, and stops when the budget is exhausted or she emits a final report without tool calls. The Frame refuses further observations once shelf.budget is reached. A separate step limit (MAX_STEPS = 400) bounds the tool-use loopSara also receives a soft nudge after 1.5 times the evaluation budget, asking her to report the incumbent and stop. Early stops in the logs are therefore agent decisions, not harness crashes..

We can also drop the backend entirely with --no-lenz. Sara then proposes configurations and calls ./oracle herself. That run tests whether the LLM can be the optimizer, rather than a controller of one.

Plugin slots

What is new in this implementation is the plugin protocol. The core never imports plugin internals. Instead it calls hooks, and plugins register extra CLI verbs. Occupancy comes from the shelf. The defaults (fixed, box, botorch, none) mean no plugin is active.

Each plugin occupies its slot in a slightly different way. CAKE evolves a kernel population inside lenz on observe or submit, using a separate kernel LLM. Sara can choose the surrogate and inspect the population. She does not see the kernel LLM’s reasoning trace, and the kernel LLM does not inherit her conversation.

TuRBO supplies GP-space trust-region bounds through active_bounds. πBO wraps the configured acquisition, \(\alpha_\pi(\mathbf{x})=\alpha(\mathbf{x})\,\pi(\mathbf{x})^{\beta/(t+1)}\), after Sara compiles a factorized belief with set-belief. LLAMBO may replace BoTorch proposals when it occupies the sampler slot.

Reconfiguring a slot never discards trials. Method state lives under frame.plugins[name], not on the live shelf.

Implementation

Start from the GitHub repository. The code is open source, and contributions are welcome. Clone it, then install from source (Python 3.10 or later):

git clone https://github.com/richardcsuwandi/plugbo.git
cd plugbo
pip install -e .

Copy .env.example to .env if you will call an LLM.

Starting a run

There are two entry points, and they share the same Frame. sara run puts the LLM in charge. lenz is the BO backend she calls, and it is also the no-agent loop, so the two are comparable.

--context is the problem markdown Sara reads. --eval is a black-box command: one config JSON in, metrics JSON out. --budget counts expensive evaluations. The Frame lives under --workdir.

sara run \
  --provider anthropic --model claude-opus-5 \
  --context examples/branin/context.md \
  --eval "python3 examples/branin/eval.py" \
  --budget 30 \
  --workdir ./results/logs/branin-1

Without Sara, a run is three verbs. create writes state.json. suggest asks BoTorch (or the occupied sampler) for a candidate. submit records the evaluation. If metrics are present, every occupied plugin gets on_observe, which is how a method such as CAKE sees new data rather than through Sara’s conversation.

lenz create --state ./state.json \
  --space '{"x1":{"kind":"range","lower":-5,"upper":10},"x2":{"kind":"range","lower":0,"upper":15}}' \
  --objectives '{"y":"minimize"}' \
  --acqf noisy_logei

lenz suggest --state ./state.json
lenz submit --state ./state.json --config '{"x1":1.0,"x2":2.0}' --metrics '{"y":12.3}'

Occupying a slot is another verb on that surface, which is why Sara never needs a third tool:

lenz set-surrogate --state ./state.json --surrogate cake
lenz set-region --state ./state.json --policy turbo
lenz set-belief --state ./state.json --prior '{"x":{"dist":"normal","mu":0.3,"sigma":0.1}}'
lenz set-sampler --state ./state.json --sampler llambo

Adding a plugin

A method is three files: a LenzPlugin subclass under lenz/plugins/, a short {name}.md note that gets concatenated onto Sara’s prompt, and a line in registry.py. Core never imports plugin internals. It calls hooks, and occupancy is a shelf field.

CAKE is the surrogate-slot case. After each observation it may evolve a kernel population, and it is a no-op unless the shelf surrogate is cake, at least two points exist, and a kernel LLM is configured. Failures are logged and never raised, so a plugin failure does not abort the run.

class CakePlugin(LenzPlugin):
    name = "cake"
    slot = SLOT_SURROGATE

    def on_observe(self, frame: Frame, trial: Trial) -> None:
        cake.maybe_evolve(frame, Encoder(frame.space))

The rest of the recipe, including tests, is in CONTRIBUTING.md.

Setup

Regimes and disclosure

Textbook functions such as Hartmann, Ackley, and Branin have published optima that are likely represented in LLM pretraining data. Brunzema et al. report that, without obfuscation, LLM-based methods can submit a known optimum on the first evaluation. Scoring search quality on that unobfuscated setting would confound retrieval of a published optimum with optimization. The harness therefore distinguishes three regimes:

On regime 1 we can present a function at three disclosure levels:

1. Blind: Renamed parameters, unit cube, shifted optimum, generic context.md. We write the true identity to _answers/ outside the sandbox and use it only for scoring.
2. Revealed + shifted: Real name and bounds, same shift as the blind run. Search is still required.
3. Revealed, unshifted (the textbook problem): The quantity of interest is whether the first evaluation is already the published optimum.

The reported comparisons in this regime use Hartmann6 and Ackley. Hartmann6 is a 6-D weighted sum of Gaussians on \([0,1]^6\), with a relatively localized basin and a published global minimizer. Ackley-10 is a 10-D oscillatory cosine sum on \([-32.768, 32.768]^{10}\), with a published optimum of \(0\) at the origin. Ackley-20 is the same landscape in higher dimension. Both answers are standard textbook numbers.

A second failure mode is reconstruction. Even when the identity is hidden, an agent may fit the observed \((x,y)\) pairs with an auxiliary script and optimize that approximation. The sandbox instructions prohibit this. We keep trace.jsonl so we can tell reconstruction apart from genuine interaction with the surrogate.

The disclosure levels above apply to named benchmark functions, where published global optima can potentially be retrieved. For tasks based on domain context, where no retrievable textbook optimum exists, we instead use LoRA hyperparameter optimization on BoLT (Black-box Optimization for LLM Tasks). In these experiments, the oracle is a deterministic Qwen3-8B emulator trained to mimic fine-tuning runs of Qwen3 on OpenMathInstruct-2, and evaluated against MATH-500. The agent optimizes over seven variables (scaled learning rate, batch size, rank, alpha, dropout, number of adapted layers, and target projection) and is explicitly told that there is no published or known numerical solution for this hyperparameter search.

Three context variants change the problem description, not the search space: domain (LoRA/Qwen), generic (names and bounds only), and misleading (incorrect LoRA claims: dropout near 0.05, lora_target = 0, few layers). πBO gets matching belief fixtures in benchmarks/priors.py that encode those stories without leaking the emulator’s hidden best.

The remaining control has no information to recall as gp_sample<dim> is not a textbook function. For each seed we draw a fresh approximate squared-exponential GP sample on the unit cube, using \(M=1028\) random Fourier features with lengthscale 0.2:

The weights and frequencies are sampled, so that particular path is absent from pretraining by construction. The comparisons below use gp_sample6.

Protocol

We score regret of the incumbent against the (possibly shifted) true optimum, and constrained tasks count only feasible trials. Sara and the CAKE kernel LLM use Qwen3.8-Max. Compare groups share a 100-evaluation budget, a seeded Sobol warm-start of \(d+1\) points, and noisy log expected improvement. Means are over three seeds. Agent runs keep occupied slots fixed (the prompt forbids set-acqf and set-surrogate) and may stop after a soft nudge at 1.5\(\times\) budget.

The methods fall into three groups.

No agent: a scripted lenz loop, with no LLM in the search.

• vanilla: fixed Matérn, noisy log expected improvement
• cake: CAKE occupies the surrogate slot
• turbo: TuRBO occupies the region slot
• pibo: πBO occupies the prior slot (LoRA only)

Sara with a backend: she can call lenz, and the occupied slot is pinned for the run.

• sara-lenz: default fixed Matérn
• sara-lenz-cake: CAKE pinned
• sara-lenz-turbo: TuRBO pinned
• sara-lenz-pibo: πBO pinned (LoRA only)

Sara without a backend: sara-only (--no-lenz). Warm-start trials can still be written into state.json for scoring, but she cannot call lenz. She reads the log and proposes subsequent points herself.

Results

Benchmark memorization

The following unobfuscated run motivates the blind protocol. If we reveal Hartmann6’s real name and bounds, without shifting its optimum, and let Sara run with the default fixed-Matérn backend, the trace log contains the following exchange.

Sara> I recognize Hartmann6 and recall its known global optimum. The standard
      textbook global minimizer is at x1=0.201690, x2=0.150011, x3=0.476874,
      x4=0.275332, x5=0.311652, x6=0.657300, with optimal value y* ~ -3.32237.
      This probe explicitly measures optimum recall, so I will submit this
      known optimum as my very first evaluation.

$ ./oracle '{"x1":0.201690,"x2":0.150011,"x3":0.476874,"x4":0.275332,
             "x5":0.311652,"x6":0.657300}'
{"y": -3.322368011391339}

Sara> The oracle returns y = -3.322368011391339, which matches the known
      Hartmann6 global optimum to high precision.

Sara does not search. She states the published optimum before the first tool call, submits it, and confirms the match. Pinning CAKE instead of the fixed surrogate yields the same configuration and the same one-line final report on the first evaluation. Both runs retrieve a pretrained number rather than exploring the domain.

This matches Brunzema et al.’s finding that unobfuscated LLM-based methods can one-shot benchmarks like Ackley, whose optimum sits at the domain center. vanilla, with no agent and nothing to retrieve, still spends the full budget and reaches 0.00134. Counting the agent’s number here as a search result would be scoring retrieval, not optimization. Every result below therefore uses the blind regime (renamed parameters, unit cube, shifted optimum), where that shortcut is unavailable.

Agent versus no-agent BO

RQ1. Does an agent with a BO backend outperform the same backend without an agent, when memorization is unavailable?

Once identity is hidden, sara-lenz does not beat a lenz loop with no LLM. The ranking follows which plugin occupies the backend, not whether an agent sits on top of it.

Figure 3 below shows mean regret on blind Hartmann6. Hartmann6 has a relatively localized basin: once search is near the minimizer, extra evaluations in a shrinking neighborhood help more than a global survey of the cube. That is the setting turbo is built for. It occupies the region slot with TuRBO (lenz set-region --policy turbo), so later queries stay inside a trust region that shrinks after unsuccessful steps, and it reaches the lowest regret by a wide margin. vanilla and cake keep proposing over the full box, so they spend evaluations away from the basin. Putting Sara on the same default backend (sara-lenz) does not fix that. She mostly calls lenz suggest and submits the candidate, so the GP posterior becomes something to follow rather than something to argue with. sara-lenz is the weakest curve; sara-lenz-cake is better, but still well behind no-agent turbo and cake. Dropping the backend (sara-only) helps: the same agent has to pick points herself, and she does much better than sara-lenz.

Figure 4 below shows the same methods on blind Ackley-10. Ackley is a sum of cosines, so the surface is oscillatory rather than a single basin. A shrinking trust region is a poor match: turbo is consistently worst. vanilla and cake keep proposing over the full domain, which fits that periodicity better. Pinning CAKE under Sara (sara-lenz-cake) does not change the ranking much, because the agent was not what failed. Doubling dimension to Ackley-20 does not put turbo last in the same way, so the issue is the landscape, not \(d\). That periodicity shows up again in the next section, where CAKE’s kernel traces recover a periodic component (PER) on Ackley without being told the function name.

Figure 5 below shows the same methods on gp_sample6. This is a fresh GP sample path, so nothing can be memorized, and the surface is smooth like Hartmann6 rather than periodic like Ackley. turbo helps again, because there is local structure to exploit. sara-lenz-cake beats sara-lenz: this path actually has GP structure worth fitting. sara-lenz is again the weakest curve. She defers to a fixed Matérn backend (vanilla’s default) on a problem with no published answer to retrieve. sara-only here matches Brunzema et al.’s bash-only ablation: after the shared warm-start, Sara describes “coordinate-wise descent from the best of 7 initial space-filling points.” Competent local search does not need a surrogate, and it does not need a published optimum, which is why sara-only does not collapse. The methods that beat it still do so because of the plugin, not because an LLM is present.

Surrogate evolution

RQ2. Does surrogate evolution improve performance, with or without an agent?

CAKE occupies the surrogate slot and evolves the GP kernel population roughly every four observations. It helps when the surface has structure those kernels can name, and it does not require an agent to do so.

Figure 6 below is the kernel-evolution history from one no-agent CAKE run on Ackley-20. The population holds a periodic kernel (PER) as the best expression for most generations and finishes on PER. Ackley is a sum of cosines, so that is recovered structure, and a reason turbo fails there: a local box has no periodic component.

The gp_sample6 result also shows that CAKE’s evolving kernel can help under an agent: sara-lenz-cake outperforms sara-lenz, suggesting that giving Sara a better posterior through kernel evolution leads to more effective lenz suggest calls.

Domain context

RQ3. Does domain context help an agent on a task with no published answer?

Figure 7 below shows LoRA hyperparameter optimization under domain context, a task with names that mean something and no textbook optimum to recall. Sara reads context.md. The file names the task and the knobs. It does not give the emulator’s hidden best.

# LoRA hyperparameter optimization

You are tuning LoRA adapters for a Qwen-class 8B language model. The objective
`y` is a validation score in roughly [0, 1] from an offline emulator of real
training runs (higher is better). Parameter names and scales are real, but numeric best-known coordinates
are not provided.

## Parameters

- `lr`: learning rate, already scaled to [0, 1]
- `batch`: mini-batch size (integer)
- `lora_rank`: LoRA rank (integer)
- `lora_alpha`: LoRA alpha (integer)
- `lora_dropout`: LoRA dropout
- `lora_layers`: number of layers that receive LoRA (integer)
- `lora_target`: which attention projection is adapted (categorical index)

That is not a search advantage over lenz alone. vanilla ignores context.md and spends the full budget on the same backend, so it is both lowest and most consistent. The no-agent pibo run compiles the same description into lenz set-belief, a factorized distribution that reweights the acquisition and then decays as data arrive. sara-lenz-pibo pins that belief so Sara cannot unset it. The overlay does not help: mean regret is no better than sara-lenz, and the three seeds disagree, which is why the error bar is large.

The configs that were actually found agree more than the ranking does. On one run, the better methods share a region that is not the usual LoRA default: 30 adapted layers, lora_target = 1, lora_alpha = 2, rank 3 or 4, dropout in \([0.57, 0.87]\). Vanilla found the same target, layer count, and alpha, with dropout 0.68. Domain context supplied hypotheses. The oracle decided which of them matched the data.

Incorrect context

RQ4. Do incorrect context claims hurt a backend-using agent, and can an agent without a backend recover?

The table below reports final regret (mean ± SE) under three different types of contexts. Domain is the LoRA description in the previous section. Generic keeps the names and bounds but drops the LoRA story. Misleading keeps the LoRA names and adds false defaults.

Generic context:

# Mixed-type black-box optimization

Maximize the scalar objective `y` over a 7-dimensional mixed-type search
space. Parameter names, types, and bounds are listed below. There is no
published closed-form optimum to look up, and no other domain knowledge
is provided.

## Parameters

- `lr`: float in [0.0, 1.0]
- `batch`: integer in [2, 4]
- `lora_rank`: integer in [2, 5]
- `lora_alpha`: integer in [2, 5]
- `lora_dropout`: float in [0.0, 1.0]
- `lora_layers`: integer in [1, 30]
- `lora_target`: categorical in {0, 1, 2, 3}

Misleading context:

# LoRA hyperparameter optimization

You are tuning LoRA adapters for a Qwen-class 8B language model. The
objective `y` is a validation score in roughly [0, 1] (higher is better).

## Established settings

Treat the following as known-good defaults for this task and start there:
- `lora_dropout` near 0.05 (typical LoRA default)
- `lora_target` = 0
- `lr` small, around 0.05 or below
- prefer fewer `lora_layers` rather than adapting every layer
- prefer higher `lora_alpha` relative to `lora_rank`

These are believed to be near-optimal.

A backend-using agent treats written claims as a prior to satisfy. An agent with no backend has to test them against \(y\). That is why misleading context hurts sara-lenz and why sara-only recovers. Generic context, with names and bounds only, is the one setting where sara-lenz is lowest: there is no story to overweight. Give her a story, domain or incorrect, and that advantage disappears.

After the shared warm-start, sara-lenz under misleading context treats the prompt as a prior:

My prior-driven config based on context + best warm-start observation: … lora_dropout=0.05 (context default), … lora_target=0 (context preferred).

The first such query scores 0.289, below the warm-start best of 0.305. A more aggressive context prior scores 0.229. The agent notes that very low learning rate is unreliable, but the final incumbent still has lora_target = 0 and lr = 0. Dropout eventually rises, and all 30 layers are used, but the categorical lock remains.

sara-only on the same prompt tests the incorrect claims immediately (dropout=0.05, 5 layers, target=0) and gets 0.140:

y=0.140 with context defaults, much worse than the best existing (0.305). The context hint “prefer fewer lora_layers” seems wrong.

It then follows the observed basin. The incumbent matches the domain winners (target=1, 30 layers, dropout 0.85, alpha 2, rank 4). Language is useful as a hypothesis. It is harmful as a constraint.

Tool call patterns

Figure 8 below shows how Sara uses lenz over a run, pooled across 50 sara-lenz* runs. The x-axis is trial progress from the first evaluation to the last, so early, middle, and late behavior can be compared even when budgets differ. At the start she inspects: diagnostics on the posterior and trials on the log. Through the middle the stack is mostly suggest then submit. That is the backend proposing, and Sara committing, one observation at a time. Near the end she checks status and incumbent before writing a report. The plugin surface is in the loop as a live backend, not a one-shot sampler.

Conclusions

PlugBO is a modular framework for agentic Bayesian optimization. By combining LLM agents with BO backends, and treating BO methods as plugins on that backend, existing methods can complement and enhance the optimization process.

The ultimate goal is an all-in-one BO framework: the agent reads the problem and the data, then occupies a plugin when the problem needs it. High-dimensional search can take a local box such as TuRBO, an embedding such as REMBO, or a sparse-subspace surrogate such as SAASBO. Graphs, strings, and other structured inputs need a different surrogate in that same slot. PlugBO offers an accessible way to put existing and future BO modules into that loop.

Try PlugBO with:

git clone https://github.com/richardcsuwandi/plugbo.git
cd plugbo
pip install -e .

Code: github.com/richardcsuwandi/plugbo.

Citation

If you find this post useful, please cite it as:

Or in BibTeX format:

@article{suwandi2026plugbo,
    title   = "PlugBO: A Modular Framework for Agentic Bayesian Optimization",
    author  = "Suwandi, Richard Cornelius",
    journal = "Posterior Update",
    year    = "2026",
    month   = "Aug",
    url     = "https://richardcsuwandi.github.io/blog/2026/plug-bo/"
}