# Scientific experimental-design environments: research and task-design audit

Prepared 17 October 2026. Public literature scope: 1 January 2023 through 17 October 2026, with selected foundational work.

## Decision

The defensible product is a compact suite for **choosing informative experiments and predicting consequences under a limited budget**. The first three demonstrations should remain explicitly synthetic: a known-graph causal response model, competing first-order reaction pathways, and selective spectral measurement. They are useful software and evaluation prototypes. They are not yet evidence of frontier-model difficulty, RL training gains, biological realism, or physical transfer.

The accompanying catalogue contains **50 task designs in six families**. Three designs correspond to locally implemented version 0.2.0 prototype demos; 47 are proposals. A task changes the scientific ambiguity, useful experiment, observation process, final decision or transfer problem. Changing names, random seeds or a coefficient alone does not count. Whether every proposed distinction merits separate commercial acceptance still needs empirical review and agreement with the buyer.

The core intellectual loop is already established. BoxingGym evaluates interactive experiment design and model discovery; SciGym uses systems-biology simulations; CausaLab combines hidden causal models, interventions and held-out predictions; NewtonBench probes hidden laws in simulated systems. A new offer should therefore compete on inspectability, useful task distinctions, reliable grading, evidence of learning opportunity and integration quality—not on being the first scientific discovery environment. [BoxingGym](https://arxiv.org/abs/2501.01540), [SciGym](https://arxiv.org/abs/2507.02083), [CausaLab](https://arxiv.org/abs/2605.26029), [NewtonBench](https://arxiv.org/abs/2510.07172).

## Research coverage and limits

I used Exa to screen **155 search-result slots across 17 search angles**, then issued **34 primary-page fetch requests** and curated **29 distinct primary research sources**. The 155 figure includes duplicate papers, DOI/abstract/PDF versions and irrelevant candidates. It is not 155 independently validated studies. Raw search URLs deduplicate to 154 exact URLs, but several are versions of the same paper; the curated 29-paper register is the meaningful selected-paper count.

Search angles covered causal intervention design, kinetic discrimination, spectroscopy, scientific-agent benchmarks, amortized design, biological dry labs, benchmark shortcuts, enzyme identifiability, adaptive acquisition, closed-loop laboratories, dynamical identification, circuit motifs, calibration, cost/transfer design, scoring rules, foundational model discrimination and practical identifiability. Source records state whether verification reached an abstract or selected methods sections. No claim is made that every complete paper and supplement was read.

The highest-confidence mathematical anchors were checked at the original publisher, conference proceedings or author-hosted paper. Preprints remain labeled. Search-engine dates were not treated as definitive publication dates: for example, Box and Hill is a 1967 paper even though one publisher page has a later online-publication date. Schroeder and colleagues' current fetched title differs from older indexing. OpenReview initially returned a browser check for SciGym; the arXiv abstract and NeurIPS proceedings record supplied the verification route.

The public evidence register is `sources.json`; the task inventory is `catalogue.json`. Raw fetched/search snapshots are internal research notes and should not be included in public downloads. No C5R source code, private measurements, prompts, trajectories or unpublished task assets are incorporated. Prior C5R analysis supplied a general lesson—separate tasks from seeds and scientific difficulty from execution difficulty—but the specifications here are original public-literature-informed adaptations.

## What the literature actually supports

### Sequential experimental design is a credible learning problem

Deep Adaptive Design learns a policy that maps accumulated experiments and observations to the next design. It is a clear precedent for amortizing expensive design calculations across many episodes. It does not imply that an LLM trained on this particular suite will improve, or that online Bayesian design is an easy baseline. A fair suite needs both simple acquisition policies and a credible statistical planner. [Foster et al., 2021](https://proceedings.mlr.press/v139/foster21a.html).

There are meaningful differences among independent reset experiments, batch experiments and stateful dynamical experiments. Multi-target causal design selects sets of interventions jointly; particle-based design addresses non-exchangeable dynamical data. A system that can only query independent scalar responses should not be marketed as covering all of these. [Tigas et al., 2023](https://proceedings.mlr.press/v202/tigas23a.html), [Iqbal et al., 2024](https://arxiv.org/abs/2402.07868).

### Prediction, identification and optimization are different objectives

A good intervention may reduce uncertainty about the graph, improve a prediction, or achieve a desired output. These are different utilities and can prefer different experiments. Causal active learning with a known graph provides one concrete intervention-optimization formulation; adaptive causal discovery targets graph recovery and stopping. Neither is automatically the right objective for a held-out-prediction game. [Zhang et al., 2023](https://www.nature.com/articles/s42256-023-00719-0), [Elahi et al., 2024](https://proceedings.mlr.press/v235/elahi24a.html).

CausaLab directly demonstrates why prediction and mechanism fidelity should be reported separately. Our first causal prototype has a **public fixed graph and function class** and hidden coefficients. It evaluates causal response identification and transfer to compound manipulations. It does not discover an unknown graph, and the variables are not validated genes. Unknown-structure tasks remain proposed extensions. [CausaLab, 2026](https://arxiv.org/abs/2605.26029).

### Mechanism discrimination needs deliberately separating experiments

The foundation of sequential model discrimination is to select conditions where rival models predict different outcomes relative to their uncertainty. The kinetic demonstration uses this principle in a tiny candidate set, with species and time as design variables. A candidate model winning within that set is not evidence that an arbitrary real reaction follows it. [Box and Hill, 1967](https://doi.org/10.1080/00401706.1967.10490441).

Exact hidden-parameter recovery is often the wrong grading target. Different parameters can make indistinguishable observations, and finite noisy data can make an identifiable model practically unidentifiable. Score useful predictions or justified equivalence classes unless parameter identifiability has been demonstrated for the allowed actions. [Raue et al., 2009](https://academic.oup.com/bioinformatics/article/25/15/1923/213246).

The kinetic catalogue therefore separates network topology, reaction order, saturation, inhibition, reversibility, induction, deactivation, temperature competition, population heterogeneity and transport limitation. These are proposed experiments, not 10 claims that the cited papers already validated our exact tasks. Enzyme-inhibition design and decision-relevant biochemical measurement selection offer methodological grounding for some of these branches. [Schorning et al., 2017](https://arxiv.org/abs/1709.04952), [Isenberg et al., 2024](https://www.nature.com/articles/s41598-024-65196-w).

### Spectroscopy offers a strong small-scale example

Adaptive spectroscopy is more than selecting arbitrary points from a curve. Useful decisions may concern model class, peak count, mixture composition, background, calibration, exposure time or measurement modality. Nabika and colleagues explicitly combine physical-model selection and parameter estimation; their spectral example uses Gaussian-shaped peaks, background and Poisson counts. Ito and colleagues select measurement points using prior spectral information. These are good anchors for a small inverse-measurement environment. [Nabika et al., 2024](https://www.nature.com/articles/s41598-024-54329-w), [Ito et al., 2025](https://iopscience.iop.org/article/10.1088/2632-2153/add0f6).

A fixed linear-Gaussian spectral model has posterior covariance independent of measured values. Greedy covariance reduction can therefore produce a useful predetermined design without demonstrating outcome-adaptive decisions. To demonstrate outcome adaptivity, the candidate model, noise or state must make posterior uncertainty depend on observed values; compare resulting action sequences on matched initial settings with different readouts. A library of competing spectral models is one principled extension. This is a mathematical design observation, not a claim that all fixed-design methods are inferior.

The demo's additive Gaussian readout noise is an explicit approximation. It must not be described as the cited paper's photon-counting model. Time-delay/Fourier acquisition, exposure allocation and irreversible bleaching are separate proposed extensions, not features already implemented. [Schroeder et al., 2025](https://arxiv.org/abs/2505.14364).

### Scientific nuisance variables are real task content

Calibration is part of inference. Nonlinear saturation, heteroscedastic noise and uncertainty propagation can change the best experiment; they should not be added as arbitrary hidden penalties. The proposed measurement-diagnosis family requires informative controls for each nuisance. If a failure cannot be revealed by any allowed control, it is a grader trap rather than a defensible inference challenge. [Helleckes et al., 2022](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009223).

Dynamic excitation, partial observation and sparse equation discovery support another coherent family, but they require time-series holdouts rather than random rows from the same trajectory. Input-history dependence, measurement choice and process resets must be visible in the task contract. [Xie and Bemporad, 2025](https://arxiv.org/abs/2506.21754), [Fasel et al., 2022](https://pmc.ncbi.nlm.nih.gov/articles/PMC9006119/).

### Transfer and resources can add real distinctions

The final six catalogue tasks change the terminal decision or resource structure: confirm a noisy winner, calibrate a cheap proxy, transfer from accessible systems to an unmeasured target, map a trade-off, pay setup costs, or select a batch before feedback returns. They are more than alternate seeds because different actions can become optimal.

Cost-informed reaction optimization explicitly accounts for inventory-dependent costs. Multi-objective catalysis research supports Pareto trade-offs. Closed-loop transfer shows why optimization and reusable scientific insight should be assessed separately. Those papers contain real experimental or retrospective evidence for their own systems; their results are not inherited by this synthetic catalogue. [Schoepfer et al., 2024](https://pubs.rsc.org/en/content/articlehtml/2024/dd/d4dd00225c), [Bennett et al., 2024](https://www.nature.com/articles/s44286-024-00033-5), [Angello et al., 2024](https://www.nature.com/articles/s41586-024-07892-1).

AlphaFlow provides a precedent for RL in a specialized multistep physical platform. It supports the relevance of order-dependent scientific decisions, not a claim that three low-cost simulators reproduce an autonomous laboratory. [Volk et al., 2023](https://www.nature.com/articles/s41467-023-37139-y).

## Three prototype contracts to preserve

| Prototype | Hidden quantity | Scientific decision | Valid interpretation | Invalid interpretation |
|---|---|---|---|---|
| Causal response | Coefficients of a known nonlinear graph | Which single intervention informs compound-intervention prediction | Synthetic causal response identification | Unknown-GRN discovery or realistic gene perturbation |
| Kinetic pathways | Serial/parallel candidate and rate constants | Which species/time separates future predictions | Candidate-mechanism discrimination in a toy first-order system | Real reaction validation or arbitrary mechanism discovery |
| Selective spectra | Mixture/background and, in the revised design, candidate spectral library | Which wavelength reduces predictive uncertainty | Synthetic inverse measurement and model selection | Chemical identity validation or calibrated photon spectroscopy |

Prediction-query conditions may be public while the correct values remain private. This is a legitimate transductive experimental-design setting: the agent chooses measurements for a declared prediction target. It should not be described as the agent never seeing what it will be asked to predict. Query-disjointness must be checked against every initial and purchasable observation.

The final scientific review covered core version 0.2.0 and records exact file hashes in `scientific_audit.json`. Budgets are eight causal measurements, three kinetic assays and eight spectral measurements, with at most two executions of an action. Prediction conditions are disjoint from all initial and purchasable queries. Kinetic targets span early, middle and late times; spectra use nine candidate shift/broadening libraries. Do not copy version 0.1 scores into these revised task cards.

Independent checks compared 200 random kinetic cases against their mass-action differential equations (maximum absolute derivative residual 1.7e-11), verified causal intervention replacement, and checked full-panel spectral design rank across all nine libraries. These are mathematical/software checks, not physical validation or proof that eight measurements identify every instance. No blocking formula bug was found. The complete-state causal score includes two public clamped-x entries among 16 outputs; this denominator caveat is explicitly retained rather than hidden.

The bounded score is inverse noise-normalized MSE: a normalized RMSE of one measurement-noise standard deviation scores 0.5. A stricter normalization gives precision a clearer meaning; it does not itself add scientific complexity. Raw RMSE, no-experiment baselines and target-agent evaluations remain essential. Independent same-prior draws and known concentration scaling do not establish out-of-family generalization.

## Catalogue tree and why the branches differ

| Family | Count | Distinctions |
|---|---:|---|
| Causal intervention circuits | 10 | Known-graph coefficients; common cause; collider; joint logic; cancelling paths; persistence; pulse generation; adaptation mechanism; contextual sign; transport |
| Kinetic mechanism discrimination | 10 | Serial/parallel topology; saturation; inhibition; reversibility; reaction order; induction; deactivation; temperature competition; heterogeneity; transport limitation |
| Adaptive spectroscopy/inverse measurement | 10 | Mixture/background; peak count; amplitude/width; broad feature/baseline; axis drift; nonlinearity; exposure allocation; bleaching; modality; Fourier delays |
| Dynamical identification | 8 | Delay; damping; hysteresis; nonlinear force; compartments; stochastic switching; bistability; hidden loss |
| Calibration/measurement diagnosis | 6 | Valid range; unequal noise; matrix effects; temporal drift; specificity; carryover |
| Experiment allocation/transfer | 6 | Replication; fidelity; proxy transfer; Pareto objectives; setup costs; delayed batches |

The circuit motifs are scientifically motivated, but their behavior depends on parameter and input regimes. A coherent feed-forward graph does not automatically produce a useful persistence task, and not every feedback circuit adapts. Generator validation must establish the intended behavior before a task is counted. [Mangan and Alon, 2003](https://pmc.ncbi.nlm.nih.gov/articles/PMC218699/), [Ma et al., 2009](https://pmc.ncbi.nlm.nih.gov/articles/PMC3068210/).

Each JSON task specifies a hypothesis, allowed actions, observations, proposed budget, submission, reward, baselines, holdout, distinctness argument, sources and acceptance gate. Numerical budgets for unimplemented tasks are design proposals, not literature findings or validated difficulty settings.

## Quality standard for a buyer-facing pilot

1. **A useful inference problem.** Verify that plausible hidden worlds can disagree on target predictions and that allowed experiments can reduce the disagreement. If all good policies fail, test identifiability before calling the task difficult.
2. **An observation contract.** Publish actions, costs, readout format, reset semantics, noise assumptions, prior family and prediction-query inputs. Keep the episode's hidden parameters, noise keys and target values outside student access.
3. **Comparable baselines.** Use the same estimator for random, fixed and adaptive acquisition whenever possible. Equalize total resource cost and define free initial data. Include no-experiment, simple heuristic, statistical-design and privileged-truth checks.
4. **A scientific grading target.** Report raw RMSE or other interpretable loss in addition to a bounded game score. A normalized score must state its scale. A monotone transformation of point error is not a proper probabilistic scoring rule.
5. **Honest uncertainty evaluation.** If agents submit distributions, use an appropriate proper score and report calibration. Avoid calling an arbitrary clipped composite reward strictly proper. [Gneiting and Raftery, 2007](https://doi.org/10.1198/016214506000001437).
6. **Independent holdouts.** Separate new noise draws, new parameters, new input/intervention combinations, new mechanism classes and changed model misspecification. These support different claims. An independent seed from the same prior is not distribution shift.
7. **Shortcut audits.** Test prior-only, variance-only, fixed experimental schedule, trivial asymptotic prediction and data-order heuristics. Procedural generation can accidentally reveal causal order through variance or covariance. [Reisach et al., 2021](https://arxiv.org/abs/2102.13647).
8. **A reproducible package.** Pin task version, dependencies, test definitions, benchmark seeds, provenance and result schema. Record failed runs as well as successful ones. Inspect actual agent-visible artifacts for private-state leakage.
9. **An opportunity to learn.** Establish a gap among feasible oracle, credible statistical baseline and intended model agent before committing to RL training. A random policy using a strong estimator near ceiling is not evidence of frontier difficulty.
10. **External acceptance.** A commissioned pilot should agree on what counts as a distinct task, the target agent/interface, minimum evidence, ownership/licensing and delivery criteria. The literature cannot supply the buyer's acceptance threshold.

## Claims audit

| Claim | Evidence status | Defensible wording |
|---|---|---|
| This is a valid sequential experimental-design formulation | Supported as a formulation; implementation must be checked | Agents spend a budget on experiments and predict withheld outcomes |
| The first causal demo discovers hidden networks | Unsupported by a public fixed-graph implementation | It identifies hidden response coefficients in a known causal model |
| The biology is realistic | Unsupported | Synthetic circuit variables have no asserted biological identity |
| Three synthetic mathematical prototypes exist | Locally implemented v0.2.0; independent equation/contract checks passed | Further end-to-end test evidence belongs to the coordinator; no physical claim |
| There are 50 completed environments | False for this package | There are 50 specified designs, including three implemented synthetic prototypes and 47 proposals |
| Random seeds make tasks scientifically diverse | Unsupported | Seeds vary instances; diversity comes from different experimental problems |
| Adaptive experiments are better | Must be measured per task and budget | Report matched acquisition-policy results and exceptions |
| A known linear model demonstrates outcome adaptivity | Not necessarily | Its covariance-based schedule may be predetermined |
| High prediction accuracy proves correct mechanisms | Unsupported | Prediction and mechanism diagnostics are separate |
| Train/test seeds establish OOD generalization | Unsupported | They establish independent instances from the stated prior |
| Source papers validate our simulator | Unsupported | Sources motivate the models; our adaptation needs independent tests |
| Simulator success transfers to wet labs | Unmeasured | Physical transfer is a future validation question |
| The package improves an LLM through RL | Unmeasured | No training or learning-curve claim until a controlled training study is run |
| The suite is commercially differentiated | A proposal, not established fact | Differentiation must be shown against existing benchmarks and buyer requirements |

## Recommended sequence after the three demos

First, finish the current scientific-code audit and benchmark simple baselines at interpretable precision. Preserve any finding that the prototypes are easy. Second, build a small additional subset whose scientific distinction changes acquisition policy: joint causal logic, kinetic reversibility, spectral model-count ambiguity, calibration drift and delayed batches are reasonable candidates. Third, evaluate the buyer's actual agent and agree a paid pilot acceptance rubric before implementing the remaining catalogue.

A useful first paid package is a small number of accepted task templates with many reproducible instances, reference policies and trace-level failure analysis. Expansion to 50 should be earned by accepted distinctions and validation, not achieved by multiplying seeds. Existing benchmarks already illustrate both task variants and broader domain coverage. [DiscoveryWorld](https://proceedings.neurips.cc/paper_files/paper/2024/hash/13836f251823945316ae067350a5c366-Abstract-Datasets_and_Benchmarks_Track.html), [Gravity-Bench](https://proceedings.mlr.press/v267/koblischke25a.html).

## Evidence register

The full source-by-source claim, intended use, limitation, review depth and bibliographic metadata are in `sources.json`. The list below is a navigation index, not a claim that all sources support every task.

- **dad2021** — [Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design](https://proceedings.mlr.press/v139/foster21a.html) (2021; peer reviewed).
- **tigas2023** — [Differentiable Multi-Target Causal Bayesian Experimental Design](https://proceedings.mlr.press/v202/tigas23a.html) (2023; peer reviewed).
- **elahi2024** — [Adaptive Online Experimental Design for Causal Discovery](https://proceedings.mlr.press/v235/elahi24a.html) (2024; peer reviewed).
- **zhang2023** — [Active learning for optimal intervention design in causal models](https://www.nature.com/articles/s42256-023-00719-0) (2023; peer reviewed).
- **reisach2021** — [Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To Game](https://arxiv.org/abs/2102.13647) (2021; peer reviewed).
- **boxinggym2025** — [BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discovery](https://arxiv.org/abs/2501.01540) (2025; preprint).
- **causalab2026** — [CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists](https://arxiv.org/abs/2605.26029) (2026; preprint).
- **newtonbench2026** — [NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents](https://arxiv.org/abs/2510.07172) (2026; conference paper).
- **scigym2025** — [Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab](https://arxiv.org/abs/2507.02083) (2025; peer reviewed).
- **nabika2024** — [Bayesian active learning with model selection for spectral experiments](https://www.nature.com/articles/s41598-024-54329-w) (2024; peer reviewed).
- **ito2025** — [Optimal spectroscopic measurement design: Bayesian framework for rational data acquisition](https://iopscience.iop.org/article/10.1088/2632-2153/add0f6) (2025; peer reviewed).
- **schroeder2025** — [Information-optimal measurement: From fixed sampling protocols to adaptive spectroscopy](https://arxiv.org/abs/2505.14364) (2025; preprint).
- **isenberg2024** — [Identifying Bayesian optimal experiments for uncertain biochemical pathway models](https://www.nature.com/articles/s41598-024-65196-w) (2024; peer reviewed).
- **schorning2017** — [Optimal designs for enzyme inhibition kinetic models](https://arxiv.org/abs/1709.04952) (2017; preprint).
- **mangan2003** — [Structure and function of the feed-forward loop network motif](https://pmc.ncbi.nlm.nih.gov/articles/PMC218699/) (2003; peer reviewed).
- **ma2009** — [Defining Network Topologies that Can Achieve Biochemical Adaptation](https://pmc.ncbi.nlm.nih.gov/articles/PMC3068210/) (2009; peer reviewed).
- **fasel2022** — [Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control](https://pmc.ncbi.nlm.nih.gov/articles/PMC9006119/) (2022; peer reviewed).
- **xie2025** — [Online design of experiments by active learning for nonlinear system identification](https://arxiv.org/abs/2506.21754) (2025; preprint).
- **helleckes2022** — [Bayesian calibration, process modeling and uncertainty quantification in biotechnology](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1009223) (2022; peer reviewed).
- **schoepfer2024** — [Cost-informed Bayesian reaction optimization](https://pubs.rsc.org/en/content/articlehtml/2024/dd/d4dd00225c) (2024; peer reviewed).
- **angello2024** — [Closed-loop transfer enables artificial intelligence to yield chemical knowledge](https://www.nature.com/articles/s41586-024-07892-1) (2024; peer reviewed).
- **volk2023** — [AlphaFlow: autonomous discovery and optimization of multi-step chemistry using a self-driven fluidic lab guided by reinforcement learning](https://www.nature.com/articles/s41467-023-37139-y) (2023; peer reviewed).
- **bennett2024** — [Autonomous reaction Pareto-front mapping with a self-driving catalysis laboratory](https://www.nature.com/articles/s44286-024-00033-5) (2024; peer reviewed).
- **gravity2025** — [Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents](https://proceedings.mlr.press/v267/koblischke25a.html) (2025; peer reviewed).
- **iqbal2024** — [Nesting Particle Filters for Experimental Design in Dynamical Systems](https://arxiv.org/abs/2402.07868) (2024; peer reviewed).
- **discoveryworld2024** — [DiscoveryWorld: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents](https://proceedings.neurips.cc/paper_files/paper/2024/hash/13836f251823945316ae067350a5c366-Abstract-Datasets_and_Benchmarks_Track.html) (2024; peer reviewed).
- **gneiting2007** — [Strictly Proper Scoring Rules, Prediction, and Estimation](https://doi.org/10.1198/016214506000001437) (2007; peer reviewed).
- **boxhill1967** — [Discrimination Among Mechanistic Models](https://doi.org/10.1080/00401706.1967.10490441) (1967; peer reviewed).
- **raue2009** — [Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood](https://academic.oup.com/bioinformatics/article/25/15/1923/213246) (2009; peer reviewed).