Data specification / 0.2.0
Scientific environments
Budgeted experimental design → predictions on unseen conditions.
Unit of delivery. A runnable synthetic task with a defined action space, observation process, budget, terminal submission schema, and numerical evaluator. Three task implementations form the initial collection; the task tree describes an expansion plan.
Purpose. Measure how well an agent chooses experiments and converts observations into predictions. These prototypes do not establish performance on real laboratory systems.
| Current task schemas | Causal circuits: 8 credits, 8 × 2 prediction matrix; Reaction kinetics: 3 credits, 15 × 1 prediction matrix; Adaptive spectroscopy: 8 credits, 12 × 1 prediction matrix. Rows follow the published query order; columns follow the published output order. |
|---|---|
| Task identity | Versioned task ID, scientific objective, instructions, supported actions, and explicit numerical output schema. |
| Public episode | Initial observations, allowed experimental conditions, finite budget, and the queries that must be predicted. |
| Experiment record | Ordered action, returned observation, charged cost, and remaining budget for every accepted experiment. |
| Action protocol | Each accepted assay costs one credit, with at most two replicates per action. Invalid actions cost nothing. Accepted final submission ends the episode; it cannot be retried. |
| Final submission | Numerical predictions in the requested shape, validated before evaluation. The recorded sample includes the submitted values. |
| Evaluation record | Terminal score and error diagnostics under the versioned task’s scoring rule, with the baseline protocol and run counts. |
| Provenance | Source-to-design mapping, implementation version, generation configuration, run metadata, and an explicit statement of limitations. |
Agent & evaluator visibility
The agent receives its task specification and experimental observations. Hidden parameters and reference outcomes belong to the evaluator. Public examples deliberately disclose demonstration data; a buyer’s evaluation instances require a separate protected deployment.
Scoring & baselines
Reward = 1 / (1 + normalized MSE) against noiseless targets, using each output’s published measurement-noise scale. Higher is better. Reports include raw RMSE, run counts, and 95% bootstrap intervals. Classical reference runs do not establish LLM training benefit.
Quality evidence
Inspect reproducible traces, numerical checks, input validation, budget enforcement, reference policies, and source mappings. The evidence ledger distinguishes verified checks from open limitations.
Scientific scope
Primary papers motivate experimental structures and modeling choices. Citations do not validate this implementation. Independent domain review, real-world transfer, and buyer-specific acceptance remain separate questions.