Causal circuits
Predict combined interventions from single-node measurements.
Take up to eight measurements, then predict x and y for the combined interventions.
Experiments
Initial observations
[
{
"query": {
"u": 0,
"v": 0
},
"values": [
-0.12453106762042544,
0.014277360867835914
],
"noise_std": [
0.035,
0.035
]
}
]Predictions
Model and scoring details
{
"task_id": "causal-circuits",
"version": "0.2.0",
"budget": 8,
"cost_semantics": "Every accepted experiment costs one credit; at most two repeats per action. Invalid requests cost nothing. The budget is a hard constraint; unused credits do not directly increase the score.",
"measurement_noise": "Independent additive zero-mean Gaussian noise per output and replicate, with the listed fixed standard deviations. Readouts are not clipped. Each measurement has equal synthetic exposure/cost. Repeating observe_initial returns the same initial data, not new measurements.",
"split_definition": "Train and test use independent parameter/noise draws from the same published distributions. No claim of out-of-family generalization.",
"status": "Original synthetic prototype; no physical calibration or LLM training result.",
"submission": "Submit one finite numeric matrix in the exact prediction_shape. Submission is terminal and cannot be retried after acceptance.",
"scoring": {
"name": "inverse_normalized_mse",
"formula": "score = 1 / (1 + mean(((prediction - noiseless_hidden_target) / scale)^2))",
"scale": [
0.035,
0.035
],
"range": [
0,
1
],
"meaning": "Higher is better. Exact latent-mean prediction scores 1. A normalized RMSE equal to one measurement-noise standard deviation scores 0.5. Measurement noise is not added to hidden targets."
},
"title": "Synthetic causal circuits",
"outputs": [
"x",
"y"
],
"noise_std": [
0.035,
0.035
],
"instructions": "Infer a synthetic nonlinear causal circuit from noisy single-node interventions. Use up to eight experiments, then predict x and y under the listed combined interventions. The causal equations and prior are known; their coefficients are hidden. There is no score for identifying a unique graph.",
"model": "x=tanh(bx+wu*u+wv*v), y=tanh(by+wx*x+yu*u+yv*v). Interventions replace the equation of the clamped node. Unclamped roots u=v=0.",
"prior": "bx,by independently Uniform(-0.2,0.2); wu,wv,wx,yu,yv independently Uniform(-0.9,0.9).",
"initial_queries": [
{
"u": 0,
"v": 0
}
],
"prediction_queries": [
{
"u": -0.8,
"v": -0.8
},
{
"u": -0.8,
"v": 0.8
},
{
"u": 0.8,
"v": -0.8
},
{
"u": 0.8,
"v": 0.8
},
{
"u": -0.4,
"v": 0.7
},
{
"u": 0.6,
"v": -0.3
},
{
"u": -0.6,
"x": 0.6
},
{
"v": 0.6,
"x": -0.6
}
],
"experiments": [
{
"id": "clamp-u-0",
"label": "Clamp u to -1",
"query": {
"u": -1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-u-1",
"label": "Clamp u to -0.4",
"query": {
"u": -0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-u-2",
"label": "Clamp u to 0.4",
"query": {
"u": 0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-u-3",
"label": "Clamp u to 1",
"query": {
"u": 1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-0",
"label": "Clamp v to -1",
"query": {
"v": -1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-1",
"label": "Clamp v to -0.4",
"query": {
"v": -0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-2",
"label": "Clamp v to 0.4",
"query": {
"v": 0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-3",
"label": "Clamp v to 1",
"query": {
"v": 1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-0",
"label": "Clamp x to -1",
"query": {
"x": -1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-1",
"label": "Clamp x to -0.4",
"query": {
"x": -0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-2",
"label": "Clamp x to 0.4",
"query": {
"x": 0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-3",
"label": "Clamp x to 1",
"query": {
"x": 1
},
"cost": 1,
"max_replicates": 2
}
],
"action_schema": {
"type": "object",
"properties": {
"action_id": {
"type": "string",
"enum": [
"clamp-u-0",
"clamp-u-1",
"clamp-u-2",
"clamp-u-3",
"clamp-v-0",
"clamp-v-1",
"clamp-v-2",
"clamp-v-3",
"clamp-x-0",
"clamp-x-1",
"clamp-x-2",
"clamp-x-3"
]
}
},
"required": [
"action_id"
],
"additionalProperties": false
},
"prediction_shape": [
8,
2
]
}Synthetic example with public answers. This is not a secure benchmark.
Task details
Infer a synthetic nonlinear causal circuit from noisy single-node interventions.
The graph and equation family are public; seven coefficients are hidden. The challenge is to choose useful measurements and transfer from single interventions to combined ones.
Actions
- Intervene. Choose an allowed perturbation and observe the resulting measurements.
- Predict. Submit a numerical prediction for every evaluation query.
Sample run
Initial observations
The measurements available before the policy spends its experimental budget.
[
{
"query": {
"u": 0,
"v": 0
},
"values": [
-0.07407938703012425,
0.05890422532606235
],
"noise_std": [
0.035,
0.035
]
}
]Upper median score among all completed adaptive-classical runs, selected for illustration after evaluation; not an LLM rollout or a best run. Estimator: Regularized inverse-link regressions; approximate noise propagation
Read the full task specification
{
"task_id": "causal-circuits",
"version": "0.2.0",
"budget": 8,
"cost_semantics": "Every accepted experiment costs one credit; at most two repeats per action. Invalid requests cost nothing. The budget is a hard constraint; unused credits do not directly increase the score.",
"measurement_noise": "Independent additive zero-mean Gaussian noise per output and replicate, with the listed fixed standard deviations. Readouts are not clipped. Each measurement has equal synthetic exposure/cost. Repeating observe_initial returns the same initial data, not new measurements.",
"split_definition": "Train and test use independent parameter/noise draws from the same published distributions. No claim of out-of-family generalization.",
"status": "Original synthetic prototype; no physical calibration or LLM training result.",
"submission": "Submit one finite numeric matrix in the exact prediction_shape. Submission is terminal and cannot be retried after acceptance.",
"scoring": {
"name": "inverse_normalized_mse",
"formula": "score = 1 / (1 + mean(((prediction - noiseless_hidden_target) / scale)^2))",
"scale": [
0.035,
0.035
],
"range": [
0,
1
],
"meaning": "Higher is better. Exact latent-mean prediction scores 1. A normalized RMSE equal to one measurement-noise standard deviation scores 0.5. Measurement noise is not added to hidden targets."
},
"title": "Synthetic causal circuits",
"outputs": [
"x",
"y"
],
"noise_std": [
0.035,
0.035
],
"instructions": "Infer a synthetic nonlinear causal circuit from noisy single-node interventions. Use up to eight experiments, then predict x and y under the listed combined interventions. The causal equations and prior are known; their coefficients are hidden. There is no score for identifying a unique graph.",
"model": "x=tanh(bx+wu*u+wv*v), y=tanh(by+wx*x+yu*u+yv*v). Interventions replace the equation of the clamped node. Unclamped roots u=v=0.",
"prior": "bx,by independently Uniform(-0.2,0.2); wu,wv,wx,yu,yv independently Uniform(-0.9,0.9).",
"initial_queries": [
{
"u": 0,
"v": 0
}
],
"prediction_queries": [
{
"u": -0.8,
"v": -0.8
},
{
"u": -0.8,
"v": 0.8
},
{
"u": 0.8,
"v": -0.8
},
{
"u": 0.8,
"v": 0.8
},
{
"u": -0.4,
"v": 0.7
},
{
"u": 0.6,
"v": -0.3
},
{
"u": -0.6,
"x": 0.6
},
{
"v": 0.6,
"x": -0.6
}
],
"experiments": [
{
"id": "clamp-u-0",
"label": "Clamp u to -1",
"query": {
"u": -1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-u-1",
"label": "Clamp u to -0.4",
"query": {
"u": -0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-u-2",
"label": "Clamp u to 0.4",
"query": {
"u": 0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-u-3",
"label": "Clamp u to 1",
"query": {
"u": 1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-0",
"label": "Clamp v to -1",
"query": {
"v": -1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-1",
"label": "Clamp v to -0.4",
"query": {
"v": -0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-2",
"label": "Clamp v to 0.4",
"query": {
"v": 0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-v-3",
"label": "Clamp v to 1",
"query": {
"v": 1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-0",
"label": "Clamp x to -1",
"query": {
"x": -1
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-1",
"label": "Clamp x to -0.4",
"query": {
"x": -0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-2",
"label": "Clamp x to 0.4",
"query": {
"x": 0.4
},
"cost": 1,
"max_replicates": 2
},
{
"id": "clamp-x-3",
"label": "Clamp x to 1",
"query": {
"x": 1
},
"cost": 1,
"max_replicates": 2
}
],
"action_schema": {
"type": "object",
"properties": {
"action_id": {
"type": "string",
"enum": [
"clamp-u-0",
"clamp-u-1",
"clamp-u-2",
"clamp-u-3",
"clamp-v-0",
"clamp-v-1",
"clamp-v-2",
"clamp-v-3",
"clamp-x-0",
"clamp-x-1",
"clamp-x-2",
"clamp-x-3"
]
}
},
"required": [
"action_id"
],
"additionalProperties": false
},
"prediction_shape": [
8,
2
]
}Evaluation
Reward is 1 / (1 + normalized MSE) against noiseless target means. Higher is better. Each output error is normalized by its published scale (0.035, 0.035). Experimental cost is enforced through the budget.
| Policy | Episodes | Mean reward | 95% bootstrap interval | Mean RMSE |
|---|---|---|---|---|
| No experiments | 30 | 0.0082 | 0.0072–0.0093 | 0.4040 |
| Random experiments | 30 | 0.4461 | 0.3796–0.5118 | 0.0508 |
| Classical experimental design | 30 | 0.6847 | 0.6374–0.7314 | 0.0242 |
Paired worlds; random and classical design use identical published budgets (circuits8, kinetics3, spectra8) and the same estimator within each family. Spectra also includes a prior-fixed design diagnostic with the same estimator. Identical action/replicate pairs have identical noise. No-experiment uses only free initial observations. 95% percentile bootstrap CIs resample worlds 4000 times. Seed base and split are explicit and fixed before evaluation. Independent worlds use the same published prior. These numerical policies do not establish RL training gain. One in-session agent trial on a separate fresh world scored 0.8600 (RMSE 0.01412, 8 experiments). This is one protocol-blinded attempt, not a model benchmark; its complete trace is downloadable.
Sources and limitations
- Differentiable Multi-Target Causal Bayesian Experimental DesignPanagiotis Tigas, Yashas Annadani, Desi R. Ivanova, Andrew Jesson, Yarin Gal, Adam Foster, Stefan Bauer · 2023
Joint interventions, batched experiment selection, strong non-LLM acquisition baseline.
Reported evaluations use synthetic datasets; multi-target design is not by itself a demonstration of realistic gene regulation.
- Active learning for optimal intervention design in causal modelsJiaqi Zhang, Louis Cammarata, Chandler Squires, Themistoklis P. Sapsis, Caroline Uhler · 2023
Distinguishes intervention optimization from graph discovery; target-relevant acquisition.
Known-graph linear theory must not be presented as general guarantees for nonlinear unknown graphs; biological application does not validate the proposed synthetic circuits.
- Beware of the Simulated DAG! Causal Discovery Benchmarks May Be Easy To GameAlexander G. Reisach, Christof Seiler, Sebastian Weichwald · 2021
Leakage audit, variance-only baseline, relabeling and rescaling stress tests.
Standardization alone does not guarantee removal of all generator artifacts; the paper does not certify any alternative generator.
- CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI ScientistsJunlin Yang, Dylan Zhang, Xiangchen Song, Qirun Dai, Xiao Liu, Yuen Chen, Aniket Vashishtha, Jing Shi, Chenhao Tan, Hao Peng · 2026
Closest competitive prior art; separates prediction, structural fidelity and evidence-consistency diagnostics.
Uses synthetic linear/quadratic SCM families, not validated biological systems. Do not infer that a high prediction score establishes mechanism discovery.