Simulation and experiments

Simulation profiles

Profiles support organic, ads, sponsored_agent, custom_agent, mcp_rag, and workflow modes. A profile declares component, version, assumptions, parameters and approval status.

Use simulation for questions such as “What may happen if we improve citation coverage, change a campaign message, add structured data, switch model/provider, or expose a new MCP/RAG source?” Results include baseline, expected change, confidence, prediction interval, cost and latency assumptions.

Simulation is never stored as an observation. OpenAI, Claude, Perplexity, Gemini and custom components can be represented without making an unsupported claim that a closed advertising or agent-placement API is available.

Observed evidence, simulated prediction and experiment outcome remain separate datasets. Calibration updates a new profile version; it never edits the earlier prediction after the result is known.

Scenarios

A scenario binds project, target, profile, hypothesis, parameters and success criteria. Keep assumptions explicit and version scenarios when changing material inputs.

Experiments

Experiments compare control and candidate variants using an observed metric. Configure hypothesis, target, metric, design, sample plan, guardrails and stop criteria. Results include observed values, confidence and winner; stop or cancel prevents future collection but preserves history.

Calibration

Calibration compares simulated predictions with later observed outcomes. Large prediction error should lower confidence or trigger model/profile revision. Never silently tune historical simulation outputs after the outcome is known.