Elicit a forecast panel

A panel can combine information that no individual has alone. It can also amplify shared errors, social pressure, or a common source. A controlled elicitation process preserves initial judgments and reveals why respondents revise.

Collect private first estimates

Give every participant the same question version, evidence cutoff, and resolution rules. Collect a probability, strongest supporting reason, strongest opposing reason, main uncertainty, and observation that would cause an update. Avoid showing a prominent aggregate or senior participant’s estimate before this first round.

The Elicitation record preserves respondent identity, human/model metadata, an optional probability, and supporting/opposing reasons. None or JSON null means no response; it is never replaced automatically with 0.5 or the group mean.

The Respondent type distinguishes a human pseudonym from model family/version, prompt reference, and corpus reference. These metadata are supplied by the caller; they do not verify that a person participated or that a model was independently trained.

Give controlled feedback

summarize_panel returns invited, responded and missing counts, median, minimum and maximum. Duplicate respondent IDs and a panel with no probabilities are rejected. Its range is disagreement, not a confidence interval.

Present useful arguments and evidence origins alongside the distribution. Ask whether another participant supplied a new fact, corrected an interpretation, or merely expressed stronger conviction. Preserve dissent when it rests on an unresolved assumption.

A reasonable small exercise might use two predeclared rounds. More rounds should justify their cost rather than treating convergence as the goal. Compare initial and final aggregates using the same rule.

Independence and model panels

Repeated calls to one model can explore variability and alternative decompositions. They are not independent human experts, and their similar answers do not establish calibrated certainty. Shared retrieval corpora and prompt context can make errors strongly dependent.

Keep respondent-level records outside the summary. They are necessary for later analysis of influence, attrition, and performance. The library summarizes supplied responses; it neither contacts respondents nor fabricates interviews.

Use equal weights unless relevant held-out performance supports another policy. Familiarity, confidence, and prestige are not interchangeable with measured forecasting skill. Continue with aggregation once the elicitation record is clear.