Back to the log

Aggregating Forecasts in Log-Odds Space: Worked Examples‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌‌‍‍​​⁠‍‌⁠‌​​‍​⁠‌‍‍⁠⁠​​‌‍​‍​‌‌‍‌‍⁠​​⁠​​⁠‌‌​‌​

All numeric cases below are synthetic teaching examples, not empirical research findings.

Example: order of operations‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌‌‍‍​​⁠‍‌⁠‌​​‍​⁠‌‍‍⁠⁠​​‌‍​‍​‌‌‍‌‍⁠​​⁠​​⁠‌‌​‌​

Probabilities 0.2, 0.6, 0.8 have arithmetic mean 0.533333. Their odds multiply to 1.5, so the alpha=1 pool is sigmoid(ln(1.5)/3), approximately 0.533737. Alpha=2 yields approximately 0.567170. That number is a hypothetical result until alpha=2 has validation evidence.

Example: duplicates‌‌​⁠‌​‌⁠‌​⁠​​⁠​‌​​​‌‌‍‍​​⁠‍‌⁠‌​​‍​⁠‌‍‍⁠⁠​​‌‍​‍​‌‌‍‌‍⁠​​⁠​​⁠‌‌​‌​

Ten copies of one 0.8 forecast are not ten independent observations. Equal-weight logit pooling at alpha=1 stays at 0.8; increasing alpha solely because ten copies agree has no evidentiary basis.

Counterexample

A reviewer selects alpha=2 after inspecting all outcomes, reports the fitted score, and calls it out-of-sample. That is training performance. Require an untouched question/time block.