validate_holdout

Validate declared question, event-cluster and time separation.

Contract

Training outcomes must be available by cutoff; holdout forecasts must follow it. Questions are unique across both sets and event clusters cannot cross the boundary. Hidden model-training contamination cannot be detected.

Fields

This operation needs no existing question or forecast. Use --memory for a calculation-only session.

Request

Send this object on one line. It is expanded below for readability.

{
  "version": 1,
  "id": "example",
  "command": {
    "op": "validate_holdout",
    "training": [
      {
        "question_id": "a",
        "cluster_id": "a",
        "forecast_at": 1,
        "resolved_at": 5,
        "model": 0.5,
        "baseline": 0.5,
        "outcome": true
      },
      {
        "question_id": "b",
        "cluster_id": "b",
        "forecast_at": 1,
        "resolved_at": 5,
        "model": 0.5,
        "baseline": 0.5,
        "outcome": false
      }
    ],
    "holdout": [
      {
        "question_id": "held-a",
        "cluster_id": "held-a",
        "forecast_at": 11,
        "resolved_at": 20,
        "model": 0.5,
        "baseline": 0.5,
        "outcome": true
      },
      {
        "question_id": "held-b",
        "cluster_id": "held-b",
        "forecast_at": 11,
        "resolved_at": 20,
        "model": 0.5,
        "baseline": 0.5,
        "outcome": false
      }
    ],
    "cutoff": 10
  }
}

Response

This response is generated by executing the example against the current binary. Commit timestamps, when present, are shown as <runtime UTC seconds>; the actual protocol returns integer UTC seconds.

{
  "version": 1,
  "id": "example",
  "result": {
    "holdout_questions": 2,
    "training_questions": 2,
    "valid": true
  }
}

Verification

The documentation gate executes the setup and request, then checks the following result fields against independently specified expectations:

Float comparisons use a 1e-12 tolerance. Request schema validation and cross-record validation still apply. See errors and recovery for failure handling.