# Enterprise Context Efficiency Benchmark 0.1

Status: **Draft benchmark specification**<br>
Short name: **ECEB-0.1**<br>
Specification identifier: `urn:symgliph:spec:eceb:0.1`<br>
Last updated: 2026-08-02

## 1. Abstract

The Enterprise Context Efficiency Benchmark (ECEB) measures how little verified,
policy-permitted context an AI system can receive while preserving task quality.
It evaluates a context-construction system rather than model knowledge in
isolation.

ECEB reports a metric vector covering quality, input tokens, cost, latency,
provenance, policy compliance, and repeatability. A token or cost improvement is
qualified only when every mandatory quality, provenance, policy, budget, and
repeatability gate passes.

ECEB does not collapse incomparable failures into one opaque score. In
particular, restricted-data disclosure is a hard failure rather than a small
numeric penalty.

## 2. Conformance language

The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHOULD**, **SHOULD NOT**,
and **MAY** are to be interpreted as described in RFC 2119 and RFC 8174.

## 3. Scope

ECEB-0.1 defines:

1. an immutable enterprise-style corpus registry;
2. quality-constrained tasks with evidence and access-policy labels;
3. repeated observations for full-context, retrieval, and compiled-context
   strategies;
4. deterministic structural and answer-constraint gates;
5. a declared external-judge trust boundary; and
6. comparable token, cost, latency, provenance, policy, and repeatability
   measurements.

ECEB-0.1 does not define a universal model judge, provider pricing schedule,
connector protocol, or enterprise authorization system.

## 4. Registered schemas

| Artifact | Schema identifier |
|---|---|
| Benchmark definition | `symgliph.eceb/benchmark-v0.1` |
| Recorded run | `symgliph.eceb/run-v0.1` |
| Evaluation report | `symgliph.eceb/report-v0.1` |

JSON interchange artifacts MUST validate against the corresponding JSON Schema
in [`schema/`](schema/).

## 5. Corpus and provenance

Every corpus artifact records:

- a stable `artifact_id`;
- a portable corpus-relative `path`;
- an enterprise source kind;
- an information classification; and
- the BLAKE3 digest of its complete source bytes.

The reference source kinds are code, RFC, runbook, pull request, chat, issue,
API specification, infrastructure, and policy.

The ECEB corpus root is:

```text
BLAKE3(
  "symgliph:eceb:corpus:v0.1\0" ||
  canonical_json(sorted_artifact_1) || "\n" ||
  ...
)
```

Artifacts are sorted by their complete record ordering. The corpus root commits
to identity, path, kind, classification, and source digest. Before evaluation,
an implementation MUST re-hash every registered file and reject missing or stale
source bytes.

## 6. Task definition

Each task declares:

- the user query;
- the artifacts required to support an acceptable answer;
- allowed information classifications;
- explicitly denied artifacts;
- an ascending token-budget ladder;
- an absolute quality floor;
- a maximum regression from the full permitted-context baseline; and
- deterministic answer terms.

Required evidence and denied evidence are independent labels. A required artifact
MUST exist in the benchmark registry. Selecting an unknown artifact is a policy
violation and a provenance failure.

ECEB-0.1 uses a linear four-value classification vocabulary: `public`,
`internal`, `confidential`, and `restricted`. A task lists permitted values
explicitly; the evaluator MUST NOT infer access from ordinal position.

## 7. Strategies

ECEB-0.1 registers three context strategies:

`full_permitted_context`
: The maximum policy-permitted baseline context. It excludes sources the actor
  may not receive and therefore is not equivalent to an unrestricted corpus
  dump.

`naive_top_k`
: A declared fixed retrieval baseline. ECEB records it but does not require it
  for conformance.

`compiled`
: A budget-aware context packet produced by the evaluated system.

Every comparative run MUST include the full permitted-context baseline and at
least one compiled budget for every task. All strategies MUST use the same model,
judge profile, corpus root, task text, decoding parameters, and pricing snapshot
unless a report explicitly declares a factorial experiment.

## 8. Recorded observations

An observation binds one task, strategy, budget, and repetition to:

- provider-reported input and output tokens;
- input and output cost in micro-US-dollars;
- end-to-end latency in milliseconds;
- judge quality in thousandths;
- the complete answer;
- selected artifact identifiers and observed source digests; and
- a BLAKE3 digest of the exact rendered model context.

The model and judge fields SHOULD include immutable provider revisions when the
provider exposes them. The pricing snapshot MUST identify its source and date.
Cached-input discounts, embedding requests, tool calls, and retries MUST be
included or reported as separate observations; they MUST NOT silently disappear
from the compiled strategy's cost.

## 9. Trust boundary

The reference evaluator verifies structure, budgets, source provenance, evidence
coverage, access policy, deterministic answer constraints, and repeated context
identity. It does not independently reproduce a model-judge score recorded in
`quality_milli`.

A publishable model-quality claim MUST therefore provide the judge prompt,
rubric, model identity, raw judgments, and human-review status. A conformance
fixture MAY use synthetic scores when it is labeled as non-empirical.

Required answer terms are a deterministic floor, not a semantic-quality metric.
They prevent a high declared judge score from passing an answer that omits an
explicit benchmark invariant.

## 10. Quality-constrained minimum budget

For task `q`, quality threshold `τ`, and compiled context under budget `B`:

```text
B*(q, τ) = min B
```

subject to:

```text
quality(answer(context_B(q))) >= τ
provenance_coverage == 100%
policy_violations == 0
prompt_tokens <= B
repeatability == true
answer_constraints == satisfied
```

The dynamic quality threshold is:

```text
τ = max(
  task.minimum_quality_milli,
  baseline_quality_milli - task.maximum_quality_regression_milli
)
```

The evaluator walks the declared budget ladder in ascending order. The first
budget whose required repetitions all pass is the minimum viable budget. Missing
observations at a budget are failures for that budget, not implied results.

## 11. Mandatory gates

A strategy-budget group passes only when:

1. it contains at least `required_repetitions` observations;
2. every prompt fits its declared token budget;
3. every quality score meets the dynamic floor;
4. every required answer term occurs case-insensitively in every answer;
5. every required artifact is selected with its current complete-source digest;
6. no selected artifact is denied, disallowed, or unknown; and
7. every repetition has the same exact-context BLAKE3 digest.

An implementation MUST evaluate all applicable failures and SHOULD report them
together. It MUST NOT stop at the first favorable budget while concealing a
policy violation observed at that same budget.

## 12. Metrics

For baseline tokens `T_full` and minimum viable compiled tokens `T_compiled`:

```text
token_savings_bps = floor(10,000 × (T_full - T_compiled) / T_full)
```

Cost savings use the same equation over total input plus output micro-US-dollar
cost. Negative savings saturate at zero in the v0.1 report.

The report MUST retain, per task:

- baseline and compiled prompt tokens;
- minimum viable declared budget;
- token savings;
- baseline and compiled cost;
- cost savings;
- baseline, floor, and compiled quality;
- provenance coverage;
- policy-violation count;
- repeatability; and
- every failed assertion.

The aggregate report MAY average token and cost savings across passing tasks.
It MUST also report the numerator and denominator of passing tasks. Failed tasks
MUST NOT be dropped from the denominator.

## 13. Repeatability

`required_repetitions` MUST be positive. All observations for the same task,
strategy, and budget MUST carry a unique one-based repetition number.

ECEB-0.1 defines context repeatability as equality of the exact rendered-context
digest. Answer text MAY vary because model decoding can be nondeterministic, but
every repetition must independently pass quality and answer constraints.

## 14. Policy failures

The evaluator increments `policy_violations` when selected evidence:

- has an unknown artifact identifier;
- has a classification absent from the task's allow-list; or
- is explicitly denied for the task.

A stale digest is a provenance failure even when the artifact would otherwise be
permitted. Policy and provenance are reported separately because they require
different remediation.

No numeric gain can compensate for a policy violation. A task with one or more
policy violations at its candidate budget does not qualify.

## 15. Claim classes

ECEB reports SHOULD label claims using one of these classes:

`conformance_seed`
: Synthetic or hand-authored records proving evaluator behavior. It is not a
  model-performance claim.

`public_reproduction`
: A public, immutable corpus and replayable model/judge configuration.

`enterprise_proxy`
: A public corpus constructed to resemble enterprise source mixtures and policy
  boundaries.

`enterprise_validated`
: A private or public real-enterprise evaluation with declared sampling,
  authorization, human review, and reproducibility controls.

Results from one class MUST NOT be represented as evidence for a stronger class.

## 16. Publication requirements

A comparative ECEB result MUST publish or escrow:

- benchmark, run, and report artifacts;
- immutable corpus and task roots;
- source licenses and redistribution status;
- the context-construction implementation revision;
- model, judge, tokenizer, and pricing identities;
- raw per-repetition measurements;
- every exclusion, timeout, retry, and failed task;
- uncertainty or a statement that it was not estimated; and
- human-review status.

Provider-native token counts are authoritative for cost comparison. Local token
counts MAY be included for deterministic budgeting but MUST be labeled separately.

## 17. Security considerations

An ECEB corpus can contain material intentionally designed to resemble secrets,
PII, and access-controlled records. Public fixtures MUST use synthetic values and
MUST NOT contain operational credentials or personal data.

Benchmark runners SHOULD isolate corpus connectors, redact provider logs, avoid
placing credentials in run artifacts, and treat model output as a possible data
exfiltration channel. A system that refuses a task because no policy-permitted
answer exists SHOULD receive an abstention task rather than be forced to leak.

## 18. Limitations of v0.1

ECEB-0.1 uses explicit artifact-level evidence and classification allow-lists.
It does not yet standardize field-level redaction, attribute-based access control,
temporal permissions, prompt-injection scoring, confidence intervals, or a sealed
private test set. These are candidates for later profiles.

The checked-in enterprise seed is deliberately small and synthetic. Its passing
numbers prove the evaluator and negative gates only.