Incremental storage validation

The v0.2.1 adapter removes repeated full-history work from consecutive writes to the same journal on one Store. The numerical core remains v0.2.0; database schema and JSON protocol remain version 1. SQLite durability settings, immutable events and atomic retry receipts are unchanged.

Why this change

A stage profile of the original 2,000-revision workload identified history retrieval/JSON decoding as the dominant cost. The following are mean microseconds per mutation for the last 500 revisions. These timings exclude subprocess transport and use synthetic fixed-size manual forecasts.

All profiled post-change appends used validated state. Skipping the old scan addresses the observed growth with history length. SIMD would not eliminate that scan or its increasing amount of work. The remaining time is largely transaction/index/serialization and durability work.

Same-workload comparison

The saved unmodified v0.2.0 binary and new v0.2.1 binary each ran the same seeded JSON/SQLite stress harness three times at each journal length. Runs were sequential, with order alternated between repeats. Both binaries used rollback journaling and synchronous=FULL. The table reports the median of three per-run means for the last quarter of writes; p95 columns are medians of the three per-run p95 values.

An additional 10,000-revision candidate run passed all recovery and integrity checks. Its first-quarter mean was 0.180 ms and last-quarter mean 0.194 ms; p95 was 0.235 ms. The storage process RSS high-water mark was 12,996 KiB, sampled before recovery tests; database size was 15,273,984 bytes after recovery/concurrency checks. This is one local run, not a universal latency or memory guarantee.

The comparison uses only 100 numerical requests per run to fix the seeded setup and focus on journal behavior. It does not repeat the ten-million scoring run: no numerical algorithm changed. Every storage run still checks exact retries, rejected stale writes, acknowledged and unacknowledged-write recovery, concurrent creations, journal lengths and SQLite integrity.

Cache invariants

1. Each Store owns one private SQLite connection and retains at most one fully validated question history, forecast-ID set and evidence-origin set. Retained memory grows with that history. There is no whole-ledger clone per append.
2. Every mutation acquires BEGIN IMMEDIATE before consulting the cache. A cached question/version is reusable only when its connection-local PRAGMA data_version matches the token saved during validation. Another writer cannot commit while this transaction holds the write lock.
3. Any external commit invalidates the cache, including writes to unrelated questions or historical edits. On a miss, the complete journal is read and replayed, preserving sequence, lifecycle, identity, origin and Bayesian audit checks. Tokens are never compared between connections.
4. Cached state is moved into a tentative mutation. Validation/insertion/commit failures discard that tentative state. It is installed again only after commit succeeds. Request-ID receipts remain authoritative for exact retries, and retries do not advance cached state.
5. The normal mutation changes exactly two rows: one event and one receipt. Extra same-connection changes, including trigger side effects, suppress cache retention. SQLite total_changes supplies this additional check.
6. Journal reads and evaluation always fully replay stored history. Reopening, switching questions or discarding the cache also requires replay. No persistent checkpoint or schema migration is introduced.

These rules follow SQLite’s data-version semantics, immediate transaction semantics, and total-change counting. The cache does not authenticate raw database-file edits outside SQLite or a malicious database administrator.

Correctness evidence

New tests verify warm hits versus cold/question-switch misses, stale-writer invalidation, corruption of an earlier row while the cache is warm, receipt-insert rollback, commit-time deferred-constraint failure, manual and Bayesian evidence-origin tracking, same-connection trigger changes and agreement between warm writers and writers reopened before every append. Existing fork-prevention, replay, schema and durability tests continue to run. No correctness test depends on a timing threshold.

Remaining limits

Warm lifecycle validation is amortized O(new event plus new evidence), using retained ID/origin sets. The complete database operation still includes indexed lookups/inserts and a durable commit. A cold mutation is O(history plus evidence). Repeated external commits or interleaving different journals on one Store can force repeated cold replay; this optimization does not claim constant latency for those workloads. One active history remains allocated per Store. Persistent checkpoints, a bounded multi-question cache and transaction batching are separate future changes.

Reproduce

Preserve a v0.2.0 binary before rebuilding, or extract it from the existing v0.2.0 release archive. The comparison must use two different binaries; the report records their SHA-256 hashes. Then run:

CARGO_TARGET_DIR=target cargo build -p supercast-agent --release --locked --offline
python3 scripts/benchmark_storage.py --baseline target/benchmarks/baseline-agent --binary target/release/supercast-agent --output target/benchmarks/incremental-validation
CARGO_TARGET_DIR=target cargo run -p supercast-agent --example profile_store --release --locked --offline -- 2000 > target/benchmarks/store-profile-after.json
python3 scripts/benchmark.py stress --events 100 --revisions 10000 --output target/benchmarks/incremental-validation/candidate-10000.json

Store::last_mutation_profile() exposes microsecond stage timings for the last successful new mutation. Mutation attempts clear it; exact retries leave no profile. It does not change JSON output. The example emits one JSON report containing all mutation profiles and verifies the completed journal with a full audit. Baseline phase timings came from an instrumented pre-cache build; the end-to-end comparison uses the unmodified saved baseline.

Full run reports are in target/benchmarks/incremental-validation/; the phase profiles are in target/benchmarks/store-profile-{before,after}.json.

Baseline binary SHA-256: a6a9d5484ed1cda43b47754a5c3107cc808e13896fe2225d797126b228d96609

Candidate binary SHA-256: 79b85c50d5cbe57c6dde874cd30715f26dfd21163dbfb20b0a86f2548cd101d0