Simulation Framework

The simulation framework (palimpsest-sim) provides a virtual internet for testing. It replaces real HTTP with deterministic, seed-driven responses — enabling proof that the Six Laws hold at scale.

SimulatedWeb

A SimulatedWeb hosts multiple “universes,” each owning a domain:

#![allow(unused)]
fn main() {
let mut web = SimulatedWeb::new(CrawlSeed::new(42));
web.add_universe(Box::new(LinkMaze { links_per_page: 500, total_pages: 100_000 }));
web.add_universe(Box::new(EncodingHell));
web.add_universe(Box::new(MalformedDom));
}

Calling web.fetch(&url) returns a SimulatedResponse generated deterministically from the seed and URL.

SimulatedServer

Wraps SimulatedWeb with wiremock to serve responses over real HTTP. The CrawlOrchestrator connects to it as if it were the real web.

verify_determinism

The core harness:

#![allow(unused)]
fn main() {
let result = verify_determinism(
    || build_web(seed),  // Factory creates identical web each time
    &seed_urls,
    max_depth,
    max_urls,
).await?;
}

This function:

1. Creates a SimulatedWeb from the factory
2. Runs a full crawl (orchestrator + frontier + storage + index)
3. Records all URLs, blob hashes, and index entries
4. Creates a second SimulatedWeb from the same factory
5. Runs an identical crawl
6. Asserts the two runs produced byte-identical results

Any divergence in URLs, blob hashes, or index entries causes a test failure.

verify_resumption_determinism

Tests crawl resumption:

1. Crawl 500 pages, save frontier state
2. Create new frontier, load saved state
3. Continue crawling to 1000 pages
4. Compare against a single 1000-page run

Same result = Law 1 holds across save/load boundaries.

Scale Tests