Adversarial Universes

The simulation framework includes six adversarial universes, each designed to stress a specific aspect of the crawl kernel.

LinkMaze

Domain: linkmaze.sim

A deep, wide graph. Each page contains links_per_page links to other pages in the maze. Tests frontier scheduling, deduplication, and depth limiting at scale.

#![allow(unused)]
fn main() {
LinkMaze { links_per_page: 500, total_pages: 1_000_000 }
}

EncodingHell

Domain: encoding.sim

UTF-8 edge cases: mixed encodings, byte-order marks, surrogate pairs, right-to-left text, zero-width characters, overlong sequences. Tests that content hashing and text extraction handle encoding correctly.

MalformedDom

Domain: malformed.sim

Broken HTML: unclosed tags, deeply nested tables, invalid attributes, missing doctype, mixed content models. Tests link extraction robustness — the parser must not crash or produce junk URLs.

RedirectLabyrinth

Domain: redirect.sim

Redirect chains (301 -> 302 -> 301 -> 200), redirect loops, cross-domain redirects, redirect-to-self. Tests redirect chain depth enforcement and URL normalization.

ContentTrap

Domain: trap.sim

Spider traps: infinite calendars (every date links to the next), session IDs in URLs (creating infinite unique URLs), query parameter permutations. Tests that max_urls and deduplication prevent infinite crawls.

TemporalDrift

Domain: drift.sim

Content changes between fetches. The same URL returns different content depending on the logical clock value. Tests temporal integrity — the index must correctly record each version.

#![allow(unused)]
fn main() {
TemporalDrift::new(1)  // Content changes every 1 logical tick
}

Composition

All six universes run simultaneously in scale tests:

#![allow(unused)]
fn main() {
let mut web = SimulatedWeb::new(seed);
web.add_universe(Box::new(LinkMaze { ... }));
web.add_universe(Box::new(EncodingHell));
web.add_universe(Box::new(MalformedDom));
web.add_universe(Box::new(RedirectLabyrinth));
web.add_universe(Box::new(ContentTrap));
web.add_universe(Box::new(TemporalDrift::new(1)));
}

The crawl must handle all six simultaneously — deterministic ordering across domains, correct error classification, and zero divergence between runs.