palimpsest-frontier

Deterministic seed-driven URL scheduler with politeness enforcement. Same seed + same URLs = identical dequeue order.

Frontier

#![allow(unused)]
fn main() {
impl Frontier {
    pub fn new(seed: CrawlSeed, policy: PolitenessPolicy) -> Self;
    pub fn push(&mut self, entry: FrontierEntry) -> bool;
    pub fn push_seed(&mut self, url: Url);
    pub fn push_discovered(&mut self, url: Url, depth: u32, parent: ContentHash) -> bool;
    pub fn pop(&mut self, now: DateTime<Utc>) -> Option<FrontierEntry>;
    pub fn len(&self) -> usize;
    pub fn is_empty(&self) -> bool;
    pub fn host_count(&self) -> usize;
    pub fn seen_count(&self) -> usize;
    pub fn save(&self, path: &Path) -> Result<(), FrontierPersistError>;
    pub fn load(&mut self, path: &Path) -> Result<usize, FrontierPersistError>;
    pub fn load_if_exists(seed: CrawlSeed, policy: PolitenessPolicy, path: &Path) -> Result<Self, FrontierPersistError>;
    pub fn seed(&self) -> CrawlSeed;
}
}

Internally uses BTreeMap<String, BTreeSet<FrontierEntry>> for host queues. URL deduplication via BTreeSet<String>.

FrontierEntry

#![allow(unused)]
fn main() {
pub struct FrontierEntry {
    pub url: Url,
    pub depth: u32,
    pub priority: u32,  // Lower = dequeued first
    pub parent: Option<ContentHash>,
}
}

Implements Ord: sorted by (priority, depth, url string).

PolitenessPolicy

#![allow(unused)]
fn main() {
pub struct PolitenessPolicy {
    pub min_host_delay: Duration,
    pub max_concurrent_hosts: usize,
}

impl PolitenessPolicy {
    pub fn default_policy() -> Self;  // 1s delay, 100 hosts
    pub fn aggressive() -> Self;      // 100ms delay, 500 hosts
    pub fn no_delay() -> Self;        // Zero delay, unlimited (testing only)
}
}

Persistence

save() serializes the frontier state to JSON. load() restores it. This enables crawl resumption — stop a crawl, restart later, continue from exactly where you left off.

Key Invariant

Same seed + same URLs pushed in same order = identical pop() sequence. Verified at 10,000 pages with zero divergence.