palimpsest-crawl

The orchestrator — the main crawl loop that integrates all layers: frontier scheduling, envelope construction, HTTP/browser fetching, link extraction, artifact creation, blob storage, temporal indexing, WARC output, and frontier persistence.

CrawlConfig

#![allow(unused)]
fn main() {
pub struct CrawlConfig {
    pub seeds: Vec<Url>,
    pub crawl_seed: CrawlSeed,
    pub crawl_context: CrawlContextId,
    pub max_depth: u32,
    pub max_urls: usize,
    pub politeness: PolitenessPolicy,
    pub scope: CrawlScope,
    pub concurrency: usize,
    pub user_agent: String,
    pub browser_mode: bool,
    pub output_dir: Option<PathBuf>,
}

impl CrawlConfig {
    pub fn for_test(seed_url: Url) -> Self;
    pub fn seed_hosts(&self) -> Vec<String>;
    pub fn seed_domains(&self) -> Vec<String>;
}
}

CrawlScope

#![allow(unused)]
fn main() {
pub enum CrawlScope {
    SameDomain,  // Registrable domain match
    SameHost,    // Exact host match
    Any,         // No restriction
}
}

CrawlStats

#![allow(unused)]
fn main() {
pub struct CrawlStats {
    pub urls_fetched: usize,
    pub urls_failed: usize,
    pub urls_discovered: usize,
    pub robots_blocked: usize,
    pub blobs_stored: usize,
    pub bytes_stored: u64,
    pub warc_path: Option<String>,
}
}

CrawlOrchestrator

#![allow(unused)]
fn main() {
impl CrawlOrchestrator {
    pub async fn new(config: CrawlConfig) -> Result<Self, PalimpsestError>;
}
}

The orchestrator loop:

1. Pop batch of URLs from frontier (respects politeness)
2. Build ExecutionEnvelope for each
3. Fetch concurrently via tokio::spawn
4. Extract links from HTML responses
5. Push discovered URLs back to frontier (scope-filtered)
6. Store blobs, insert index entries, write WARC records
7. Save frontier state for resumption
8. Repeat until frontier empty or max_urls reached

Key Invariant

The orchestrator is the integration point. It does not add non-determinism — all ordering comes from the frontier, all time from envelopes, all randomness from the seed.