palimpsest-fetch

HTTP client + browser capture (CDP) + link extraction + robots.txt parsing + TLS/HTTP2 fingerprint impersonation + CDP stealth mode. Every fetch wraps an ExecutionEnvelope.

HttpFetcher

#![allow(unused)]
fn main() {
impl HttpFetcher {
    pub fn new(config: FetchConfig) -> Result<Self, PalimpsestError>;
    pub fn with_defaults() -> Result<Self, PalimpsestError>;
    pub async fn fetch(&self, envelope: &ExecutionEnvelope) -> Result<FetchResult, PalimpsestError>;
}
}

Uses wreq (BoringSSL backend) instead of reqwest. When an emulation profile is set, the TLS ClientHello and HTTP/2 SETTINGS frame match the selected browser.

FetchConfig

#![allow(unused)]
fn main() {
pub struct FetchConfig {
    pub connect_timeout: Duration,                   // Default: 30s
    pub total_timeout: Duration,                     // Default: 120s
    pub max_body_size: u64,                          // Default: 256 MiB
    pub max_redirects: usize,                        // Default: 10
    pub emulation: Option<wreq_util::Emulation>,     // Default: None
}
}

When emulation is set (e.g., Emulation::Chrome133), wreq impersonates the selected browser’s TLS fingerprint (JA3/JA4 including post-quantum key shares) and HTTP/2 settings (SETTINGS frame values/order, WINDOW_UPDATE, pseudo-header ordering). 70+ browser profiles available: Chrome 100-137, Firefox 109-139, Safari 15-18.5, Edge, Opera.

BrowserFetcher

#![allow(unused)]
fn main() {
impl BrowserFetcher {
    pub fn new(config: BrowserFetchConfig) -> Self;
    pub async fn fetch(&self, url: &Url, envelope: &ExecutionEnvelope, seed: CrawlSeed)
        -> Result<BrowserFetchResult, PalimpsestError>;
}
}

Launches headless Chrome via CDP. Injects determinism overrides and (optionally) 17 stealth evasion patches. Captures DOM snapshot, sub-resources via Network events, and resource dependency graph.

BrowserFetchConfig

#![allow(unused)]
fn main() {
pub struct BrowserFetchConfig {
    pub page_timeout: Duration,          // Default: 30s
    pub viewport_width: u32,             // Default: 1920
    pub viewport_height: u32,            // Default: 1080
    pub js_enabled: bool,                // Default: true
    pub user_agent: String,              // Default: "PalimpsestBot/0.1"
    pub stealth: bool,                   // Default: false
    pub webdriver_value: WebdriverValue, // Default: False
}
}

WebdriverValue

#![allow(unused)]
fn main() {
pub enum WebdriverValue {
    False,     // Matches real non-automated Chrome (default)
    Undefined, // Property appears deleted
}
}

Explicit, auditable config choice for navigator.webdriver. Default False passes Rebrowser Bot Detector (10/10).

CDP Stealth Mode

When stealth: true, the browser fetcher applies:

Chrome launch hardening:

• --disable-blink-features=AutomationControlled
• --disable-component-extensions-with-background-pages

17 stealth evasion patches (injected via addScriptToEvaluateOnNewDocument):

All noise patches use deterministic xorshift PRNGs seeded from CrawlSeed (Law 1).

Browser Emulation Profiles

#![allow(unused)]
fn main() {
pub struct BrowserProfile { /* unified TLS + HTTP/2 + headers + JS identity */ }

pub enum ProfileMode {
    None,                        // No impersonation (default)
    Fixed(BrowserProfile),       // Same profile for all requests
    Seeded,                      // Generate from CrawlSeed
    RotatePerDomain,             // Per-domain via BLAKE3(seed + domain)
}
}

Pre-built profiles: BrowserProfile::chrome_windows(), firefox_linux(), safari_macos().

See profile module for details.

BrowserFetchResult

#![allow(unused)]
fn main() {
pub struct BrowserFetchResult {
    pub fetch_result: FetchResult,
    pub dom_snapshot: Option<DomSnapshot>,
    pub resource_graph: Option<ResourceGraph>,
    pub sub_resources: Vec<WarcRecord>,
}
}

Link Extraction

#![allow(unused)]
fn main() {
pub fn extract_links(html: &str, base_url: &Url) -> Vec<Url>;
pub fn normalize_url(url: &Url) -> Option<Url>;
pub fn normalize_url_for_comparison(url: &Url) -> String;
}

extract_links strips <script> and <style> content before scanning for href and src attributes. Output is deduplicated and sorted for determinism.

Robots.txt

#![allow(unused)]
fn main() {
pub struct RobotsRules { pub crawl_delay: Option<Duration> }

impl RobotsRules {
    pub fn parse(body: &str) -> Self;  // RFC 9309 compliant
}
}

Per-origin caching in BTreeMap (deterministic).

Stealth Regression Tests

5 integration tests against live public detection sites:

Run with: cargo test -p palimpsest-fetch --test stealth_test -- --ignored --nocapture --test-threads=1

Key Invariant

Every fetch receives an ExecutionEnvelope. The envelope seals the context before the network request begins, enabling replay and verification. Emulation profile and stealth config are deterministic inputs (Law 1).