Configuration

TOML Config File

Pass a TOML configuration file instead of CLI flags:

palimpsest crawl -c crawl.toml

Example Configuration

seeds = ["https://example.com/", "https://docs.example.com/"]

[crawl]
seed = 42
max_depth = 3
max_urls = 500
concurrency = 10
user_agent = "PalimpsestBot/0.1"
browser_mode = false
scope = "same_domain"
output_dir = "./output"

[politeness]
min_host_delay_ms = 1000
max_concurrent_hosts = 100

Configuration Fields

Seeds

seeds = ["https://example.com/", "https://docs.example.com/"]

One or more seed URLs. The crawl starts from these and discovers links outward.

Crawl Seed

seed = 42

The deterministic seed value. Controls all randomness in the system: frontier ordering, host rotation, browser JS overrides. Same seed = identical crawl.

Scope

scope = "same_domain"

Politeness Policy

[politeness]
min_host_delay_ms = 1000      # Minimum delay between same-host requests
max_concurrent_hosts = 100     # Maximum hosts being fetched in parallel

Presets (when using the API directly):

Depth and Limits

max_depth = 3       # Max link-following depth from seed (0 = seed page only)
max_urls = 500      # Hard cap on total URLs fetched
concurrency = 10    # Parallel fetch tasks

Browser Mode

browser_mode = true

Enables headless Chrome capture via CDP. Each page is loaded in a fresh browser context with determinism overrides applied (Date.now(), Math.random(), performance.now() are all seeded from CrawlSeed).

Output Directory

output_dir = "./output"

When set, artifacts are persisted to disk: content-addressed blobs, SQLite index, WARC++ file, and frontier state. When omitted, the crawl runs in-memory only.

CLI Flag Mapping