Glossary

Core Types

Crawl Kernel — The deterministic execution engine at the heart of Palimpsest. Schedules fetches, seals execution contexts, captures artifacts, stores blobs, indexes temporal state. Not a crawler — a kernel that crawlers are built on.

CrawlSeed — A 64-bit value that controls all randomness in the system. CrawlSeed::rng() returns a ChaCha8Rng PRNG. Same seed = identical behavior.

ContentHash — A 32-byte BLAKE3 hash. Used to address, store, retrieve, and verify every artifact. ContentHash::of(data) computes the hash.

CaptureInstant — A paired timestamp: wall clock (DateTime<Utc>) + logical clock (u64). Binds captures to both real-world time and crawl-internal ordering.

CrawlContextId — An opaque u64 identifier for a crawl session. Distinguishes captures from different runs.

ExecutionEnvelope — An immutable, sealed record of everything that affects a fetch: seed, timestamp, target URL, DNS snapshot, TLS fingerprint, browser config, and headers. Constructed via EnvelopeBuilder, frozen after build().

Frontier & Scheduling

Frontier — The deterministic URL scheduler. Maintains per-host priority queues in a BTreeMap, deduplicates by URL, and enforces politeness delays.

FrontierEntry — A URL in the frontier with depth, priority, and parent hash.

PolitenessPolicy — Configurable per-host rate limiting: minimum delay between requests and maximum concurrent hosts.

Artifacts & WARC

WARC++ — Palimpsest’s extension of the ISO 28500 WARC format. Adds envelope, dom-snapshot, resource-graph, and timing record types while maintaining backward compatibility.

WarcRecord — A single WARC record with type, record ID, content hash, and payload.

CaptureGroup — A bundle of related WARC records from a single fetch: envelope + request + response + optional DOM/resource graph/timing.

RecordType — Enum of WARC record types: 5 standard (warcinfo, request, response, resource, metadata) + 4 extensions (envelope, dom-snapshot, resource-graph, timing).

DomSnapshot — The rendered DOM state after JavaScript execution, captured via CDP.

ResourceGraph — The dependency graph of all sub-resources loaded for a page, with type, hash, initiator, and load ordering.

Storage

BlobStore — The trait interface for content-addressed storage. Implementations: InMemoryBlobStore, FileSystemBlobStore, ObjectStoreBlobStore.

Content-Addressed Storage — Storage where the key is the hash of the content. Same content = same key = stored once. Integrity is verifiable by recomputing the hash.

Deduplication — Structural dedup: if ContentHash::of(data_a) == ContentHash::of(data_b), the data is stored once. Not a post-process step — built into the storage model.

Index & Replay

Temporal Index — A multi-dimensional index mapping URL x time x hash x crawl_context. Not a lookup table — a queryable graph of web history.

IndexEntry — One capture record in the index: URL, CaptureInstant, ContentHash, CrawlContextId.

Replay Fidelity — The guarantee that stored artifacts are sufficient to reconstruct the original HTTP exchange, DOM state, and resource graph. Law 5.

Comparison & Analysis

Shadow Comparison — Side-by-side validation of Palimpsest output against legacy crawler WARC files (Heritrix, wget, Warcprox).

ContentChunk — A provenance-tagged text chunk for RAG pipelines. Carries source_url, captured_at, source_hash, chunk_hash, and char_offset.

Embedding — A vector of f32 values representing text semantics. Generated by an EmbeddingProvider.

EmbeddingProvider — The trait for embedding generation. HashEmbedder provides deterministic test embeddings via BLAKE3.

VectorStore — SQLite-backed storage for embeddings with brute-force cosine similarity search.

Cosine Similarity — The similarity metric between two embedding vectors. Range: -1.0 to 1.0. Used for semantic search.

Simulation

SimulatedWeb — A virtual internet for testing. Hosts multiple UniverseGenerator instances, each responding to URLs on its domain.

UniverseGenerator — The trait for generating deterministic responses. Implementations: LinkMaze, EncodingHell, MalformedDom, RedirectLabyrinth, ContentTrap, TemporalDrift.

Adversarial Universe — A simulation universe designed to stress a specific aspect of the crawl kernel (encoding, DOM parsing, redirects, spider traps, temporal changes).

Anti-Detection

JA3 — TLS fingerprinting method that hashes five fields from the ClientHello: TLS version, cipher suites, extensions, supported groups, EC point formats. Legacy but still deployed by WAFs.

JA4 — Current TLS fingerprinting standard (FoxIO). Sorts before hashing to defeat extension randomization. Three sections: header, sorted cipher hash, sorted extension hash.

BoringSSL — Google’s fork of OpenSSL used by Chrome. Palimpsest uses it (via wreq) for full ClientHello control, enabling browser-grade TLS impersonation.

Akamai h2 Fingerprint — Passive HTTP/2 fingerprint capturing SETTINGS frame values/order, WINDOW_UPDATE, PRIORITY frames, and pseudo-header ordering. Distinguishes browsers from automation clients.

CDP Stealth Mode — Anti-detection suite for headless Chrome. 17 evasion patches covering navigator.webdriver, window.chrome, plugins, WebGL, canvas noise, AudioContext noise, and more.

BrowserProfile — A unified, internally consistent browser identity tying TLS fingerprint + HTTP/2 settings + HTTP headers + JS surface into a single profile. Prevents cross-layer detection mismatches.

ProfileMode — Controls how browser profiles are selected: None (default), Fixed (one profile), Seeded (deterministic from CrawlSeed), RotatePerDomain (per-domain via BLAKE3).

WebdriverValue — Explicit config for navigator.webdriver in stealth mode. False (matches real Chrome, default) or Undefined (property deleted). Auditable, not hidden.