Temporal Index

The temporal index is a graph, not a flat lookup table. It records every capture across four dimensions: URL, time, content hash, and crawl context. This enables queries like “show me every version of this page” or “what changed between these two crawls.”

CaptureInstant

Every capture is timestamped with a paired clock:

#![allow(unused)]
fn main() {
pub struct CaptureInstant {
    pub wall: DateTime<Utc>,    // Real-world time
    pub logical: u64,           // Monotonic counter within a crawl
}
}

Wall time records when the capture happened. Logical time records the ordering within a single crawl session, immune to clock drift.

IndexEntry

#![allow(unused)]
fn main() {
pub struct IndexEntry {
    pub url: Url,
    pub captured_at: CaptureInstant,
    pub content_hash: ContentHash,
    pub crawl_context: CrawlContextId,
}
}

Each entry represents one capture of one URL at one point in time, with a pointer (content hash) to the stored artifact.

Backends

InMemoryIndex

Uses BTreeMap for deterministic ordering. Suitable for testing and short-lived crawls.

#![allow(unused)]
fn main() {
let mut index = InMemoryIndex::new();
index.insert(entry);
let results = index.query(&IndexQuery::for_url(&url));
}

SqliteIndex

Persistent SQL-backed index with WAL mode for concurrent reads:

#![allow(unused)]
fn main() {
let mut index = SqliteIndex::open(Path::new("./output/index.sqlite"))?;
index.insert(entry)?;
let results = index.query(&IndexQuery::for_url(&url))?;
}

The schema includes a UNIQUE constraint on (url, wall_time, content_hash) to prevent duplicate entries.

Queries

IndexQuery supports multi-dimensional filtering:

• By URL — all captures of a specific URL
• By time range — all captures within a window
• By content hash — find which URLs produced a specific blob
• By crawl context — all captures from a specific crawl session

Results are ordered by captured_at (ascending), then URL string.

Use Cases

History — “Show me every version of https://example.com/ across all crawls”:

#![allow(unused)]
fn main() {
let history = index.query(&IndexQuery::for_url(&url))?;
for entry in &history {
    println!("{} -> {}", entry.captured_at.wall, entry.content_hash.as_hex());
}
}

Change Detection — Compare content hashes across captures to identify when a page changed.

Provenance — Every RAG chunk and embedding links back to an IndexEntry via source_url, captured_at, and source_hash.