The Web's Memory Layer Should Be Provable

What Is Palimpsest?

Palimpsest is a deterministic crawl kernel — not a crawler, not a Wayback clone. It is the foundational memory layer of the web: same input + same seed = identical crawl, identical artifacts, identical replay. Every design decision bends around six immutable laws covering determinism, content addressability, temporal integrity, and replay fidelity.

The system includes raw HTTP and headless Chrome capture, a SQLite temporal index, content-addressed blob storage (local + S3/GCS/Azure), WARC++ output compatible with legacy crawlers, robots.txt compliance, concurrent fetching, crawl resumption, shadow comparison against Heritrix/wget, and a deterministic simulation framework that proves correctness at 5,000 pages with zero divergence.

The Six Laws

Setting Up Your Machine for Claude Code

Before writing a single line of code, your environment needs to be precisely configured. Claude Code is an AI-native development tool that operates as a CLI agent — it reads your codebase, executes commands, edits files, and runs tests. But it needs the right foundation to be effective.

# Install Node.js if you don't have it
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt-get install -y nodejs

# Install Claude Code globally
npm install -g @anthropic-ai/claude-code

# Verify installation
claude --version

export ANTHROPIC_API_KEY="sk-ant-..."

# Install rustup (Rust toolchain manager)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Source the environment
source "$HOME/.cargo/env"

# Install stable toolchain
rustup default stable

# Verify
rustc --version  # Should be 1.85+
cargo --version

# Ubuntu/Debian
sudo apt-get update
sudo apt-get install -y build-essential pkg-config libssl-dev

# Install Google Chrome for browser capture
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | \
  sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt-get update
sudo apt-get install -y google-chrome-stable

# Verify
google-chrome --version

sudo apt-get install -y gh
gh auth login

git config --global user.name "Your Name"
git config --global user.email "you@example.com"

Establishing the Project Foundation

The key insight from this build: Claude Code is most effective when given clear architectural constraints upfront. The CLAUDE.md file is the contract between you and the AI — it defines invariants, coding standards, and non-negotiable rules.

The Foundation: CLAUDE.md

This is the full project instruction file that Claude Code reads on every startup:

# .claude/hooks/guard-determinism.sh
# Runs before every Edit/Write tool call
# Warns about: rand crate, HashMap, Instant::now(),
# .unwrap()/.expect() in library code

# .claude/hooks/check-invariants.sh
# Runs before every response ends
# Verifies: Six Laws compliance, error classification,
# content addressability, temporal integrity

# Available skills:
/shadow-compare   # Compare against legacy crawlers
/replay-test      # Verify replay fidelity
/crawl-verify     # Determinism verification
/envelope-audit   # Audit ExecutionEnvelope immutability
/frontier-sim     # Simulate frontier scheduling
/invariant-check  # Audit against Six Laws
/threat-model     # Security review
/sentinel         # Monitor Anthropic API changes

The Build Process

With the environment configured and constraints established, the actual build proceeded bottom-up through the dependency graph. Claude Code implemented each crate, wrote tests, ran clippy, and performed Six Laws audits at every commit boundary.

Launch Claude Code

# Start Claude Code in the project directory
cd palimpsest
claude

# Claude Code reads CLAUDE.md, .claude/rules/, and .claude/hooks/
# automatically on startup. It understands the project's invariants
# before you type a single command.

The Conversation Pattern

The most effective pattern was direct and terse. No lengthy explanations needed — Claude Code had the context from CLAUDE.md.

# Start building
"let start working on this project"

# After reviewing the plan
"yes"

# Keep momentum
"proceed"

# Test against reality
"zuub.com"

# Scale up
"let run it"

# Ship
"lets commit and tag"

Key Milestones

The build progressed through three tagged releases in a single session:

Key Code Artifacts

These are the critical abstractions that make the system work. Each one enforces one or more of the Six Laws at the code level.

CrawlSeed: The Source of All Determinism

Every random decision in the entire system flows from this single seed through a ChaCha8 PRNG. No rand::thread_rng(), no OsRng, no entropy sources anywhere in the kernel.

ExecutionEnvelope: The Sealed Context

Every fetch runs inside an immutable envelope that captures seed, timestamp, DNS snapshot, TLS fingerprint, and browser config. This is what makes replay possible — given the same envelope, the system must produce identical artifacts.

Browser Determinism Layer

Injected before every page navigation, this JavaScript overrides all non-deterministic browser APIs with seed-derived values. Same seed = same Date.now(), same Math.random(), same performance.now().

Determinism Verification Harness

The core proof: crawl a simulated web twice with the same seed, assert byte-identical URL order, blob hashes, and index entries. Any divergence is a Law 1 violation. Proven at 5,000 pages with zero divergence.

Distributed Crawling

One coordinator. N workers. Zero external infrastructure. The frontier server exposes the same deterministic scheduler over HTTP — workers pull URLs, fetch pages, push discovered links back. Horizontal scaling without Redis, Kafka, or message queues.

Architecture

# Terminal 1 — Start the frontier server
palimpsest serve -p 8090 -s 42 --politeness-ms 500

# Terminal 2 — Worker A
palimpsest worker --server http://localhost:8090 -o ./output-a

# Terminal 3 — Worker B
palimpsest worker --server http://localhost:8090 -o ./output-b

# Seed the crawl
curl -X POST http://localhost:8090/seeds \
  -d '{"urls":["https://example.com/","https://zuub.com/"]}'

# Monitor progress
curl http://localhost:8090/status
# {"queue_size":847,"seen_count":1203,"host_count":2,"seed_value":42}

API Endpoints

What Made This Work

Constraints First, Code Second

CLAUDE.md was written before any Rust code. The Six Laws, error taxonomy, dependency policy, and testing philosophy were all defined upfront. Claude Code enforced these constraints on every tool call through hooks. The result: zero invariant violations across 269 tests and 27 commits.

Real-World Validation Early

We didn't wait until the end to test against production HTML. The zuub.com crawl at commit 9 exposed a real bug (politeness starvation) that no unit test would have caught. The wget side-by-side comparison found three more bugs (script tag extraction, scheme normalization, WARC angle brackets). Every bug found by reality was fixed and tested before moving on.

Deterministic Simulation Testing

The simulation framework (palimpsest-sim) generates a virtual internet with six adversarial universes: LinkMaze, EncodingHell, MalformedDom, RedirectLabyrinth, ContentTrap, and TemporalDrift. Crawling this simulated web twice with the same seed and asserting byte-identical results is the ultimate proof of Law 1 (Determinism). We proved it at 5,000 pages.

Parallel Sub-Agents

Claude Code's agent system let us launch three independent tasks in parallel (sub-resource capture, object store backend, crawl resumption) on isolated git worktrees. While the agents explored complex CDP event wiring, we implemented the simpler features directly. The agents provided architectural insight even when they didn't produce final code.