Crate Map
All 15 Crates
| Crate | Layer | Responsibility | Key Invariant |
|---|---|---|---|
palimpsest-core | Foundation | Types, BLAKE3 hashing, seeded PRNG, error taxonomy | No IO. Pure types only. |
palimpsest-envelope | Foundation | Sealed execution context | Immutable after construction |
palimpsest-frontier | Foundation | Deterministic URL scheduler with politeness | Same seed = same traversal order |
palimpsest-fetch | Capture | HTTP client + browser capture (CDP) + link extraction | Every fetch wraps an envelope |
palimpsest-artifact | Capture | WARC++ serialization, capture groups | Content-addressed outputs |
palimpsest-storage | Persistence | Content-addressable blobs (memory, fs, S3/GCS/Azure) | Dedup is structural |
palimpsest-index | Persistence | Temporal graph: URL x time x hash x context | Queryable history |
palimpsest-replay | Persistence | HTTP reconstruction, DOM rehydration | Bit-identical replay from artifacts |
palimpsest-crawl | Orchestration | Main crawl loop and coordination | Integrates all layers |
palimpsest-shadow | Persistence | Comparison engine vs legacy crawlers | Cross-format validation |
palimpsest-extract | Capture | HTML-to-text + RAG chunking with provenance | Deterministic extraction |
palimpsest-embed | Capture | Embedding generation, vector search, change detection | BLAKE3-based test embeddings |
palimpsest-server | Interface | HTTP frontier server + retrieval API + metrics | Thread-safe state |
palimpsest-sim | Orchestration | Deterministic simulation testing framework | Proves Laws 1-6 |
palimpsest-cli | Interface | Command-line interface (10 subcommands) | Thin wrapper |
Dependency Graph
palimpsest-cli
├── palimpsest-core
├── palimpsest-crawl
│ ├── palimpsest-core
│ ├── palimpsest-envelope
│ ├── palimpsest-frontier
│ │ └── palimpsest-core
│ ├── palimpsest-fetch
│ │ └── palimpsest-core
│ ├── palimpsest-artifact
│ │ └── palimpsest-core
│ ├── palimpsest-storage
│ │ └── palimpsest-core
│ └── palimpsest-index
│ └── palimpsest-core
├── palimpsest-frontier
├── palimpsest-index
├── palimpsest-storage
├── palimpsest-replay
├── palimpsest-server
│ ├── palimpsest-frontier
│ ├── palimpsest-index
│ └── palimpsest-storage
├── palimpsest-shadow
├── palimpsest-artifact
├── palimpsest-envelope
├── palimpsest-extract
└── palimpsest-fetch
Key Pattern
Every crate depends on palimpsest-core for shared types (CrawlSeed, ContentHash, CaptureInstant, PalimpsestError). No crate performs IO unless its responsibility requires it. The foundation layer is pure computation.