Docker Deployment
Dockerfile
Multi-stage build: Rust 1.86 builder stage compiles a release binary, Debian slim runtime stage runs it.
# Build the image
docker build -t palimpsest .
# Single crawl
docker run -v ./output:/data palimpsest crawl https://example.com -d 2 -o /data
# View help
docker run palimpsest --help
The final image includes only the stripped binary and minimal runtime dependencies (ca-certificates, libssl3).
Docker Compose
The compose file runs four services sharing a named volume:
docker compose up
| Service | Command | Port | Purpose |
|---|---|---|---|
api | api -p 8080 --data-dir /data | 8080 | Retrieval API |
frontier | serve -p 8090 -s 42 --politeness-ms 500 | 8090 | Frontier server |
worker | worker --server http://frontier:8090 -o /data | — | Fetch worker |
crawl | crawl <URL> -d 2 -m 50 -o /data | — | One-shot crawl |
The crawl service uses the crawl profile — run it explicitly:
docker compose run --profile crawl crawl
Shared Volume
All services share the palimpsest-data named volume mounted at /data. This contains blobs, the SQLite index, WARC files, and frontier state.
Production Considerations
- Set resource limits (
mem_limit,cpus) per service - The frontier server is stateful — run a single instance
- Workers are stateless — scale horizontally with
docker compose up --scale worker=N - Mount the data volume to persistent storage for durability
- Expose only the
apiservice port externally; keepfrontierinternal