Monitoring & Metrics
Prometheus Endpoint
The API server exposes metrics at GET /metrics in Prometheus text exposition format:
curl http://localhost:8080/metrics
# HELP palimpsest_urls_fetched Total URLs successfully fetched.
# TYPE palimpsest_urls_fetched counter
palimpsest_urls_fetched 4521
# HELP palimpsest_urls_failed Total URLs that failed to fetch.
# TYPE palimpsest_urls_failed counter
palimpsest_urls_failed 12
# HELP palimpsest_urls_discovered Total URLs discovered via link extraction.
# TYPE palimpsest_urls_discovered counter
palimpsest_urls_discovered 15890
...
Available Metrics
| Metric | Type | Description |
|---|---|---|
palimpsest_urls_fetched | counter | Total URLs successfully fetched |
palimpsest_urls_failed | counter | Total fetch failures |
palimpsest_urls_discovered | counter | Total URLs discovered via links |
palimpsest_robots_blocked | counter | Total URLs blocked by robots.txt |
palimpsest_bytes_stored | counter | Total bytes written to blob storage |
palimpsest_blobs_stored | gauge | Unique blobs in storage |
palimpsest_api_requests | counter | Total API requests served |
palimpsest_frontier_pops | counter | Total frontier pop operations |
palimpsest_frontier_pushes | counter | Total frontier push operations |
All counters use AtomicU64 with Ordering::Relaxed — lock-free, thread-safe, no impact on crawl ordering (Law 1 safe).
Structured Logging
Palimpsest uses tracing with tracing-subscriber for structured logging:
# Set log level via environment
RUST_LOG=info palimpsest crawl https://example.com -o ./output
# Debug level for specific crate
RUST_LOG=palimpsest_frontier=debug palimpsest crawl ...
# JSON output for log aggregation
RUST_LOG=info palimpsest crawl ... 2>&1 | jq .
Grafana Dashboard Suggestions
| Panel | Query | Type |
|---|---|---|
| Throughput | rate(palimpsest_urls_fetched[1m]) | Graph |
| Error Rate | rate(palimpsest_urls_failed[1m]) / rate(palimpsest_urls_fetched[1m]) | Gauge |
| Discovery Ratio | palimpsest_urls_discovered / palimpsest_urls_fetched | Stat |
| Robots Blocked | rate(palimpsest_robots_blocked[1m]) | Graph |
| Storage Growth | palimpsest_bytes_stored | Graph |
| API Load | rate(palimpsest_api_requests[1m]) | Graph |
Alerting Suggestions
- Error rate > 5% — possible network or DNS issues
- Throughput drop > 50% — politeness starvation or backend slowdown
- Frontier pops = 0 — crawl may be stalled
- Storage growth flatline — dedup working well, or crawl stopped