palimpsest-server
HTTP frontier server, retrieval API, and Prometheus metrics. Three distinct services in one crate.
Frontier API
Distributed crawling coordination. Workers pop URLs, fetch them, and push discoveries back.
FrontierState
#![allow(unused)]
fn main() {
pub struct FrontierState {
pub frontier: Mutex<Frontier>,
pub seed: CrawlSeed,
}
}Endpoints
| Method | Path | Request Body | Response |
|---|---|---|---|
| POST | /seeds | {"urls": ["..."]} | {"accepted": N} |
| POST | /pop | {} | {"url": "...", "depth": 0, "priority": 0} |
| POST | /discovered | {"urls": [{"url": "...", "depth": 1, "parent_hash": "..."}]} | {"accepted": N} |
| GET | /status | — | {"queue_size": N, "seen_count": N, "host_count": N, "seed_value": N} |
| GET | /health | — | "ok" |
Retrieval API
Content serving for AI pipelines and search.
RetrievalState
#![allow(unused)]
fn main() {
pub struct RetrievalState {
pub index: Mutex<SqliteIndex>,
pub storage: FileSystemBlobStore,
pub chunk_config: ChunkConfig,
}
}Endpoints
| Method | Path | Query Params | Description |
|---|---|---|---|
| GET | /v1/content | url | Raw captured content |
| GET | /v1/chunks | url | RAG-ready chunks with provenance |
| GET | /v1/history | url | All captures with timestamps |
| GET | /v1/search | q | Full-text search |
| GET | /health | — | Health check |
Metrics
#![allow(unused)]
fn main() {
pub struct Metrics {
pub urls_fetched: AtomicU64,
pub urls_failed: AtomicU64,
pub urls_discovered: AtomicU64,
pub robots_blocked: AtomicU64,
pub bytes_stored: AtomicU64,
pub blobs_stored: AtomicU64,
pub api_requests: AtomicU64,
pub frontier_pops: AtomicU64,
pub frontier_pushes: AtomicU64,
}
impl Metrics {
pub fn new() -> Self;
pub fn render(&self) -> String; // Prometheus text exposition format
}
}All counters use AtomicU64 with Ordering::Relaxed — thread-safe, no locks, no control flow impact (Law 1 safe).