# SMESH Crawler

> A local, agent-orchestrated web crawler that finds exact public-web evidence and historical changepoints while deterministic Rust policy controls every fetch.

SMESH Crawler exposes three MCP tools: `start_crawl`, `start_wayback_search`, and `get_job`. Models may advise on already-fetched public material, but cannot fetch URLs, widen scope, bypass robots policy, or expand budgets. AI support through OpenRouter is optional and off by default.

## Install

- [Latest native MCPB bundles](https://github.com/copyleftdev/smesh-crawler/releases/latest): Linux, macOS Intel, macOS Apple silicon, and Windows packages with SHA-256 checksums.
- [Local MCP setup](https://github.com/copyleftdev/smesh-crawler/blob/main/docs/MCP-SERVICE.md): Codex, ChatGPT desktop, Anthropic-compatible hosts, and generic STDIO configuration.
- [Release process](https://github.com/copyleftdev/smesh-crawler/blob/main/docs/RELEASE.md): build limits, validation, and clean-machine expectations.

## Architecture and policy

- [Project source](https://github.com/copyleftdev/smesh-crawler): MPL-2.0 Rust implementation and vendored SMESH primitives.
- [Architecture RFC](https://github.com/copyleftdev/smesh-crawler/blob/main/docs/RFC-0001.md): orchestration, deterministic authority, and evidence flow.
- [Resilience model](https://github.com/copyleftdev/smesh-crawler/blob/main/docs/RESILIENCE.md): robots, SSRF, redirects, budgets, circuits, and unknown-site handling.
- [Wayback changepoint case study](https://github.com/copyleftdev/smesh-crawler/blob/main/docs/DOGFOOD-WAYBACK-OPENAI.md): interval-censored historical evidence.
- [Full machine-readable project summary](./llms-full.txt): detailed tools, constraints, install routes, and terminology.

## Legal

- [License](https://github.com/copyleftdev/smesh-crawler/blob/main/LICENSE): Mozilla Public License 2.0.
- [Privacy](https://github.com/copyleftdev/smesh-crawler/blob/main/docs/PRIVACY.md): local state and optional provider disclosure.