SMESH CRAWLER

LOCAL MCP · DETERMINISTIC AUTHORITY · OPTIONAL AI

FIND THE NEEDLE.
NOT JUST THE PAGES.

SMESH is an agent-orchestrated web crawler that hunts exact evidence and historical changepoints—while a deterministic Rust kernel controls every fetch, budget, and stop.

01 Install local MCP Watch a crawl unfold
A crawl rendered as causality—not activity.

Polite by default. Robots-aware and origin-paced.

Local by design. Your crawl state stays on your machine.

Evidence before confidence. Models advise; policy decides.

01 · INSTALL

ONE BINARY.
FOUR DOORS.

Choose the host you already use. Every route launches the same local STDIO server and the same three bounded tools.

RUST · SOURCE INSTALL

Put the native server on your PATH.

Best for builders who want a transparent, reproducible local install.

cargo install --git https://github.com/copyleftdev/smesh-crawler --locked

AI triage is optional and off by default. Crawling works without an OpenRouter key.

02 · LIVE CRAWL

WATCH POLICY
BECOME MOTION.

The film below replays bounded crawl metadata. Every particle, fold, and rejection maps back to a causal event.

THE SIGNAL AWAKENS

Preparing the evidence field…

03 · THE METHOD

A QUESTION
WITH A STOP CONDITION.

Traditional crawlers optimize traversal. SMESH optimizes the shrinking distance between a question and defensible evidence.

  1. TRANSFORM

    Turn the question into evidence facets.

    URLs and page context become bounded candidates with cost and relevance priors.

  2. REDUCE

    Keep only what remains unresolved.

    Accepted excerpts reduce the target state. Rejected evidence never pretends to be progress.

  3. SOLVE

    Spend the next fetch where it matters.

    The frontier selects marginal coverage per cost—not whatever link happened to appear first.

  4. VERIFY

    Stop at the boundary of proof.

    Every claim retains its source, excerpt, authority, and uncertainty. Missing evidence stays missing.

04 · FETCH AUTHORITY

THE AGENT
DOES NOT HOLD
THE KEYS.

Models may rank or summarize already-fetched public material. They cannot fetch URLs, cross scope, weaken robots policy, or expand budgets.

Read the resilience model
Scope
Same-origin by default
Robots
Fail closed on network and 5xx errors
Network
SSRF checks on every redirect hop
Payload
Two-megabyte response ceiling
Concurrency
One local MCP child job by default
AI
Optional, budgeted, and advisory

05 · DIFFERENCE

USE THE RIGHT
CRAWLER FOR THE JOB.

ToolOptimizes forUse it when
wget / HTTrackTraversal and mirroringYou need a faithful local copy.
ScrapyAuthored extraction pipelinesYou know the site and schema.
PlaywrightBrowser executionThe page requires a rendered browser.
Heritrix / NutchAcquisition at scaleCoverage and archiving are the goal.
SMESH CrawlerEvidence coverage per costYou need an answer, its provenance, and a reason to stop.

READY WHEN THE QUESTION IS

CRAWL LESS.
KNOW MORE.

Choose an install routeExplore the source