Installation

Prerequisites

Why the C/Go toolchain?

Palimpsest uses wreq with BoringSSL for TLS fingerprint impersonation. BoringSSL is compiled from source during cargo build, which requires CMake, a C compiler, and Go.

macOS

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# Install build dependencies (Xcode command line tools + CMake + Go)
xcode-select --install
brew install cmake go

# Clone and build
git clone https://github.com/copyleftdev/palimpsest.git
cd palimpsest
cargo build --release

# Verify
./target/release/palimpsest --help

Chrome for browser capture:

# Chrome is usually at:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --version

# Or install via Homebrew:
brew install --cask google-chrome

Linux (Ubuntu/Debian)

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# Install build dependencies
sudo apt update
sudo apt install -y build-essential cmake golang-go pkg-config libclang-dev

# Clone and build
git clone https://github.com/copyleftdev/palimpsest.git
cd palimpsest
cargo build --release

# Verify
./target/release/palimpsest --help

Chrome for browser capture:

# Install Chrome
wget -q -O - https://dl.google.com/linux/linux_signing_key.pub | sudo gpg --dearmor -o /usr/share/keyrings/google-chrome.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/google-chrome.gpg] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee /etc/apt/sources.list.d/google-chrome.list
sudo apt update && sudo apt install -y google-chrome-stable

# Verify
google-chrome --version

Linux (Fedora/RHEL)

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env

# Install build dependencies
sudo dnf install -y gcc gcc-c++ cmake golang clang-devel pkg-config

# Clone and build
git clone https://github.com/copyleftdev/palimpsest.git
cd palimpsest
cargo build --release

Linux (Arch)

sudo pacman -S rust cmake go clang pkg-config
git clone https://github.com/copyleftdev/palimpsest.git
cd palimpsest
cargo build --release

Windows

Option A: Native (MSVC)

# 1. Install Rust from https://rustup.rs/ (choose MSVC toolchain)

# 2. Install Visual Studio Build Tools (C/C++ workload)
#    https://visualstudio.microsoft.com/visual-cpp-build-tools/

# 3. Install CMake
#    https://cmake.org/download/ (add to PATH during install)

# 4. Install Go
#    https://go.dev/dl/ (add to PATH during install)

# 5. Clone and build
git clone https://github.com/copyleftdev/palimpsest.git
cd palimpsest
cargo build --release

# 6. Verify
.\target\release\palimpsest.exe --help

Option B: WSL2 (recommended)

Windows Subsystem for Linux gives you a full Linux environment. Follow the Linux (Ubuntu/Debian) instructions above inside WSL2:

# Install WSL2 with Ubuntu
wsl --install -d Ubuntu

# Then inside the WSL2 terminal, follow the Linux instructions

Option C: Docker (any platform)

If you don’t want to install build tools, Docker works on all platforms:

docker build -t palimpsest .
docker run palimpsest --help
docker run -v ./output:/data palimpsest crawl https://example.com -d 2 -o /data

See Docker Deployment for the full compose setup.

Verifying the Build

After building, you should see all 10 subcommands:

$ palimpsest --help
Usage: palimpsest <COMMAND>

Commands:
  crawl           Start a crawl with seed URLs
  replay          Reconstruct a captured URL from artifacts
  history         Show capture history for a URL
  extract         Extract text and RAG chunks from captured content
  shadow-compare  Compare against legacy crawler WARC files
  serve           Start a distributed frontier server
  worker          Connect to a frontier server and crawl
  api             Start the retrieval API server
  stats           Print workspace statistics
  migrate         Run storage migrations

Running the Test Suite

# All tests (288, excludes long-running scale tests)
cargo test --workspace

# Simulation framework only
cargo test -p palimpsest-sim --test simulation_tests

# Scale tests (1K + 5K pages, ~90 seconds)
cargo test -p palimpsest-sim --test scale_test

# Stress test (10K pages)
cargo test -p palimpsest-sim --test stress_test

# Stealth regression tests (requires Chrome + network)
cargo test -p palimpsest-fetch --test stealth_test -- --ignored --nocapture --test-threads=1

Troubleshooting

BoringSSL build fails

The most common build issue. Check:

cmake --version   # Need 3.x+
go version        # Need 1.19+
clang --version   # Or gcc — need a C compiler

On macOS, ensure Xcode command line tools are installed: xcode-select --install

On Windows, ensure Visual Studio Build Tools include the “Desktop development with C++” workload.

Chrome not found (browser capture)

Palimpsest looks for Chrome/Chromium in PATH. If installed in a non-standard location:

# macOS — add to PATH
export PATH="/Applications/Google Chrome.app/Contents/MacOS:$PATH"

# Windows — add to PATH
set PATH=%PATH%;C:\Program Files\Google\Chrome\Application

openssl-sys linker errors

Palimpsest uses BoringSSL (via wreq), not OpenSSL. If you see openssl-sys errors, another dependency may be pulling it in. Check with:

cargo tree -i openssl-sys

If present, the boring-sys2 crate’s prefix-symbols feature should prevent symbol conflicts on Linux. On macOS this is not typically an issue.