WARC++ Format

WARC++ extends ISO 28500 (the standard WARC format) with structured metadata for execution context, DOM snapshots, resource graphs, and timing breakdowns. Standard WARC readers can parse the basic records. Palimpsest-aware readers get the full execution context.

Standard Record Types

These follow ISO 28500 exactly:

Type	Purpose
`warcinfo`	Crawl-level metadata
`request`	HTTP request
`response`	HTTP response
`resource`	Standalone resource
`metadata`	Additional metadata

Extension Record Types

Type	Purpose
`envelope`	Full `ExecutionEnvelope` (seed, timestamp, DNS, TLS, browser config)
`dom-snapshot`	Rendered DOM state after JavaScript execution
`resource-graph`	Dependency graph of all resources loaded for a page
`timing`	Detailed timing breakdown (DNS, connect, TLS, TTFB, transfer, render)

Content Hash Header

Every record includes a Palimpsest-Content-Hash header:

Palimpsest-Content-Hash: blake3:af1349b9f5f9a1a6a0404dea36dcc949...

This enables content-addressable retrieval and integrity verification without reading the full record.

Envelope Record Example

WARC/1.1
WARC-Type: envelope
WARC-Record-ID: <urn:uuid:a1b2c3d4-...>
Content-Type: application/json
Palimpsest-Envelope-Version: 1
Palimpsest-Content-Hash: blake3:c7d2fe...

{
  "seed": 42,
  "timestamp": {"wall": "2026-04-12T10:30:00.123456789Z", "logical": 1234},
  "request_headers": [["User-Agent", "PalimpsestBot/0.1"]],
  "dns_snapshot": {"host": "example.com", "addrs": ["93.184.216.34"], "ttl": 300},
  "tls_fingerprint": {"protocol": "TLSv1.3", "cipher": "TLS_AES_256_GCM_SHA384", "cert_chain_hash": "blake3:..."},
  "browser_config": null
}

Resource Graph Record Example

{
  "root": "https://example.com/",
  "resources": [
    {"url": "https://example.com/style.css", "type": "stylesheet", "hash": "blake3:...", "initiated_by": "https://example.com/"},
    {"url": "https://example.com/app.js", "type": "script", "hash": "blake3:...", "initiated_by": "https://example.com/"}
  ],
  "load_order": [0, 1]
}

Serialization Rules

Rule	Value
Text encoding	UTF-8
JSON format	Compact (no pretty-print)
Timestamps	RFC 3339 with nanosecond precision
Record separator	CRLFCRLF (per ISO 28500)
Max payload	4 GiB (WARC spec limit)

Backward Compatibility

Standard WARC tools (warc-tools, warcio, pywb) can read the request, response, warcinfo, resource, and metadata records without modification. They skip extension records (envelope, dom-snapshot, resource-graph, timing) per the WARC spec’s extension handling rules. The Palimpsest-* headers are ignored by non-Palimpsest readers.

Keyboard shortcuts

Palimpsest Documentation