Ripple
Distributed, type-safe, incremental stream processing in OCaml.
Ripple extends the core insight of self-adjusting computation — when an input changes, only the affected subgraph recomputes — from a single process to a cluster of cooperating nodes.
What makes it different
Most stream processors recompute the entire operator graph when an input arrives. Ripple doesn’t. A min-heap tracks which nodes are dirty. An incremental fold updates the running total by subtracting the old value and adding the new. The result: 250ns per stabilization at 10,000 symbols, regardless of graph size.
At a glance
| Metric | Value |
|---|---|
| Stabilization (10K symbols) | 250 ns |
| bin_prot serde roundtrip | 82 ns |
| Schema compatibility check | 128 ns |
| VWAP pipeline throughput | 2.16M events/sec |
| 6M event replay recovery | 2.1 seconds |
| Heap growth over 1M events | 0.1% |
| Crash recovery correctness | 100/100 random crash points |
The stack
16 libraries, 3 binaries, ~6,200 lines of OCaml. Built on Jane Street’s Core, Async, and bin_prot.
lib/
├── kernel/ Effect injection, domain types
├── graph/ Heap-based stabilization, incremental fold
├── schema/ Type-safe schemas, delta algebra, compatibility
├── wire/ bin_prot protocol with CRC-32C integrity
├── transport/ Sequence-ordered delta buffer
├── checkpoint/ Snapshot/restore, pluggable stores
├── window/ Tumbling, sliding, session windows
├── time_series/ Aggregators (count, sum, mean, vwap, stddev)
├── topology/ Pipeline composition
├── observability/ Prometheus metrics, W3C tracing, introspection
├── coordinator/ Consistent hashing, partition assignment
├── worker/ Lifecycle state machine
├── rpc/ Async RPC delta transport
├── connector/ File, Kafka interfaces
└── ripple/ Top-level facade