Monitoring

This page covers how to monitor a Ripple cluster: Prometheus scraping configuration, the full list of pre-registered metrics, recommended alert thresholds, and structured logging.

Prometheus Scraping

Every Ripple worker and coordinator exposes metrics on an HTTP endpoint:

Kubernetes Auto-Discovery

Workers and coordinators have Prometheus annotations in their pod templates:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9102"
    prometheus.io/path: "/metrics"

With Prometheus Kubernetes service discovery, no additional scrape configuration is needed. Prometheus will automatically discover and scrape all Ripple pods.

Manual Scrape Configuration

If not using Kubernetes service discovery:

scrape_configs:
  - job_name: 'ripple-workers'
    static_configs:
      - targets:
        - 'worker-0:9102'
        - 'worker-1:9102'
        - 'worker-2:9102'
    scrape_interval: 5s

  - job_name: 'ripple-coordinator'
    static_configs:
      - targets:
        - 'coordinator-0:9202'
    scrape_interval: 10s

Pre-Registered Metrics

Graph Engine Metrics

Key metric: ripple_graph_stabilization_ns. The p99 of this histogram should stay below 10,000 ns (10 us). If it exceeds this, the worker is processing too many symbols per partition.

Transport Metrics

Key metric: ripple_transport_backpressure_level. Values above 0.8 trigger a Critical alert. Sustained values above 0.95 cause the worker to pause input consumption.

Window Metrics

Key metric: ripple_window_very_late_events_total. A non-zero rate indicates either clock skew in the data source or insufficient allowed lateness.

System Metrics

Key metric: ripple_system_heap_words. If this grows continuously, there is a memory leak. A healthy worker should show < 0.1% heap growth per checkpoint interval.

Alert Thresholds

Recommended Alert Rules

groups:
  - name: ripple
    rules:
      - alert: StabilizationSlow
        expr: histogram_quantile(0.99, ripple_graph_stabilization_ns) > 10000000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Stabilization p99 > 10ms on {{ $labels.worker }}"

      - alert: BackpressureHigh
        expr: ripple_transport_backpressure_level > 0.8
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Backpressure at {{ $value | humanizePercentage }} on {{ $labels.worker }}"

      - alert: LateEventsExcessive
        expr: >
          rate(ripple_window_very_late_events_total[5m])
          / rate(ripple_graph_stabilizations_total[5m]) > 0.01
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Very late events exceed 1% on {{ $labels.worker }}"

      - alert: HeapGrowth
        expr: >
          deriv(ripple_system_heap_words[10m]) > 10000
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Continuous heap growth on {{ $labels.worker }}"

      - alert: WorkerDown
        expr: up{job="ripple-workers"} == 0
        for: 30s
        labels:
          severity: critical
        annotations:
          summary: "Worker {{ $labels.instance }} is down"

      - alert: NoStabilizations
        expr: >
          rate(ripple_graph_stabilizations_total[5m]) == 0
          and up{job="ripple-workers"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Worker {{ $labels.worker }} has not stabilized in 5 minutes"

Threshold Summary

Structured Logging

All Ripple log entries are machine-parseable s-expressions:

type entry =
  { level : level          (* Debug | Info | Warn | Error *)
  ; timestamp_ns : Int64.t
  ; worker_id : string option
  ; component : string     (* "graph", "transport", "checkpoint", etc. *)
  ; message : string
  ; context : (string * string) list
  }
[@@deriving sexp]

Example log output:

((level Info)(timestamp_ns 1709000000000000000)(worker_id (w0))(component graph)(message stabilized)(context ((nodes 4001)(recomputed 3)(dirty 0))))

Log Levels

Parsing Logs

Because logs are s-expressions, they can be parsed with any sexp library or with standard Unix tools:

# Extract all Error-level entries
grep '(level Error)' ripple.log | sexp query '(field message)'

# Count stabilizations per worker
grep '(message stabilized)' ripple.log | grep -o '(worker_id [^)]*' | sort | uniq -c

In production, ship logs to a centralized system (ELK, Loki, Datadog) and parse the sexp structure for indexing and alerting.