Linux packet-drop debugging
Find where the packet disappeared. Keep the evidence.
A bounded workflow for tracing an SKB through Linux kernel functions, TC, XDP, tunnels, transforms, and drops—then replaying the route without root.
The evidence lifecycle
Capture first. Explain second.
Start with a host fact, not a packet-loss theory. skbx discovers
target-kernel types through BTF, records ordered observations in
an append-only JSONL stream, and assigns stable event:
handles. Replay groups those events into route:
handles. Explanation retrieves existing evidence; it does not
generate new events.
Install the pinned source graph.
You need Rust 1.85 or newer, Clang/LLVM with the BPF backend,
bpftool, libelf and libpcap development packages,
and a kernel exposing /sys/kernel/btf/vmlinux.
cargo install --git https://github.com/copyleftdev/skbx --locked skbx-cliAsk the host what is possible.
doctor reports BTF, tracefs, symbol visibility,
frame-pointer readiness, privilege, and required tools as
machine-readable evidence. Fix failed prerequisites before
interpreting a missing packet as a networking result.
skbx doctor --jsonResolve probes before attachment.
A plan is deterministic and side-effect free. Begin with the
function closest to the reported boundary—such as
ip_rcv for IPv4 receive—and inspect the exact
attachment decision before loading eBPF.
skbx plan --probe ip_rcv --jsonBound the observation window.
Capture a short, intentional interval and retain the native JSONL. The duration, parser state, output path, and reliability counters are part of the evidence contract.
sudo skbx capture --probe ip_rcv \
--duration 10 \
--output trace.jsonl \
icmp
Replace icmp with a narrow libpcap expression for
the incident traffic. Use --filter-track-skb when
the investigation depends on clone, copy, COW, or frame-to-SKB
lineage.
Read the footer before the story.
A valid native stream begins with capture_start and
ends with capture_end. Treat a missing footer—or
non-zero reserve, recursion, read, decode, enrichment, or output
failures—as an incomplete observation.
tail -n 1 trace.jsonl | jq .Rebuild routes without privilege.
Replay is deterministic and rootless. It groups ordered events
by packet identity, reports consensus and outliers, and emits a
stable route: handle.
skbx replay trace.jsonl --format json
Select any event: handle from the capture to retrieve
its event plus nearby evidence sharing the same packet identity:
skbx explain trace.jsonl event:<handle>Turn boundaries into questions.
The load balancer sent it; the pod did not see it.
Trace the host receive path, namespace and interface evidence, then compare the last observed kernel hook with the expected veth or tunnel boundary.
XDP or TC changed the outcome.
Capture program identity and paired entry/exit evidence. Decode XDP actions instead of inferring them from a later absence.
Encapsulation hid the inner packet.
Record tunnel evidence and apply independent outer and inner-L2 or inner-L3 predicates so the filter itself stays auditable.
The route looks incomplete.
Check the reliability footer first, then widen the probe plan or observation window. Never fill a gap with an invented event.
Continue the investigation
Inspect the evidence model.
skbx is inspired by pwru and extends the workflow with stable handles, a versioned native stream, deterministic replay, and an explicit agent contract. The repository keeps the detailed capability comparison and validation evidence.