Metrics#

Kapkan exposes Prometheus metrics under the kapkan_ namespace at GET /metrics on the API listener (api.listen, conventionally 127.0.0.1:8080 — the value in the example config; the field is required, so set it). They cover the full pipeline — flow ingest, the detection engine, RTBH mitigation, notifications, and the in-kernel XDP data plane where it is enabled — so you can scrape throughput, latency, active attacks and announced routes into your existing monitoring.

Available metrics#

The mode label on kapkan_mitigate_announced_routes is the fastest way to confirm dry-run state at a glance: while dry_run: true, every would-be route is counted under mode="dry_run" and nothing under mode="real". See Mitigation and the Safety model.

The two kapkan_mitigate_dataplane_* gauges are the mitigator's account of what it installed, attributed to the bans that own it and filed under each ban's frozen dry-run flag. kapkan_dataplane_rules measures the neighbouring quantity from the other end: every rule the kernel is actually running — your config's statics plus those same dynamic rules — filed under the datapath's own flag. So under mode="real", kapkan_mitigate_dataplane_rules should track the dynamic half of kapkan_dataplane_rules.

Graph both and compare them rather than adding them up. They are two independent paths to the same number, and a lasting gap between them in real mode is a fault — a withdraw that failed, or rules the kernel expired underneath a ban that still considers itself active — which a single summed figure would hide.

Do not alert on the dry_run buckets. A dry-run ban is never handed to the installer, so nothing enters the kernel: kapkan_mitigate_dataplane_rules{mode="dry_run"} reports the rules that would have been installed, while the dynamic half of kapkan_dataplane_rules stays at zero. That gap is the intended difference between intent and measurement, and since dry_run is the shipped default it is what a new deployment will show.

The proto label is the wire protocol the flow arrived on: sflow5, netflow5, netflow9, or ipfix (e.g. kapkan_ingest_flows_total{proto="sflow5"}).

Data plane (XDP)#

The kapkan_dataplane_* family reports the in-kernel XDP data plane (dataplane.enabled: true). Most of these series are created as the data plane writes them, so on a host that mitigates through BGP alone they are absent rather than zero.

Much of the family is lifecycle, and that is deliberate. An XDP program that is not attached looks exactly like one that is — the daemon is up, the API answers, bans are recorded — and the only visible difference is that the packets you asked to drop are not being dropped. So "is it attached" is a metric rather than something to infer from a log line at boot.

Terminal verdicts and observations#

With the two split, exactly one terminal verdict is counted per packet, so

sum(rate(kapkan_dataplane_packets_total[1m]))

is packets through the datapath, and

sum(rate(kapkan_dataplane_packets_total{verdict=~"drop_.*"}[1m]))
  / sum(rate(kapkan_dataplane_packets_total[1m]))

is the drop rate. Neither query should ever include kapkan_dataplane_observations_total.

During a dry run the observation to watch is dryrun_would_drop: it counts the packets that would have been dropped, and it is the whole argument for turning dry_run off.

sum(rate(kapkan_dataplane_observations_total{kind="dryrun_would_drop"}[1m]))

The filter-bypass alarm#

kapkan_dataplane_filter_bypass_packets_total{reason="ipv6_exthdr_cap"} counts packets the data plane forwarded without evaluating a single rule. Every other pass_* verdict means the rules ran and none of them said drop; this one means the rules never ran at all — allow lists, drop rules and rate limits alike. For those packets the filter was, in effect, switched off.

The cause is a parse budget: the datapath walks at most eight IPv6 extension headers, and a packet carrying more is passed on unexamined. That it is passed and not dropped is deliberate — a parse limit must never become a default-deny — which means this counter is the entire mitigation. Operating & monitoring covers the mechanism and what to do about it.

No legitimate traffic chains eight extension headers, so the alert threshold is zero rather than some rate you tune:

sum(rate(kapkan_dataplane_filter_bypass_packets_total[5m])) > 0

Both series are published even while nothing has been bypassed, so a healthy data plane shows an explicit 0 rather than "No data" — that is how you can tell the alert is wired up at all.

Counters across a restart#

kapkan_dataplane_packets_total and kapkan_dataplane_bytes_total start at zero with the process, but the kernel maps they are read from do not — a process that adopts an existing pin set inherits the previous one's totals. The scraper seeds its baseline from the first read and publishes only what this process observed, so rate() is correct at startup instead of showing a spike as wide as the previous process's whole lifetime. The absolute kernel totals are on /api/v1/status if you need them.

The same reseeding happens when the counters go backwards, which can only mean they were reset underneath the process (pins rebuilt, a map recreated). A rebuild therefore shows up as a gap in the rate, never as a negative spike.

Map sizing and memory#

kapkan_dataplane_map_entries and kapkan_dataplane_map_bytes are the feedback loop for dataplane.limits. BPF map memory is charged to the unit's memory cgroup in one step at load, so a MemoryMax= that was fine before the data plane existed can OOM the unit at startup. Two maps dominate the footprint — kapkan_rl_src4 and kapkan_rl_src6, sized from dataplane.limits.max_ratelimit_sources — and being LRU hashes they are pre-allocated in full whether or not a single source is ever rate-limited. Lower the limit and read the result off these gauges rather than inferring it:

sum(kapkan_dataplane_map_bytes)

Per-ban drop counts#

How much the kernel dropped for one victim is deliberately not here. /metrics is unauthenticated, and a victim address in a label would publish who is under attack to anyone who can reach the listener. Those per-ban and per-rule counters are on the authenticated /api/v1/bans instead, under each ban's dataplane object.

Fingerprint plane#

The fingerprint plane (dataplane.fingerprint.enabled: true) reports its userspace reader here; its in-kernel copy sampler is on kapkan_dataplane_observations_total{kind="fp_emitted"|"fp_throttled"|"fp_ring_full"} above.

fp_throttled climbing while fp_emitted plateaus is the sampler doing its job under a handshake flood; a sustained fp_ring_full means the reader cannot drain the ring fast enough.

Edge nodes#

An edge node exposes its own series on status_listen (/metrics), beside the Go runtime's:

The brain adds nothing per edge node to its own /metrics; a node's liveness and last report are read from GET /api/v1/edge/nodes, and the zones' rates and would-be sets across the fleet from GET /api/v1/edge/zones/status.

Scraping#

Add Kapkan as a scrape target in your Prometheus config:

scrape_configs:
  - job_name: kapkan
    metrics_path: /metrics
    static_configs:
      - targets: ["127.0.0.1:8080"]

Use the host and port from your api.listen value (conventionally 127.0.0.1:8080). If you expose the listener beyond localhost, set an API token — /metrics stays reachable without it, so your scraper needs no credentials.

To inspect the raw output directly:

curl -s localhost:8080/metrics | grep kapkan_

Grafana dashboard#

An official Grafana dashboard for the kapkan_* set is available to download directly at kapkan-overview.json (its source lives in the repository under engine/deploy/grafana/). It covers the full pipeline — ingest flow/datagram/decode rates and dropped flows, active and started attacks, tracked hosts, hot-path latency quantiles, announced routes and FlowSpec rules by mode, bans rejected by reason, mitigation fallbacks, and notification and storage results. A Data plane row covers the XDP metrics above: packets per second by verdict, rules in the kernel by mode, attachment by interface and mode, attach errors, policy generation and apply-time quantiles, and BPF map entries.

In Grafana, go to Dashboards → New → Import, upload the JSON, and pick your Prometheus data source when prompted (it is templatized as DS_PROMETHEUS). The dashboard targets Grafana 10+ and is import-verified against Grafana 11. See the README next to it for provisioning.

Related#

• REST API — the data endpoints served alongside /metrics.
• In-kernel data plane — the XDP backend behind the kapkan_dataplane_* set.
• Storage (ClickHouse) — the optional persistence behind kapkan_storage_rows_total.
• Deployment — running Kapkan as a service and choosing a listen address.