Storage (ClickHouse)#

Point Kapkan at a ClickHouse server to keep attack, traffic, audit and edge history — the answer to "what hit us last Tuesday". Without it, Kapkan runs entirely in-process on live data.

You need a running ClickHouse server (any recent version) for this. Kapkan talks to its HTTP interface, which listens on port 8123 by default — that is the port in the example URL below. Installing ClickHouse is out of scope here; see clickhouse.com.

Enabling#

storage:
  clickhouse:
    url: "http://127.0.0.1:8123"   # empty/absent disables persistence
    database: "kapkan"             # created if absent
    username_env: "KAPKAN_CH_USER" # optional; credentials come from the env
    password_env: "KAPKAN_CH_PASS"
    ttl_days: 7                    # rows auto-expire (per-row TTL)
    flush_interval_seconds: 5
    batch_size: 1000
    queue_size: 100000
    traffic_interval_seconds: 10

Credentials are read from the named environment variables (username_env / password_env), never from the config file. Omit both if your ClickHouse accepts unauthenticated local connections (the default default user often does) — set them only when your server requires auth.

Changing any storage setting requires a full restart — a config reload that touches the storage block is rejected (reload: storage settings cannot change at runtime (restart required)).

Verify#

After enabling storage, confirm rows are actually landing — persistence is best-effort and fails quietly. Two checks:

• Watch the kapkan_storage_rows_total{result="written"} metric climb (vs result="dropped" or result="error"). See Metrics.
• Query ClickHouse directly:

  curl 'http://127.0.0.1:8123/?query=SELECT%20count()%20FROM%20kapkan.attack_events'
  

On a bad credential or permission problem the engine logs clickhouse schema init failed at startup and clickhouse insert failed when a batch is rejected — check the logs first if counts stay at zero.

How it works#

Kapkan talks to ClickHouse's HTTP interface with the Go standard library — no driver dependency. The only external dependency is the ClickHouse server itself.

On start it idempotently creates six MergeTree tables:

Each carries a ttl_days per-row TTL, so retention is bounded without operator intervention.

The writer credential needs CREATE (database + tables) and INSERT rights on first run — and again on the first start of a release that adds tables (this one adds the three edge_* tables); between those, INSERT is enough. Schema init attempts every statement and logs each refusal on its own (a credential that may INSERT but not CREATE sees those lines on every start; they are harmless once the tables exist), so the three edge tables and the column upgrades below are tried even when an earlier statement was refused, and an upgrade whose credential can no longer CREATE keeps the tables it always had. If edge history table not created appears at start, run one start with a credential that may CREATE, or create the table yourself with the statement in engine/internal/storage/edge_rows.go.

The attack_events table's reason column holds the compact JSON of the detection reason — why the attack fired (threshold provenance, warm-up, protocol shares) — and is empty on the attack-ended row. Existing deployments gain the column automatically: schema init runs an ADD COLUMN IF NOT EXISTS, skipped silently if the writer credential lacks ALTER rights (fresh installs already have it).

Edge history#

The three edge_* tables keep what the edge zone status shows for ten seconds — per zone, over time. They are filled from the edge nodes' advisory reports by the brain's write path, so every row is a node's claim; the brain sums, never acts on it. The rules below are the write path's contract.

• Raw windows, bucketed on read. edge_windows holds the nodes' ten-second windows as they came (ORDER BY (zone, ts, node)); a history query sums them into buckets with toStartOfInterval, exactly like traffic. There is no rps column — it is sum(requests) / sum(window_seconds) — and no compaction on the brain.
• Two clocks. ts is the window's close by the node's clock when it is within ten minutes behind or a minute ahead of the brain's; otherwise the brain's clock is written instead and one clock_skew event records the finding. received_at is always the brain's clock.
• No tenant column. Which tenant owns a zone is read from the live zones file at query time (zones[].tenant) — one truth, so re-labelling a zone leaves no stale rows.
• Only the telling sources. edge_sources stores a source only when a node denied it, challenged it, or previewed either (would-deny, would-challenge) — at most 20 per node, zone and window: the would-be sources first, then the refused and challenged ones, the busiest within each tier. Sources that were allowed, cleared or merely marked are visitors and are never written.
• Nothing for quiet zones. A zone in policy.mode: none, or a deciding zone that closed no window lately (the report carries it with no close time and nothing counted), writes no row and counts as nothing. A certificate issuance leaves a trace even on an idle zone: the CA's HTTP-01 probes are requests the node counts (a few 2xx per validation, decided 0).
• One row per window. A report the node re-sends — the same node, zone and close time — is written once, and a report carries one window per zone. A window for a zone the zones file does not have, a window (or a certificate, a challenge) for a zone the reporting node's placement does not cover, a window with counters but no close time, or a second window for one zone in one report is skipped; every skipped part counts in kapkan_edge_history_dropped_total{reason} (unknown_zone, outside_scope, no_at, duplicate, extra_window, bad_source, source_cap) — while storage is on; with it off nothing is written and nothing is counted. The report itself is still accepted and shown live; only the history skips the part. duplicate is expected once per burst end when a node reports more often than its windows close.
• Transitions as events. edge_events records what changed between two reports of a node — its version, dry-run, the rendered document, a generation installed or refused, the terminator alive or not, its HTTP/3 state, a certificate issued, renewed or gone, a zone-wide challenge started or ended, a clock skew found or recovered, a report that had to shrink — each once per change, and, from the polls, node_alive / node_lost (the loss stamped when it happened: the last poll plus stale_after). The first report after a brain start is the baseline and writes no events — except clock_skew, which is about the window's clock and is written on a first report too when that clock is off. A certificate or a challenge is recorded for a zone the zones file has, and a report that had to shed its tail (report_truncated) is not read as "gone" for what it no longer lists. The presence ticker runs every min(stale_after/2, 5 s), never under a second; after a brain start a node is baselined at its first poll (silently), or as lost once stale_after has passed without one — a restart is never a fleet-wide node_alive burst. That second baseline is silent too: a node the brain has not heard from since it started gets no node_lost row, because nothing transitioned — a node that was already down across a restart appears in the events only when it comes back, as node_alive. The presence transitions are also logged at INFO, storage or no storage.

Volume, per day: edge_windows grows by 8 640 × (deciding zones with traffic × alive nodes) rows (one ten-second window each), edge_sources by at most 20 times that, and edge_events by the handful of transitions a fleet has. On disk that is small: an acceptance run measured edge_windows at roughly a hundred compressed bytes per row and the other two at tens of bytes — an order of magnitude to plan with, not a figure to multiply, since the sample was a few dozen rows in tiny parts (which compress badly) and your zone and node names are not ours. Measure what it actually costs on your server:

SELECT table, sum(rows) AS rows, formatReadableSize(sum(bytes_on_disk)) AS on_disk
FROM system.parts
WHERE database = 'kapkan' AND active
GROUP BY table ORDER BY table

Retention is ttl_days, like every other table. The history is read through GET /api/v1/edge/history (buckets), …/edge/history/sources (who was denied, challenged or would have been, over a period) and GET /api/v1/edge/events (the transitions) — and the tables are yours to query directly too.

Backpressure#

Rows go onto a bounded queue (queue_size) with a non-blocking send and are flushed in batches (batch_size / flush_interval_seconds). Rows are lost rather than held whenever ClickHouse cannot take them, and the engine keeps running; kapkan_storage_rows_total says which of the two losses it was, and they do not mean the same thing:

• result="error" — the insert itself failed. The batch is logged (clickhouse insert failed) and discarded. A ClickHouse that is down counts here, not under dropped: a refused connection fails immediately, so each flush empties the queue as fast as it fills.
• result="dropped" — the queue was full when a row was handed over, so it never reached an insert. That needs a sink that stalls rather than refuses — a server holding the connection, a network dropping the packets, a flush waiting out its timeout — or a burst larger than queue_size arriving inside one flush. Detection is never blocked either way.

A dropped series with no error beside it therefore points at latency, not at an outage. See Metrics.

What is and isn't persisted#

The traffic table currently persists per-host snapshots only. When the optional geoip block is configured (a MaxMind .mmdb database), Kapkan resolves per-source ASNs and persists each attack's top ASNs in attack_events.top_asns (pipe-joined "AS<n> <org>", empty when geoip is off). Per-ASN time-series and per-hostgroup totals are not yet snapshotted — both are candidates for a follow-up.

Related#

• Configuration reference — the full storage key list.
• Audit log — the audit_events table and its read endpoint.
• Metrics — the kapkan_storage_rows_total counter.