GeezerVison

You have Telegraf writing host metrics into InfluxDB and Kapacitor sitting next to it doing nothing. This TICKscript tutorial wires the two ends together with tasks that watch CPU, disk and memory, plus a Kapacitor deadman check that notices when a host goes quiet. The alerts land in three places: a Slack incoming webhook, an email inbox and a plain HTTP POST to ntfy. Every Kapacitor alert example in this article fired for real on a lab box called hooks-api.

Original content from computingforgeeks.com - post 188762

We build it the way we’d run it in production. Tasks publish to one topic, topic handlers decide who hears about it, a template turns one threshold script into many tasks, and record and replay lets you test a changed alert without stressing a server again. The errors we hit along the way are in their own section at the end. We confirmed every path below working with Kapacitor OSS 1.8.9, InfluxDB 1.13.2 and Telegraf 1.40.3 on Ubuntu 26.04 in November 2026.

Follow the alert path before wiring it

Kapacitor sits between your metrics and your notification channels, and the only part that is worth drawing is the hand-off in the middle. Stream tasks get every point InfluxDB receives through a subscription, batch tasks pull with an InfluxQL query on a schedule, and both push alert events to a named topic. The topic handlers then fan each event out to Slack, SMTP or an HTTP endpoint.

That split matters more than it looks. If every TICKscript names its own handlers, changing the on-call email means editing and redefining every task. With a topic in the middle, the tasks never change: you swap a handler file and every alert that publishes to host-alerts follows. The table lists the four tasks this guide builds and where each one sends its events.

Task	Type	Watches	Sends to
cpu_alert	stream	cpu-total busy %, 30s mean	Slack and email handlers inside the TICKscript
disk_root	stream, from a template	used_percent of /	topic host-alerts
mem_batch	batch	mean mem used_percent over 2m	topic host-alerts
deadman	stream	points per minute from the host	topic host-alerts

We kept the CPU task on direct handlers on purpose, so you can see both styles and the difference in how they’re managed. In your own setup we would put everything on topics once the first task works.

The diagram is the lab as built. cpu_alert sits outside the topic on purpose, so the article can show both ways of attaching handlers.

Set up the receivers on the Kapacitor host

This side of the wiring is the receivers. You need a working TICK stack first, with Telegraf writing to the telegraf database and Kapacitor subscribed to InfluxDB. Our TICK stack install on Ubuntu sets that up from the signed InfluxData repository, and it is what this lab ran on. On a fresh server, the Ubuntu 26.04 server baseline covers the user, firewall and time sync first.

We did not have a Slack workspace to post into from the lab, so the Slack side is a local stand-in that accepts the same POST a Slack incoming webhook does, logs the JSON body and answers ok. Kapacitor cannot tell the difference, and the payloads you’ll see below are exactly what Slack would receive. To point at real Slack, you change one URL. The setup order is:

1. Install the tools the tests use: stress-ng for load, jq for reading JSON logs, Postfix and mailutils for local mail.
2. Start the webhook receiver on 127.0.0.1:9000 as a systemd service.
3. Run ntfy on 127.0.0.1:8080 as the generic HTTP POST target.
4. Point Postfix at loopback only and alias oncall to a local mailbox.

Ubuntu ships ntfy in its own archive, so everything comes from apt. Postfix asks a configuration question during install; pick “Local only”.

sudo apt-get install -y stress-ng jq postfix mailutils ntfy

The webhook receiver is a short Python script with no dependencies outside the standard library, so it runs on the stock python3 that Ubuntu already has. Create it with your editor.

sudo vim /usr/local/bin/hook-receiver.py

Paste the receiver below. It writes each POST body, pretty printed, to /var/log/hook-receiver.log with a timestamp and the request path, which is all we need to compare what Kapacitor sent against what Slack expects.

#!/usr/bin/env python3
# Stand-in for a Slack incoming webhook: logs every POST body and answers "ok" like Slack does.
import json, sys
from datetime import datetime, timezone
from http.server import BaseHTTPRequestHandler, HTTPServer

LOG = "/var/log/hook-receiver.log"

class Hook(BaseHTTPRequestHandler):
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
        with open(LOG, "a") as f:
            f.write(f"--- {datetime.now(timezone.utc).isoformat(timespec='seconds')} POST {self.path}\n")
            try:
                f.write(json.dumps(json.loads(body), indent=2) + "\n")
            except ValueError:
                f.write(body.decode(errors="replace") + "\n")
        self.send_response(200)
        self.send_header("Content-Type", "text/plain")
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, *args):
        pass

port = int(sys.argv[1]) if len(sys.argv) > 1 else 9000
HTTPServer(("127.0.0.1", port), Hook).serve_forever()

Next, give it a unit file so it survives reboots and restarts on failure, because a receiver that quietly died halfway through testing looks exactly like a Kapacitor that never sent anything.

sudo vim /etc/systemd/system/hook-receiver.service

The unit runs the script on port 9000, the port the Slack URL in kapacitor.conf points at later.

[Unit]
Description=Local webhook receiver for Kapacitor alert tests
After=network.target

[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/hook-receiver.py 9000
Restart=on-failure

[Install]
WantedBy=multi-user.target

For ntfy, append two lines to /etc/ntfy/server.yml so it listens on loopback only: base-url: "http://hooks-api:8080" and listen-http: "127.0.0.1:8080". If you’d rather run ntfy as a real phone notification server, our self-hosted ntfy with Docker Compose guide adds HTTPS and auth, and the Kapacitor side stays the same. Then start both services:

sudo systemctl daemon-reload
sudo systemctl enable --now hook-receiver ntfy

Postfix only needs to accept mail from localhost and deliver it locally. These settings name the host, keep the listener on loopback and route oncall to the eugene mailbox. Replace eugene with your own login user here and in the mailbox path later.

sudo postconf -e myhostname=hooks-api.lab.example myorigin=hooks-api.lab.example "mydestination=hooks-api.lab.example, localhost" inet_interfaces=loopback-only
echo "oncall: eugene" | sudo tee -a /etc/aliases
sudo newaliases
sudo systemctl restart postfix

With a real mail relay you’d skip Postfix entirely and point Kapacitor at the relay’s host, port and credentials in the next step. The rest of the article does not care which one you use.

Connect Slack, SMTP and ntfy in kapacitor.conf

Now the Kapacitor side. All three channels are configured in /etc/kapacitor/kapacitor.conf, and the package ships each one disabled. Keep a copy of the original first, because a diff against it is the quickest way to review what you changed later.

sudo cp /etc/kapacitor/kapacitor.conf /etc/kapacitor/kapacitor.conf.orig
sudo vim /etc/kapacitor/kapacitor.conf

In the [smtp] section, enable it and set the sender and the default recipient. These are the keys we changed; leave the rest of the section as shipped.

[smtp]
  enabled = true
  host = "localhost"
  port = 25
  username = ""
  password = ""
  from = "[email protected]"
  to = ["[email protected]"]
  no-verify = true

That no-verify = true line is there because our local Postfix offers STARTTLS with a certificate for hooks-api.lab.example while Kapacitor connects to localhost, and the mismatch silently killed every email. The troubleshooting section shows the log line. With a real relay and a valid certificate, leave it at false and use the relay’s hostname in host.

Find the commented [[httppost]] example and add a real block below it. The endpoint name is what TICKscripts and topic handlers refer to, and alert-template turns the full alert JSON into the one-line body ntfy shows as the notification text.

[[httppost]]
  endpoint = "ntfy"
  url = "http://127.0.0.1:8080/tick-alerts"
  headers = { Title = "Kapacitor", Tags = "rotating_light" }
  alert-template = "{{.Level}} {{.Message}}"

The Title and Tags headers are ntfy’s own: the title shows above the message and the tag becomes an emoji on a phone. Last, the [[slack]] block. With a real workspace, the url is the incoming webhook URL Slack gives you; ours points at the local receiver with a fake path in the same shape.

[[slack]]
  enabled = true
  default = true
  workspace = "lab"
  url = "http://127.0.0.1:9000/services/T000/B000/XXXX"
  token = ""
  channel = "#alerts"
  global = false

Restart Kapacitor and ask it to test each service. A service test sends a sample message through the configured channel without needing a task.

sudo systemctl restart kapacitor
kapacitor service-tests smtp slack httppost

SMTP and Slack pass. The httppost test fails, and the reason is worth knowing before you go hunting for a firewall problem.

Service             Success   Message
smtp                true
slack               true
httppost            false     failed to POST alert data: Post "http://localhost:3000/": dial tcp 127.0.0.1:3000: connect: connection refused

The CLI test ignores your [[httppost]] block and posts to its built-in default, http://localhost:3000/. To test the real endpoint, call the service-test API and pass the endpoint and URL yourself.

curl -s -X POST -H 'Content-Type: application/json' \
  -d '{"endpoint":"ntfy","url":"http://127.0.0.1:8080/tick-alerts"}' \
  http://localhost:9092/kapacitor/v1/service-tests/httppost

This time ntfy received the POST and Kapacitor reported success, which proves the URL in the endpoint block is reachable from the Kapacitor process.

{"success":true,"message":""}

Don’t read too much into the body ntfy got from that test: it’s the raw alert JSON with empty fields, because the service test doesn’t apply alert-template. Real alerts later in the article arrive as the clean one-liner the template promises. Also, a green SMTP test only means Kapacitor handed the message to its mail client. Check /var/mail/eugene (or your relay’s log) before trusting it.

That is the terminal from the lab run of the three service tests. All three green is the starting point, not proof of delivery.

Write a stream task that pages on CPU

With the channels working, the first task watches CPU on the stream side. Telegraf reports usage_idle per core plus a cpu-total series, so the script filters to the total, averages idle over a sliding 30 second window and turns it into a busy percentage. Save it as cpu_alert.tick in your home directory.

vim ~/cpu_alert.tick

This is the exact script we defined. Note the single quotes around strings and the double quotes around field and tag names; TICKscript treats those two very differently.

// Stream task: total CPU busy per host, averaged over a 30s window.
dbrp "telegraf"."autogen"

stream
    |from()
        .measurement('cpu')
        .where(lambda: "cpu" == 'cpu-total')
        .groupBy('host')
    |window()
        .period(30s)
        .every(10s)
    |mean('usage_idle')
        .as('idle')
    |eval(lambda: 100.0 - "idle")
        .as('used')
    |alert()
        .id('{{ index .Tags "host" }}/cpu')
        .message('{{ .ID }} is {{ .Level }}: CPU {{ index .Fields "used" | printf "%.1f" }}% busy')
        .warn(lambda: "used" > 70)
        .crit(lambda: "used" > 90)
        .stateChangesOnly()
        .log('/var/log/kapacitor/cpu_alert.log')
        .slack()
        .email()

Two lines carry most of the behaviour. .stateChangesOnly() means you get one message when the level changes, not one every 10 seconds while the box is hot. And because the dbrp line is inside the file, the define command doesn’t need a -dbrp flag. Define the task, enable it and confirm that it is executing.

kapacitor define cpu_alert -tick ~/cpu_alert.tick
kapacitor enable cpu_alert
kapacitor list tasks

The task shows as an enabled stream task bound to telegraf.autogen, with Executing set to true once the enable has gone through.

ID        Type      Status    Executing Databases and Retention Policies
cpu_alert stream    enabled   true      ["telegraf"."autogen"]

If Executing reads false, run kapacitor show cpu_alert and look at the Error: line near the top. Now load both vCPUs for 100 seconds with stress-ng, which is long enough for the 30 second mean to climb through both thresholds and then fall back.

sudo stress-ng --cpu 2 --timeout 100s

Read the task’s own log with jq to see the transitions in order:

sudo jq -c '{time, level, message}' /var/log/kapacitor/cpu_alert.log

Three lines, one per state change. WARNING landed 10 seconds before CRITICAL because the window was still averaging in idle samples, and OK came 80 seconds later once the load had stopped and the window had drained.

{"time":"2026-11-12T05:44:20Z","level":"WARNING","message":"hooks-api/cpu is WARNING: CPU 77.2% busy"}
{"time":"2026-11-12T05:44:30Z","level":"CRITICAL","message":"hooks-api/cpu is CRITICAL: CPU 100.0% busy"}
{"time":"2026-11-12T05:45:50Z","level":"OK","message":"hooks-api/cpu is OK: CPU 56.9% busy"}

The same three events went to Slack. This is the CRITICAL payload as the webhook receiver logged it, which is the body Slack’s incoming webhook would get. Kapacitor sends the message as an attachment and sets color from the level: danger for CRITICAL, warning for WARNING and good for OK, so the bar beside each message changes colour in the channel.

--- 2026-11-12T05:44:36+00:00 POST /services/T000/B000/XXXX
{
  "attachments": [
    {
      "fallback": "hooks-api/cpu is CRITICAL: CPU 100.0% busy",
      "color": "danger",
      "text": "hooks-api/cpu is CRITICAL: CPU 100.0% busy",
      "mrkdwn_in": [
        "text"
      ]
    }
  ],
  "channel": "#alerts",
  "icon_emoji": "",
  "text": "",
  "username": ""
}

Email arrived too, with the alert message as the subject. The body is where it falls down. Without a .details() template, Kapacitor puts the whole alert data structure in the body as HTML-escaped JSON, quoted-printable encoded on top, which is unreadable on a phone.

Subject: hooks-api/cpu is WARNING: CPU 77.2% busy
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

{"Name":"cpu","TaskName":"cpu_alert","G=
roup":"host=3Dhooks-api","Tags":{"host":"ho=
oks-api"},"ServerInfo":{"Hostname":"localhost",=

We fix that body later with a .details() block, and we use record and replay so the CPU only has to be loaded once more, for the recording, and never again while we iterate on the fix. First, though, the other three tasks, which don’t name any handler at all.

Above is the CPU task’s own alert log from the stress-ng run, one line per level change. Below is the same CRITICAL as the Slack stand-in received it.

The colour in the attachment is how Slack would tint the message bar for each level.

Define topic handlers for host alerts

On the delivery side, a topic handler is a small YAML file that says “for events on this topic, do this”. Each file names its topic, a kind that matches a configured service, and that service’s options. You load them with kapacitor define-topic-handler, and they live in Kapacitor’s own store, so you edit and reload one without touching any task.

We created three, all on host-alerts. The Slack one only overrides the channel, and it goes in slack-host-alerts.yaml.

id: slack-host-alerts
topic: host-alerts
kind: slack
options:
  channel: '#alerts'

The ntfy handler uses kind post and points at the ntfy endpoint from kapacitor.conf, so it inherits the URL, headers and template from there. Save this as ntfy-host-alerts.yaml.

id: ntfy-host-alerts
topic: host-alerts
kind: post
options:
  endpoint: ntfy

Email gets a match expression so the inbox only sees CRITICAL events while Slack and ntfy get everything. The level on the right side is a bare word, not a quoted string; we got that wrong twice, and both failures are in the troubleshooting section. This is the working email-critical.yaml.

id: email-critical
topic: host-alerts
kind: smtp
match: level() == CRITICAL
options:
  to:
    - [email protected]

Load all three in one loop, then list them:

for f in slack-host-alerts.yaml ntfy-host-alerts.yaml email-critical.yaml; do kapacitor define-topic-handler $f; done
kapacitor list topic-handlers

Each handler shows against the topic with its kind. Every task that publishes to host-alerts from now on reaches all three:

Topic       ID                Kind
host-alerts email-critical    smtp
host-alerts ntfy-host-alerts  post
host-alerts slack-host-alerts slack      

To re-check one handler after an edit, kapacitor show-topic-handler host-alerts email-critical prints its match line and options, which is how we caught the quoting problem.

Build a reusable template for disk

Most host checks are the same shape: take one field from one measurement, average it per host, compare it to two thresholds. A template captures that shape once with typed var declarations, and each task built from it fills in the values from a small JSON file. When you later add Telegraf inputs, such as the ones in our Telegraf input plugins guide for Docker, Nginx and PostgreSQL, a new alert is a new vars file rather than a new script.

This is threshold_stream.tick. Variables without a default (string, lambda) must be supplied by every task; the ones with a value are optional.

// Template: mean of one field per host over a window, published to a topic.
var measurement string
var where_filter = lambda: TRUE
var field string
var window = 1m
var warn lambda
var crit lambda
var topic = 'host-alerts'

stream
    |from()
        .measurement(measurement)
        .where(where_filter)
        .groupBy('host')
    |window()
        .period(window)
        .every(window)
    |mean(field)
        .as('value')
    |alert()
        .id('{{ index .Tags "host" }}/{{ .Name }}')
        .message('{{ .ID }} is {{ .Level }}: {{ .Name }} at {{ index .Fields "value" | printf "%.1f" }}')
        .warn(warn)
        .crit(crit)
        .stateChangesOnly()
        .topic(topic)

The vars file for the root filesystem check is disk_root.json. Each value carries its type, and lambdas are written as strings with the inner quotes escaped.

{
  "measurement": {"type": "string", "value": "disk"},
  "where_filter": {"type": "lambda", "value": "\"path\" == '/'"},
  "field": {"type": "string", "value": "used_percent"},
  "window": {"type": "duration", "value": "30s"},
  "warn": {"type": "lambda", "value": "\"value\" > 80.0"},
  "crit": {"type": "lambda", "value": "\"value\" > 90.0"}
}

Register the template, then define a task from it. A template has no dbrp line of its own, so the task needs the -dbrp flag here.

kapacitor define-template threshold_stream -tick threshold_stream.tick
kapacitor list templates
kapacitor define disk_root -template threshold_stream -vars disk_root.json -dbrp telegraf.autogen

The template listing confirms which variables Kapacitor parsed out of the script, and the list should match every var line in the file.

ID               Type      Vars
threshold_stream stream    crit,field,measurement,topic,warn,where_filter,window

Later, kapacitor show disk_root reports Template: threshold_stream, and redefining the template updates every task built from it. That is the payoff: fix a message format once and all your threshold alerts change together.

Use a batch task for memory

Batch tasks work from the other direction: instead of receiving every point, they run an InfluxQL query on a schedule. We would pick batch for anything where a longer lookback matters more than reacting within seconds, and memory is a good example, because a two minute mean ignores short spikes that would make a stream task flap. Save this as mem_batch.tick.

// Batch task: query InfluxDB every 30s for mean memory use over the last 2m.
dbrp "telegraf"."autogen"

batch
    |query('SELECT mean(used_percent) AS used FROM "telegraf"."autogen"."mem"')
        .period(2m)
        .every(30s)
        .groupBy('host')
    |alert()
        .id('{{ index .Tags "host" }}/mem')
        .message('{{ .ID }} is {{ .Level }}: memory {{ index .Fields "used" | printf "%.1f" }}% used over 2m')
        .warn(lambda: "used" > 75.0)
        .crit(lambda: "used" > 85.0)
        .stateChangesOnly()
        .topic('host-alerts')

The deadman task in the next section is defined at the same time, so we enabled all three topic tasks together.

kapacitor define mem_batch -tick mem_batch.tick
kapacitor define deadman -tick deadman.tick
kapacitor enable disk_root mem_batch deadman
kapacitor list tasks

Four tasks are now executing, and the batch one is the only task that talks to InfluxDB with a query instead of a subscription.

ID        Type      Status    Executing Databases and Retention Policies
cpu_alert stream    enabled   true      ["telegraf"."autogen"]
deadman   stream    enabled   true      ["telegraf"."autogen"]
disk_root stream    enabled   true      ["telegraf"."autogen"]
mem_batch batch     enabled   true      ["telegraf"."autogen"]

To trip both, we filled the 19 GB root disk with a 13 GB file and held about 6.8 GB of the 8 GB of RAM with stress-ng, then deleted the file after 80 seconds. Those sizes fit our lab VM only. Size yours from df -h / and free -m, and never run this on a production box, because a full root disk stops InfluxDB writes and an oversized stress-ng run invites the OOM killer.

fallocate -l 13G /var/tmp/fill.img
stress-ng --vm 1 --vm-bytes 6800M --vm-keep --timeout 200s

The disk reached 94% and memory 7,487 of 7,933 MB. Remove the file with rm -f /var/tmp/fill.img when you’re done. Nothing in either task mentions ntfy, so the topic subscription was what put these on the ntfy topic, polled here with the ntfy client.

timeout 5 ntfy subscribe --poll --since 2h http://127.0.0.1:8080/tick-alerts

Each line is one notification, with the Kapacitor title and tag from the httppost headers and the body from alert-template (trimmed to the first four disk and memory events here):

{"id":"3km8F04jW1N6","time":1794462436,"expires":1794505636,"event":"message","topic":"tick-alerts","title":"Kapacitor","message":"CRITICAL hooks-api/disk is CRITICAL: disk at 93.8","tags":["rotating_light"]}
{"id":"7EPBurCd9bv0","time":1794462490,"expires":1794505690,"event":"message","topic":"tick-alerts","title":"Kapacitor","message":"CRITICAL hooks-api/mem is CRITICAL: memory 87.6% used over 2m","tags":["rotating_light"]}
{"id":"W5LbSkZwhZL6","time":1794462496,"expires":1794505696,"event":"message","topic":"tick-alerts","title":"Kapacitor","message":"OK hooks-api/disk is OK: disk at 22.5","tags":["rotating_light"]}
{"id":"AQLadCZoUPS6","time":1794462610,"expires":1794505810,"event":"message","topic":"tick-alerts","title":"Kapacitor","message":"WARNING hooks-api/mem is WARNING: memory 78.8% used over 2m","tags":["rotating_light"]}

Two details in that output are easy to misread. The disk alert went straight to CRITICAL with no WARNING first, because the 30 second mean jumped from 23% to 93.8% between two windows and stateChangesOnly reports the level it lands on. And memory came down through WARNING on the way back to OK, which is the two minute batch window slowly diluting the high samples. If you want a recovery to skip WARNING, that’s a job for .warnReset() and .critReset(), not for the thresholds.

At this point the lab held three handlers on one topic, one template and four executing tasks. The ntfy side of the same disk and memory events looked like this.

Each message is the one-line alert template, so a phone notification stays readable.

Add a deadman check for silent hosts

Threshold alerts cannot fire on data that never arrives. If Telegraf dies, the CPU task just sees nothing and stays quiet, which looks exactly like a healthy host. The deadman helper covers that gap by counting points per interval and alerting when the count drops below a floor. Save this as deadman.tick.

// Deadman: alert when a host sends fewer than 1 cpu point per minute.
dbrp "telegraf"."autogen"

stream
    |from()
        .measurement('cpu')
        .where(lambda: "cpu" == 'cpu-total')
        .groupBy('host')
    |deadman(1.0, 1m)
        .id('{{ index .Tags "host" }}/deadman')
        .message('{{ .ID }} is {{ if eq .Level "OK" }}reporting again{{ else }}silent: {{ index .Fields "emitted" | printf "%.0f" }} points in the last minute{{ end }}')
        .topic('host-alerts')

deadman() is not a node of its own. Run kapacitor show deadman and the DOT graph at the bottom shows what it expands to: a stats node that emits the point count every minute, a derivative node and an alert node. That expansion is why the message template reads the emitted field. Once the task had seen some data, the relevant part of the graph looked like this.

stats2 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stats2 -> derivative4 [processed="2"];

derivative4 [avg_exec_time_ns="0s" errors="0" working_cardinality="1" ];
derivative4 -> alert5 [processed="1"];

To test it, stop Telegraf on the host and note the time, so you can measure how long the page takes to arrive.

sudo systemctl stop telegraf; date +%T; systemctl is-active telegraf

Telegraf stopped at 05:51:23. The CRITICAL event reached ntfy and the Slack receiver at 05:53:00, the first full one minute interval with zero points, and the OK followed at 05:54:00 after we started Telegraf again at 05:53:27:

{"time":"2026-11-12T05:53:00Z","message":"CRITICAL hooks-api/deadman is silent: 0 points in the last minute"}
{"time":"2026-11-12T05:54:00Z","message":"OK hooks-api/deadman is reporting again"}

Budget for up to two minutes between a host dying and the page, because the interval has to close before it’s counted. Shorter intervals react faster but fire on a single slow flush from Telegraf. Like every host-alerts event, the deadman CRITICAL also goes through the email handler’s match, which is the next thing that broke on us.

The capture shows the stop time and the deadman CRITICAL that followed it, which is the delay you should plan your paging around.

Test alert changes with record and replay

Back to the unreadable email. Fixing it means editing cpu_alert.tick, and checking the fix normally means overloading the server again. Record and replay avoids that, because you capture a slice of real data once while the condition is happening and then feed the same points to the task as often as you like. The loop we used has four steps:

1. Record the task’s input while the problem is happening.
2. Edit the TICKscript file.
3. Redefine the task from the edited file.
4. Replay the recording against it and read what the handlers delivered.

Start a recording while a short CPU burst runs.

sudo stress-ng --cpu 2 --timeout 50s >/dev/null 2>&1 &
kapacitor record stream -task cpu_alert -duration 90s
kapacitor list recordings

The record command blocks for the full 90 seconds and prints the recording ID. The listing shows how small a recording is: 90 seconds of this task’s input came to 1.3 kB.

27343c25-5eba-4d7b-ac27-177f1c69020b
ID                                   Type    Status    Size      Date
27343c25-5eba-4d7b-ac27-177f1c69020b stream  finished  1.3 kB    12 Nov 26 06:01 UTC    

Now add a .details() block to the alert node in ~/cpu_alert.tick, right after the .message() line. Details is the HTML body for email; Slack and ntfy keep using the message.

        .details('''
<h3>{{ .ID }} is {{ .Level }}</h3>
<p>CPU busy: {{ index .Fields "used" | printf "%.1f" }}% (30s mean)</p>
<p>Task: {{ .TaskName }} at {{ .Time }}</p>
''')

Redefine the task from the edited file, which keeps it enabled, then replay the recording against it.

kapacitor define cpu_alert -tick ~/cpu_alert.tick
kapacitor replay -recording 27343c25-5eba-4d7b-ac27-177f1c69020b -task cpu_alert
kapacitor list replays

The replay ran on the default fast clock, which pushes the recorded points through as quickly as the task can take them.

ID                                   Task      Recording                            Status   Clock   Date
cea30109-9ee5-489d-8417-d48c9d3b62d0 cpu_alert 27343c25-5eba-4d7b-ac27-177f1c69020b finished fast    12 Nov 26 06:02 UTC    

The new emails now carry a body a human can read. This is the last one the replay produced.

Subject: hooks-api/cpu is OK: CPU 53.4% busy
Content-Type: text/html; charset=UTF-8

<h3>hooks-api/cpu is OK</h3>
<p>CPU busy: 53.4% (30s mean)</p>
<p>Task: cpu_alert at 2026-11-12 06:03:40.972078643 &#43;0000 UTC</p>

Treat replay as live, because it is. It ran the real alert node with its real handlers, so four actual emails went out and the mailbox grew from 9 to 13 messages. On a production Kapacitor, replay into a copy of the task that only logs, or warn your on-call channel first. Also note the time in that body: 06:03:40, while the email was sent at 06:02:21. The fast clock rebases the recording to start at replay time and runs ahead of the wall clock, so alert timestamps from a replay can sit in the future. The {{ .Time }} value also comes out with + in place of the plus sign, because the email body is HTML-escaped.

The whole record and replay loop fits on one screen, which is why it beats loading a server every time a message template changes.

Keep the Kapacitor API off the network

Everything above talked to Kapacitor on port 9092 with no credentials. Anyone who reaches that port can define a task with an exec handler, so it should not be reachable beyond the host. The control that actually holds on the open source build is the network: bind the API to loopback in the [http] section of kapacitor.conf when InfluxDB runs on the same host, or put a firewall or an authenticating TLS reverse proxy in front of it.

[http]
  bind-address = "127.0.0.1:9092"

Our lab kept the default ":9092" throughout, so we did not capture the loopback change. After a restart, ss -ltnp | grep 9092 should show the listener on 127.0.0.1 only, and kapacitor list tasks should still report every task as executing, since InfluxDB on the same host reaches the subscription over loopback.

Do not mistake auth-enabled for a lock. The Kapacitor security page says authentication and authorization are not fully implemented in open source Kapacitor. With [auth] disabled, the server loads its noauth service, whose source says it returns admin users for all requests and ignores the password. So a Basic auth request with any username and password still gets in. We only tested bearer tokens in the lab, not Basic auth, and that gap is exactly why the shared secret below is not a substitute for loopback or a proxy.

What the shared secret does gate is JWT bearer tokens, which Chronograf and some scripts send. The current release tightened that path, per the Kapacitor release notes: bearer authentication now requires a non-empty shared secret. With only auth-enabled = true in [http], every bearer request is refused. Our capture holds three answers in order: a bearer request, a curl request with no credentials at all, and the kapacitor CLI with no credentials in its URL.

HTTP/1.1 401 Unauthorized
{"error":"bearer authentication is not enabled","message":"bearer authentication is not enabled"}
HTTP/1.1 401 Unauthorized
{"error":"unable to parse authentication credentials","message":"unable to parse authentication credentials"}
Error: unable to parse authentication credentials

Add a long random shared-secret under [http] and restart Kapacitor. Ours was a lab value; generate yours with openssl rand -hex 32. The lab block kept bind-address = ":9092" so we could test from outside, which is the setting to change to loopback on a real host.

[http]
  bind-address = ":9092"
  auth-enabled = true
  shared-secret = "LabSharedSecret-ChangeMe-4306"
  log-enabled = true
  write-tracing = false

Clients then send an HS512 JWT whose payload carries a username and an exp time, signed with that secret. We generated tokens with this helper, saved as ~/mkjwt.py.

#!/usr/bin/env python3
# Usage: mkjwt.py <shared-secret> <username>  -> HS512 token valid for 10 minutes
import base64, hashlib, hmac, json, sys, time

def b64(data):
    return base64.urlsafe_b64encode(data).rstrip(b"=").decode()

secret, user = sys.argv[1], sys.argv[2]
header = b64(json.dumps({"alg": "HS512", "typ": "JWT"}).encode())
payload = b64(json.dumps({"username": user, "exp": int(time.time()) + 600}).encode())
sig = hmac.new(secret.encode(), f"{header}.{payload}".encode(), hashlib.sha512).digest()
print(f"{header}.{payload}.{b64(sig)}")

Make a token and call the tasks endpoint with it:

TOKEN=$(python3 ~/mkjwt.py LabSharedSecret-ChangeMe-4306 eugene)
curl -s -i -H "Authorization: Bearer $TOKEN" "http://localhost:9092/kapacitor/v1/tasks?fields=status" | head -1

The signed token gets a 200. A second request we sent with a malformed bearer token is rejected with a parse error rather than the “not enabled” message, which tells you the secret is now active.

HTTP/1.1 200 OK
HTTP/1.1 401 Unauthorized
{"error":"invalid token: token contains an invalid number of segments","message":"invalid token: token contains an invalid number of segments"}

We would only turn this on when something off the host genuinely needs the API, such as Chronograf on another server, and even then behind a firewall rule or proxy. Once it is on, a request with no credentials gets a 401, so scripts and the CLI need either a token or a -url carrying credentials. Because the open source build does not check those credentials, treat the 401 as a speed bump for mistakes, not as access control.

Both bearer states from the lab are in that terminal: refused with an empty secret, then a 200 for the signed token and a parse error for a junk one.

Check these errors when alerts go missing

Every one of these came up in the lab. Most of them fail quietly, so the error string is the fastest way to recognise them.

Error: must specify dbrp

kapacitor define exits with status 3 and this message when the TICKscript has no dbrp line and you did not pass -dbrp. Add dbrp "telegraf"."autogen" at the top of the script, or append -dbrp telegraf.autogen to the define command. Our template has no dbrp line, which is why the disk task needed the flag; a template that carries its own dbrp line works too, as long as the task definition then leaves the flag out.

cannot assign *ast.ReferenceNode to type string

The full message was Error: invalid TICKscript: line 3 char 10: cannot assign *ast.ReferenceNode to type string, did you use double quotes instead of single quotes? from .measurement("cpu"). In TICKscript, double quotes mean a field or tag reference and single quotes mean a string literal. Use .measurement('cpu'), and keep double quotes for names inside lambdas, such as "used" > 90.

failed to evaluate match expression: no tag exists for CRITICAL

Our first email-critical.yaml had match: level() == "CRITICAL". The handler loads fine, no email ever arrives, and the Kapacitor log shows this error on each event. The same double quote rule applies inside a match expression, so "CRITICAL" was read as a tag name. Quoting it with single quotes instead gave a second error:

lvl=error msg="failed to evaluate match expression" service=alert handler=email-critical topic=host-alerts err="mismatched type to binary operator. got int == string. see bool(), int(), float(), string(), duration()"

level() returns an integer, and the bare level names are built-in constants for those integers, so the comparison has to use a constant rather than a string. The fix takes three steps:

1. Write match: level() == CRITICAL in the YAML with no quotes at all.
2. Reload it with kapacitor define-topic-handler email-critical.yaml.
3. Check the stored match with kapacitor show-topic-handler host-alerts email-critical.

After that, the disk CRITICAL reached the inbox and the OK recovery was filtered out, as intended. The handler never reports a bad match when you load it, so the Kapacitor log is the only place this shows up.

SMTP test succeeds but no email: x509 certificate is valid for another name

kapacitor service-tests smtp reported true, but the mailbox stayed empty. The Kapacitor journal had err="tls: failed to verify certificate: x509: certificate is valid for hooks-api.lab.example, not localhost", and Postfix logged lost connection after STARTTLS from localhost[127.0.0.1]. Kapacitor upgrades to TLS whenever the server offers it and checks the name. Set host to the name on the certificate, or for a loopback relay only, set no-verify = true.

httppost service test posts to localhost:3000

Covered above, but it’s the error people search for: failed to POST alert data: Post "http://localhost:3000/". The CLI service test uses built-in options, which you can see with curl -s http://localhost:9092/kapacitor/v1/service-tests/httppost. POST your own endpoint and url to that same API path to test the real target.

could not create TLS config for connecting to meta service

Before finding the shared secret option, we tried enabled = true in the [auth] section. Kapacitor refused to start with run: create server: could not create TLS config for connecting to meta service: Could not load TLS client key/certificate: open /etc/kapacitor/cert.pem: no such file or directory. The shipped config points [auth] at a meta service address and a client certificate, and Kapacitor tries to load that certificate as soon as the section is enabled. We did not take [auth] further in the lab. We left it disabled, reverted the change, and relied on loopback binding as described above.

Verify every route end to end

Before calling an alerting setup done, we would want one real event through every task and handler pair, not just green service tests. This is what each path delivered in the lab, with the trigger we used:

Trigger	Task	Slack receiver	Email	ntfy
stress-ng –cpu 2	cpu_alert	WARNING, CRITICAL, OK	All three (direct handler)	Not routed
13 GB file on /	disk_root	CRITICAL, OK	CRITICAL only	CRITICAL, OK
stress-ng –vm 6800M	mem_batch	CRITICAL, WARNING, OK	Missed: match was still broken	CRITICAL, WARNING, OK
systemctl stop telegraf	deadman	CRITICAL, OK	Missed: match was still broken	CRITICAL, OK
kapacitor replay	cpu_alert	Not checked	4 emails with details body	Not routed

The misses in the email column are the quoting bug from the troubleshooting section: those events fired while the match expression was still wrong, so the handler logged an error and dropped them. The first two disk fills hit the same bug, which is why the disk row’s email cell comes from a third fill after the fix, while its Slack and ntfy cells come from the first one. On that third run the CRITICAL reached the inbox and its OK was filtered out, which is exactly the behaviour the table should show for every row. That’s why we test each route with a real event rather than trusting the service tests.

Moving the Kapacitor Slack handler from the lab receiver to a real workspace is a one-line change of url in the [[slack]] block, then kapacitor service-tests slack and one more stress-ng run. With auth-enabled on, the global flag goes before the subcommand, as in kapacitor -url http://user:pass@localhost:9092 service-tests slack. If you are comparing this with Prometheus, the same thresholds expressed as alert rules are in our PromQL cheat sheet. For teams already on InfluxDB 3, the InfluxDB 3, Telegraf and Grafana build is the path that does not involve Kapacitor at all. The full handler reference is in the Kapacitor event handlers documentation.

Keep reading