Skip to content

Alertmanager

Alternative Web-Interfaces / Dashboards

Community managed alerts

amtool

amtool is a cli tool for interacting with the Alertmanager API. It is bundled with all releases of Alertmanager

Aliases based on context: Docker:

alias amtool='docker exec alertmanager amtool --alertmanager.url http://localhost:9093'

Kubernetes prometheus:

alias amtool='kubectl -n oas exec -it -c prometheus-alertmanager \
  deployment/prometheus-alertmanager -- amtool --alertmanager.url http://localhost:9093'

Kubernetes kube-prometheus-stack:

alias amtool='kubectl -n oas exec -it -c alertmanager \
  statefulset/alertmanager-kube-prometheus-stack-alertmanager -- \
  amtool --alertmanager.url http://localhost:9093'

Inside pod:

alias amtool='amtool --alertmanager.url http://localhost:9093'

Show alerts:

amtool alert query
amtool alert query -o json | json_pp
amtool alert query instance='https://example.org'

Show silences: amtool silence query Silence alert (expiry defaults to 1h):

amtool silence add instance='https://example.org' -c 'Upstream issue'
amtool silence add 'instance=~.*example.org' -c 'Upstream issue' -d 3d

Expire (delete) silence:

amtool silence expire 82d7b097-b21d-40b0-ad77-4397afe09531

Add a test alert

amtool alert add alertname=foo node=bar

API

Query status

curl --netrc https://alertmanager.oas.varac.net/api/v2/status | jq .

All alerts:

curl --netrc https://alertmanager.oas.varac.net/api/v2/alerts | jq .

Filter out silenced alerts:

curl --netrc 'https://alertmanager.oas.varac.net/api/v2/alerts?silenced=false' | jq .

Filter out silenced and watchdog alerts:

curl -Ssl --netrc 'https://aldertmanager.oas.varac.net/api/v2/alerts?silenced=false&receiver=email' | jq length

Sending out a test alert mail:

curl -H "Content-Type: application/json" \
  -d '[{"labels":{"alertname":"TestAlert1"}}]' localhost:9093/api/v1/alerts

Alertmanager bots

Alertmanager by itself only supports a handful of 3rd party services/protocols directly. Unfortunatly matrix, signal etc are not supported. The webhook-reciever can be used to integrate middle-bots to reach more services.

Multi-messenger bots

Matrix

Test matrix-alertmanager:

curl -H "Content-Type: application/json" -XPOST \
  -d @~/projects/monitoring/prometheus/alertmanager/webhook-test.json \
   'alertmanager.example.org:3001/alerts?secret=...'

SMS

Signal

Mattermost

Alertmanager libraries