How to Get Alerted When a Docker Container Goes Down
A docker container down alert comes down to two moving parts: watch the container's state (exited, unhealthy, or restarting) and push a notification to a channel you actually read, like Slack, email, PagerDuty, or a phone push. The two reliable ways to do this are a small script that polls docker events or docker inspect and fires a webhook, or a monitoring agent (Prometheus + Alertmanager, or an app like Docker HQ) that watches container state for you and pushes the alert.
Detecting a stopped container is the easy part. What trips people up is getting the alert to you fast, keeping it from spamming you on every restart, and having it still fire when the whole host is unreachable. So let's start from the simplest thing that works and build up to something you can trust on a production server.
Before you build a docker container down alert, define "down"
"Down" is ambiguous, and getting the definition right saves you from noisy alerts. A container can be:
- Exited: the process stopped, cleanly or with an error. Check with
docker ps -a. - Restarting: it keeps crashing and Docker keeps restarting it (a crash loop).
- Unhealthy: the process is running but its health check is failing.
- Paused: deliberately paused, usually not an incident.
Most people mean "exited unexpectedly" or "stuck in a crash loop." A container that Docker restarted once in the middle of the night and recovered is usually not worth waking up for. One that has been restarting every 10 seconds for five minutes definitely is.
See the current state of everything at once:
docker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.State}}'
Let Docker restart it first, then alert on failure
Before you alert on anything, give the container a chance to recover on its own. A restart policy handles transient crashes so your phone stays quiet:
docker run -d --restart=on-failure:5 --name web nginx
Or in Compose:
services:
web:
image: nginx
restart: on-failure:5
on-failure:5 restarts the container up to five times if it exits with a non-zero code, then gives up. unless-stopped keeps it running across daemon restarts unless you stopped it by hand. The point is that you want to be alerted when recovery fails, not on every hiccup that Docker fixes by itself.
Restart policies pair naturally with health checks. If you haven't set those up, our guide to Docker health checks with examples walks through writing a HEALTHCHECK that actually reflects whether your app is serving traffic, not just whether the process exists.
Option 1: A polling script with a webhook
The most portable approach needs nothing but bash, cron, and a webhook URL. This checks every container's state and posts to Slack when one isn't running:
#!/usr/bin/env bash
set -euo pipefail
WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"
# Names of containers that should always be running
WATCH=(web api db)
for name in "${WATCH[@]}"; do
state=$(docker inspect -f '{{.State.Status}}' "$name" 2>/dev/null || echo "missing")
if [[ "$state" != "running" ]]; then
curl -s -X POST -H 'Content-type: application/json' \
--data "{\"text\":\":red_circle: Container *$name* is *$state* on $(hostname)\"}" \
"$WEBHOOK"
fi
done
Run it every minute with cron:
* * * * * /opt/scripts/check-containers.sh
This works, but it has two flaws. It fires every single minute while a container stays down, which turns into a wall of Slack messages. And it can't tell you anything when the host itself is offline, because the script runs on that host.
Fixing the spam: only alert on state changes
Watch docker events instead of polling. Events stream in real time, so you alert the moment a container dies and only once per event:
#!/usr/bin/env bash
set -euo pipefail
WEBHOOK="https://hooks.slack.com/services/XXX/YYY/ZZZ"
docker events --filter 'event=die' --format '{{.Actor.Attributes.name}} {{.Actor.Attributes.exitCode}}' |
while read -r name code; do
curl -s -X POST -H 'Content-type: application/json' \
--data "{\"text\":\":red_circle: Container *$name* died (exit $code) on $(hostname)\"}" \
"$WEBHOOK"
done
The die event fires once when a container's main process exits. You get exactly one message per death, with the exit code, which is usually enough to know whether it was OOM-killed (137), a clean stop (0), or an application crash. Run this as a systemd service so it restarts if the box reboots. You can also filter for event=health_status to catch containers that go unhealthy without exiting.
Option 2: Prometheus, cAdvisor, and Alertmanager
If you already run Prometheus, this is the sturdy option. cAdvisor exports per-container metrics, and Alertmanager handles routing, grouping, and silencing so you don't get paged twice for the same thing.
A basic alert rule that fires when a container has been down for two minutes:
groups:
- name: containers
rules:
- alert: ContainerDown
expr: absent(container_last_seen{name="web"}) or time() - container_last_seen{name="web"} > 60
for: 2m
labels:
severity: critical
annotations:
summary: "Container {{ $labels.name }} is down"
The for: 2m clause is what keeps you sane. The condition has to hold for two straight minutes before Alertmanager sends anything, so a quick restart never reaches you. Alertmanager then routes to email, Slack, PagerDuty, or Opsgenie based on the severity label. This is more moving parts to run, but if you're managing a fleet it pays off. For the full picture of what to collect and why, see our complete guide to Docker container monitoring.
The host-down blind spot
Every method above shares one weakness: if the server loses power or drops off the network, whatever runs on that server can't tell you. A polling script on a dead host sends nothing. Prometheus scraping a dead target will fire an alert, but only if your Prometheus lives on a different machine.
The general fix is a dead man's switch, an external check that expects a regular heartbeat and alerts you when the heartbeat stops. Push a ping to a service like Healthchecks.io on a schedule:
# In cron, after your container check passes
curl -fsS -m 10 --retry 3 https://hc-ping.com/your-uuid > /dev/null
When the host dies, the pings stop, and the external service alerts you about the silence. It's the one alert that survives the host it's watching.
Doing this from your phone
If you don't want to stand up Prometheus or babysit a systemd unit on every host, this is roughly what Docker HQ does. It connects to your servers over SSH, watches container state and host health from outside the box, and sends a push notification to your phone when a container exits or a host stops responding, without you deploying an agent on the server. When the alert lands you can open the app, read the container's logs, and restart it or drop into a shell right there. You can compare that against rolling your own on the pricing page.
Whichever route you pick, the same monitoring data drives capacity decisions too. Watching CPU and memory alongside up/down state catches the container that's about to get OOM-killed before it actually dies, which is covered in how to monitor Docker container CPU and memory usage.
Frequently asked questions
How do I know if a container was killed for running out of memory?
Check the exit code. An OOM kill shows exit code 137 (128 + signal 9). Confirm it with docker inspect -f '{{.State.OOMKilled}}' <container>, which returns true if the kernel's OOM killer took it. Include the exit code in your alert message so you can tell an OOM kill from a normal crash at a glance.
Should I alert on every restart or only when the container stays down?
Only when it stays down or crash-loops. A single restart that recovers is noise. Use for: 2m in Prometheus, or alert only on the die event combined with a restart policy that has already given up. If a container restarts more than a few times in a short window, that crash loop is worth an alert on its own.
Can I get alerts without running an agent on the server?
Yes. Watch the containers from outside over SSH or the Docker remote API, or use a dead man's switch that expects a heartbeat from the host. Docker HQ takes the SSH approach so there's nothing to install on the box you're watching. The tradeoff is that an external checker needs network access to the host and valid credentials.
What's the difference between docker events and polling docker ps?
docker events is a real-time stream, so you learn about a death the instant it happens and get exactly one event per state change. Polling docker ps on a timer can miss a container that died and restarted between checks, and it re-reports the same down state on every poll. Events are the better base for alerting; use polling only as a periodic sanity check.
Start with the docker events script and a restart policy today. It's about 15 lines and covers the common case. Add the external heartbeat before you rely on any of it, because the alert that matters most is the one that still fires when the whole host goes dark.