Incident Response When a Docker Host Goes Down

A docker host down incident is almost always one of three things: the machine is off or network-partitioned, the Docker daemon crashed, or the disk filled up and everything stalled. Work it in this order: confirm whether the box is unreachable or the daemon just died, get a shell in over SSH, check dockerd and disk space first, then bring services back with a controlled restart. You can rule out those three causes in about two minutes.

Below is the runbook I actually use, in the order I run it, plus the commands to copy.

Triage a docker host down incident: host or daemon?

These are different incidents with different fixes, so separate them immediately.

Ping and try to connect. From your laptop:

ping -c 3 your-host.example.com
ssh -o ConnectTimeout=5 deploy@your-host.example.com 'echo ok'

If ping fails and SSH times out, the host itself is unreachable. That's a machine or network problem, not a Docker problem. Check your cloud console (is the instance running? did it get stopped or terminated?), the hypervisor, or the power/network on bare metal. No amount of docker commands help until you can log in.

If SSH connects, the host is alive and you're dealing with the daemon, a specific container, or a resource problem. Keep going.

This triage split is the whole reason to have out-of-band access. When the page fires, you want to answer "can I even reach it" from wherever you are, not just from the one laptop with your keys on it. Getting the alert and being able to SSH in from your phone turns a 20-minute scramble into a 2-minute check. That's the gap Docker HQ is built to close.

Get in and check the daemon

Once you have a shell, check whether Docker is actually running:

systemctl status docker

If it's inactive or failed, look at why before you blindly restart it:

journalctl -u docker --no-pager -n 100

Common causes you'll see in those logs: a corrupt config after an edit, a bad /etc/docker/daemon.json, or an OOM kill. If the daemon is just stopped and the logs are clean, start it:

sudo systemctl start docker
docker ps

If dockerd refuses to start, validate your daemon config. A single trailing comma in daemon.json will stop the whole daemon:

sudo dockerd --validate --config-file /etc/docker/daemon.json

Check disk before you do anything clever

This is the one people skip, and it's the most common silent killer. A full disk makes containers hang, logs stop writing, and the daemon behaves in ways that look like corruption. Check it early:

df -h
df -i   # inodes, in case you're out of those instead of bytes

If / or /var/lib/docker is at 100%, that's very likely your root cause. Reclaim space safely:

docker system df            # see what's using space first
docker container prune -f   # stopped containers
docker image prune -f       # dangling images

Be careful with docker system prune -a --volumes. It will delete unused volumes, and "unused" includes any volume not currently attached to a running container. If your database container is stopped because the host just died, its volume can look unused. Don't run the volume-nuking prune during an active incident unless you're certain what's attached. If you don't have a recent copy of that data, that's a separate lesson, and backing up Docker volumes is the fix for next time.

Oversized container logs are a frequent disk hog. Find the worst offenders:

du -sh /var/lib/docker/containers/*/*-json.log | sort -rh | head

Bring services back in a controlled way

With the daemon up and disk clear, restart your workloads. If you use Compose, go to the project directory and bring the stack up:

cd /srv/myapp
docker compose up -d
docker compose ps

If containers were set to restart automatically, many will already be back. Confirm which ones aren't:

docker ps -a --filter 'status=exited' --format 'table {{.Names}}\t{{.Status}}'

For a container that keeps dying, read its logs rather than restarting in a loop:

docker logs --tail 200 --timestamps myapp_web

Check the exit code too. docker inspect tells you whether it was an OOM kill, a non-zero application exit, or a signal:

docker inspect myapp_web --format '{{.State.ExitCode}} {{.State.OOMKilled}} {{.State.Error}}'

An OOMKilled: true means the container hit its memory limit or the host ran out of RAM. Check host memory with free -h. If the host itself was starved, restarting the same container without a memory limit will just repeat the outage.

Start dependencies in order

If your app depends on a database or cache, start those first and confirm they're healthy before the app tier. A healthcheck in your Compose file makes this automatic, but during manual recovery just do it by hand:

docker compose up -d postgres redis
docker compose exec postgres pg_isready
docker compose up -d web worker

Confirm recovery, then watch

Don't close the incident when the containers show Up. Confirm the service actually works: hit the health endpoint, run a real request, check that the queue is draining. Then keep an eye on the host for a bit, because a host that just OOM'd or filled its disk often does it again within the hour.

curl -fsS http://localhost:8080/healthz && echo ' healthy'
docker stats --no-stream

This is where continuous monitoring earns its keep. If you'd had CPU, RAM, and disk trends in front of you, the disk-full outage would have shown up as a warning days earlier instead of a 3am page. If you want to set that up properly, our complete guide to Docker container monitoring walks through what to watch and the alert thresholds that actually matter. Getting a push alert the moment a host or container goes down, rather than finding out from a customer, is worth wiring up before you need it. You can do that in Docker HQ across every host you manage.

Write down what happened

Before you sign off, capture the timeline while it's fresh: what alerted, what you found, what fixed it. Two sentences is fine. The goal is that the next person (possibly future you) doesn't re-diagnose the same thing from scratch. If the fix was manual, that's a signal it should be automated, whether that's a disk-usage alert, a log-rotation config, or a memory limit on the container that keeps eating the box.

And fix the log rotation while you remember, because unbounded JSON logs come back. Set it globally in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then sudo systemctl restart docker for it to take effect on new containers. Existing containers keep their old settings until recreated. If your recovery involved redeploying a stack from scratch, our walkthrough on deploying Docker Compose to a remote server covers doing that cleanly.

Frequently asked questions

How do I tell if the whole host is down or just one container?

Try to SSH in. If you get a shell, the host is up and it's a Docker or container problem, so run docker ps and systemctl status docker. If SSH times out and ping fails, the machine or its network is down, and you need the cloud console or physical access, not Docker commands.

Will my containers restart automatically after the host reboots?

Only if they have a restart policy. A container started with --restart unless-stopped or restart: unless-stopped in Compose comes back after a reboot; the default no policy does not. Check with docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' <container>. Note that unless-stopped won't restart a container you had deliberately stopped before the crash.

What's the safest way to free disk space during an outage?

Start with docker container prune -f and docker image prune -f, which only remove stopped containers and dangling images. Avoid docker system prune -a --volumes mid-incident, because it deletes volumes not attached to a running container, and your stopped database volume can qualify. Always run docker system df first to see where the space actually went.

Why does my container keep getting OOMKilled after I restart it?

The container is exceeding its memory limit, or the host is out of RAM. Check docker inspect --format '{{.State.OOMKilled}}' <container> and free -h. Restarting without changing anything repeats the outage. Set a sane --memory limit, find the leak, or move the workload to a bigger host.