How to fix Error response from daemon: The swarm does not have a leader in Docker

First, determine which manager nodes are still accessible. On any node that can communicate:

docker node ls

This may fail with the "no leader" error. If so, check individual node status:

docker info | grep -A 20 "Swarm:"

Look for:
- Is Manager: true
- Managers: X
- Nodes: Y

Check if the Docker daemon is running on all manager nodes:

# On each manager node
systemctl status docker
docker info

List the Raft state directory to see if data exists:

ls -la /var/lib/docker/swarm/raft/

Manager nodes must be able to communicate on specific ports. Test connectivity from each manager:

# Test cluster management port (TCP 2377)
nc -zv <other_manager_ip> 2377

# Test node communication port (TCP/UDP 7946)
nc -zv <other_manager_ip> 7946

# Test overlay network traffic (UDP 4789)
nc -zvu <other_manager_ip> 4789

Check firewall rules:

# For iptables
sudo iptables -L -n | grep -E "2377|7946|4789"

# For firewalld
sudo firewall-cmd --list-all

Ensure these ports are open between all manager nodes:
- 2377/tcp: Cluster management
- 7946/tcp+udp: Node communication
- 4789/udp: Overlay network traffic

If manager nodes are simply down (not lost), bring them back online:

# On each offline manager
sudo systemctl start docker

# Wait for the node to rejoin
sleep 30

# Check if leader is elected
docker node ls

If managers are stuck, try restarting Docker:

sudo systemctl restart docker

Check Docker logs for errors:

sudo journalctl -u docker -f

Look for messages about Raft elections, connectivity issues, or leader election timeouts.

Important: If you have 3 managers and only 1 is healthy, you cannot recover quorum this way. You need at least 2 of 3 managers to achieve majority.

If you cannot bring enough managers back online, reinitialize the swarm from a surviving manager with intact Raft data:

# On the manager with the most recent data
docker swarm init --force-new-cluster --advertise-addr <manager_ip>

This command:
- Creates a new single-node swarm using existing Raft data
- Preserves service definitions, secrets, and configs
- Maintains worker node registrations (they'll reconnect)
- Makes this node the new leader

After forcing a new cluster:

# Verify the new swarm is working
docker node ls

# Check services are intact
docker service ls

# Get the new join tokens
docker swarm join-token manager
docker swarm join-token worker

Warning: Only use --force-new-cluster when you cannot restore quorum normally. Running it on multiple nodes simultaneously can cause split-brain scenarios.

After recovering, clean up failed nodes and restore redundancy:

# List all nodes
docker node ls

# Remove unreachable manager nodes
docker node demote <failed_node_id>
docker node rm --force <failed_node_id>

Add new manager nodes to restore fault tolerance:

# On the leader, get the manager join token
docker swarm join-token manager

# On the new manager node
docker swarm join --token <manager_token> <leader_ip>:2377

Promote existing workers to managers if needed:

docker node promote <worker_node_id>

Verify the cluster is healthy:

docker node ls
# Look for one "Leader" and multiple "Reachable" manager statuses

Prevent future quorum loss by following Docker's recommendations:

Use an odd number of managers (3, 5, or 7):
- 3 managers: Tolerates 1 failure
- 5 managers: Tolerates 2 failures
- 7 managers: Tolerates 3 failures (maximum recommended)

Never use 2 managers - if one fails, you have 50% which is not a majority.

Spread managers across failure domains:

# When adding managers, use different availability zones/racks
docker node update --label-add zone=us-east-1a manager1
docker node update --label-add zone=us-east-1b manager2
docker node update --label-add zone=us-east-1c manager3

Use static IPs for manager nodes to ensure reliable communication.

Configure manager-only nodes:

# Prevent workloads from running on managers
docker node update --availability drain <manager_node_id>

This dedicates managers to cluster management and improves Raft performance.

Implement proactive monitoring to detect quorum issues early:

Monitor manager health:

#!/bin/bash
# Check swarm health
if ! docker node ls &>/dev/null; then
  echo "CRITICAL: Swarm has no leader!"
  # Send alert
fi

# Count reachable managers
managers=$(docker node ls --filter role=manager -q 2>/dev/null | wc -l)
reachable=$(docker node ls --filter role=manager --format "{{.Status}}" 2>/dev/null | grep -c "Ready")
echo "Managers: $reachable/$managers reachable"

Backup swarm configuration regularly:

# Backup swarm data (run on a manager)
sudo tar -czvf swarm-backup-$(date +%Y%m%d).tar.gz /var/lib/docker/swarm/

Use Docker's autolock feature for additional security:

docker swarm update --autolock=true
# Save the unlock key securely!

If autolock is enabled and a manager restarts, you'll need to unlock it:

docker swarm unlock