How to fix connect: connection timed out in Kubernetes

KubernetesINTERMEDIATEHIGH

This error occurs when network connectivity fails between pods, services, or external endpoints. Fix it by verifying IP forwarding settings, checking DNS resolution, reviewing network policies, and ensuring CNI plugins are functioning correctly.

What this error means

The "connect: connection timed out" error indicates that a network connection attempt failed to establish within the expected time limit. In Kubernetes, this typically means a pod cannot reach another pod, service, or external endpoint due to networking issues. This error can occur at multiple layers: DNS resolution failures prevent hostname lookup, IP forwarding disabled on nodes blocks inter-pod traffic, network policies restrict communication, or CNI plugin misconfiguration prevents pod networking entirely. The timeout behavior distinguishes this from "connection refused" (target reachable but not listening) — with timeouts, packets never receive a response, suggesting routing, firewall, or fundamental connectivity issues.

How to fix "connect: connection timed out"

1Verify and enable IP forwarding on nodes

Check if IP forwarding is disabled (common cause):

bash

# SSH to affected node and check
sysctl net.ipv4.ip_forward
# Output: 0 means disabled (problematic)

Enable IP forwarding:

bash

# Enable immediately
sudo sysctl -w net.ipv4.ip_forward=1

# Persist across reboots
echo "net.ipv4.ip_forward=1" | sudo tee /etc/sysctl.d/99-ipforward.conf
sudo sysctl -p /etc/sysctl.d/99-ipforward.conf

2Enable bridge netfilter settings

Bridge netfilter is required for iptables rules on Linux bridges:

bash

# Enable on each node
sudo sysctl -w net.bridge.bridge-nf-call-iptables=1
sudo sysctl -w net.bridge.bridge-nf-call-ip6tables=1

# Persist across reboots
cat <<EOF | sudo tee /etc/sysctl.d/99-bridge-nf.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
EOF
sudo sysctl -p /etc/sysctl.d/99-bridge-nf.conf

3Debug DNS resolution from a pod

Create a debug pod with diagnostic tools:

bash

kubectl run -it --rm debug --image=nicolaka/netshoot --restart=Never -- bash

# Inside pod, test DNS resolution
nslookup kubernetes.default
# Should show IP, not "connection timed out"

# Check resolv.conf
cat /etc/resolv.conf
# Should include nameserver pointing to CoreDNS

If DNS fails, check CoreDNS directly:

bash

# Get CoreDNS pod IP
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# Query CoreDNS directly from debug pod
nslookup kubernetes.default <COREDNS_POD_IP>

4Verify CoreDNS status and scale if needed

Check CoreDNS pod status:

bash

kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

If CoreDNS is overwhelmed, scale it up:

bash

kubectl scale --replicas=3 -n kube-system deployment/coredns

Verify CoreDNS is receiving queries:

bash

kubectl logs -n kube-system -l k8s-app=kube-dns -f

5Test service connectivity directly

From a debug pod, test both DNS and direct IP:

bash

# Test by service name (uses DNS)
curl -v --connect-timeout 5 http://my-service:8080/health

# Get service cluster IP
kubectl get service my-service

# Test by cluster IP directly (bypasses DNS)
curl -v --connect-timeout 5 http://10.100.225.223:8080/health

Interpretation:
- Direct IP works but service name fails → DNS issue
- Both fail → network policy or service not running

6Check network policies blocking traffic

List network policies affecting traffic:

bash

kubectl get networkpolicies --all-namespaces
kubectl describe networkpolicy <policy-name> -n <namespace>

If policies exist, ensure DNS egress is allowed:

yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53

Use tcpdump on worker node to verify traffic:

bash

# On node, capture traffic to pod IP
sudo tcpdump -i any -n host 10.244.1.25
# SYN without ACK = traffic blocked or port not listening

How to fix connect: connection timed out in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "connect: connection timed out"

Advanced notes

Related errors

Official resources & further reading