How to fix Hubble observability error in Kubernetes

KubernetesINTERMEDIATEMEDIUMUpdated Dec 6, 2025· 141 views

Hubble errors occur when network observability components fail to start or cannot communicate. Fix by verifying Hubble deployment, checking Hubble Relay connectivity, and reviewing certificate configuration.

Quick fix

Fixes Hubble observability error

kubectl get daemonset -n kube-system cilium -o yaml | grep -A 5 "hubble"
kubectl get deployment -n kube-system cilium-ui
kubectl get deployment -n kube-system hubble-relay
  1. Verify Hubble is enabled in Cilium
  2. Check Hubble Relay pod status
  3. Test Hubble agent connectivity
  4. Check Hubble gRPC port accessibility
kubernetes — error
kubectl get daemonset -n kube-system cilium -o yaml | grep -A 5 "hubble"
kubectl get deployment -n kube-system cilium-ui
kubectl get deployment -n kube-system hubble-relay
Hubble observability error

What this error means

Cilium Hubble provides network observability by collecting flow events from eBPF. When Hubble fails, operators cannot see network traffic patterns, policy violations, or connection issues. Errors typically involve Hubble Relay unable to connect to agents, TLS issues, or service startup failures.

Typical symptoms

  • Hubble UI or CLI reports "connection refused"
  • Error "connection_refused" when querying Hubble Relay
  • Hubble Relay pod logs show connection errors
  • Cannot retrieve flow logs or network events
  • Hubble returns TLS certificate errors
  • Hubble service endpoint shows no backing pods
  • Cilium agent unable to expose Hubble API
  • Hubble gRPC server not listening on expected port

Common causes

  1. Hubble Relay service cannot reach Cilium agent Hubble API
  2. TLS certificate mismatch or expired
  3. Hubble not enabled in Cilium configuration
  4. Hubble deployment missing or crashing
  5. Network policies blocking Hubble traffic
  6. Port 4244 (Hubble gRPC) not accessible
  7. Hubble service DNS resolution failing
  8. ServiceMonitor misconfiguration for Prometheus

How to fix "Hubble observability error"

1Verify Hubble is enabled in Cilium

Check if Hubble is configured:

bash
kubectl get daemonset -n kube-system cilium -o yaml | grep -A 5 "hubble"
kubectl get deployment -n kube-system cilium-ui
kubectl get deployment -n kube-system hubble-relay
2Check Hubble Relay pod status

Verify Relay is running:

bash
kubectl get pods -n kube-system -l app=hubble-relay
kubectl logs -n kube-system -l app=hubble-relay --tail=50

If not running, check deployment for errors.

3Test Hubble agent connectivity

From Hubble Relay pod, test connection to agents:

bash
kubectl exec -it -n kube-system deployment/hubble-relay -- \
  hubble-relay-cli nodes -o json

This lists all connected Cilium nodes.

4Check Hubble gRPC port accessibility

Verify port 4244 is listening on agents:

bash
kubectl exec -it -n kube-system ds/cilium -- \
  ss -tlnp | grep 4244

If not listening, Hubble API not started on agent.

5Review Hubble TLS certificate configuration

Check certificate secrets:

bash
kubectl get secret -n kube-system | grep hubble
kubectl describe secret -n kube-system hubble-relay-client-certs

Ensure certificates are valid and properly mounted.

6Verify network policies allow Hubble traffic

Ensure network policies do not block Hubble:

bash
kubectl get networkpolicy --all-namespaces
# Test if policies allow port 4244
kubectl exec -it -n kube-system ds/cilium -- \
  nc -zv localhost 4244
7Check Hubble service and endpoints

Verify service is properly configured:

bash
kubectl get svc -n kube-system hubble-relay
kubectl describe svc -n kube-system hubble-relay
kubectl get endpoints -n kube-system hubble-relay
8Enable Hubble via Helm or direct configuration

If Hubble is disabled, enable it:

bash
# Via Helm
helm upgrade cilium cilium/cilium \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true

# Or edit ConfigMap
kubectl edit cm -n kube-system cilium-config

Advanced notes

For production observability, deploy Hubble Relay separately from agents and give it dedicated resources. Integrate Hubble with Prometheus for metrics collection. Use Hubble UI for interactive debugging of network issues. Set up alerts on flow patterns indicating potential security issues. Hubble has minimal performance impact when enabled but use metrics filtering to avoid overwhelming database. Archive flow logs for compliance and forensics.

Frequently asked questions

What does "Hubble observability error" mean?
Cilium Hubble provides network observability by collecting flow events from eBPF. When Hubble fails, operators cannot see network traffic patterns, policy violations, or connection issues. Errors typically involve Hubble Relay unable to connect to agents, TLS issues, or service startup failures.
How do I fix "Hubble observability error"?
Hubble errors occur when network observability components fail to start or cannot communicate. Fix by verifying Hubble deployment, checking Hubble Relay connectivity, and reviewing certificate configuration.
What causes "Hubble observability error"?
Hubble Relay service cannot reach Cilium agent Hubble API TLS certificate mismatch or expired Hubble not enabled in Cilium configuration
Is the "Hubble observability error" error serious?
It is a medium-severity error — more of a blocker than a risk. The steps above resolve it.

Official resources & further reading