How to fix failed calling webhook in Kubernetes

Verify webhook is running:

kubectl get pods -n <webhook-namespace> -o wide
kubectl describe pod <webhook-pod> -n <webhook-namespace>
kubectl logs <webhook-pod> -n <webhook-namespace>

Pod should be Running and Ready. Check logs for errors.

Check service configuration:

kubectl get svc -n <webhook-namespace>
kubectl get endpoints <webhook-service> -n <webhook-namespace>
kubectl describe svc <webhook-service> -n <webhook-namespace>

Service should have endpoints pointing to ready pods.

Test TLS connection to webhook:

kubectl port-forward -n <namespace> svc/<webhook-service> 8443:8443

# In another terminal
curl -k https://localhost:8443/validate

If connection fails or times out, webhook is unreachable.

Check timeout and failure policy:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl describe validatingwebhookconfig <name> | grep -A10 timeoutSeconds

Default timeout is 10s (1-30s range). Failur policy: Fail=reject, Ignore=allow on timeout.

Add load balancing:

kubectl scale deployment -n <namespace> <webhook> --replicas=3

Ensure multiple replicas are running and ready. Webhook service will load-balance across them.

Reduce latency in webhook code:
- Remove external API calls or cache responses
- Optimize policy evaluation logic
- Profile webhook for bottlenecks
- Add connection pooling
- Reduce webhook scope via objectSelector/namespaceSelector

Example: validate only specific namespaces or resource types.

Increase timeoutSeconds (use cautiously):

kubectl patch validatingwebhookconfig <name> --type merge \
  -p '{"webhooks":[{"name":"<webhook>","timeoutSeconds":15}]}'

Max 30s. Higher timeouts may mask performance issues.