Service account tokens in Kubernetes expire after a default period (1 hour for most tokens), causing authentication failures and pod communication breakdowns. When a token expires, the API server rejects requests with "invalid bearer token" errors. Fix by refreshing tokens, rotating service accounts, or using projected volumes for automatic token refresh.
Kubernetes uses bearer tokens for authentication between components, pods, and the API server. Service account tokens have expiration dates to improve security. When a token expires, the API server rejects all requests using that token with an authentication error. This breaks pod-to-API communication, pod-to-pod service account authentication, and kubectl access if using token-based auth.
Verify if a token is actually expired:
# Create a token review to check validity
kubectl create -f - <<EOF
apiVersion: authentication.k8s.io/v1
kind: TokenReview
metadata:
name: token-check
spec:
token: <your-token-here>
EOF
# Check response - if authenticated is false, token is expired or invalid
kubectl get tokenreview token-check -o jsonpath='{.status.authenticated}'If using OIDC (Azure, Google, etc.):
# For Azure (AKS)
az aks get-credentials --resource-group <group> --name <cluster>
# For GKE
gcloud container clusters get-credentials <cluster-name> --zone <zone>
# For AWS (EKS)
aws eks update-kubeconfig --name <cluster> --region <region>This refreshes the OIDC token and updates ~/.kube/config.
Create a short-lived token with automatic rotation:
# Generate a token valid for 1 hour (3600 seconds)
kubectl create token <service-account-name> -n <namespace> --duration=3600s
# Or longer (max 168 hours / 7 days)
kubectl create token <service-account-name> -n <namespace> --duration=168h
# Script can call TokenRequest API directly for automatic rotation:
curl -X POST https://kubernetes.default/api/v1/namespaces/<ns>/serviceaccounts/<sa>/token \\
-H "Authorization: Bearer $BEARER_TOKEN" \\
-d '{"apiVersion":"authentication.k8s.io/v1","kind":"TokenRequest","spec":{"expirationSeconds":3600}}'Configure pods to auto-rotate tokens using volume projection:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
serviceAccountName: my-service-account
containers:
- name: app
image: myapp:latest
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: token
readOnly: true
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 3600 # Kubelet auto-rotates 80% through lifespan
audience: https://kubernetes.defaultKubelet automatically refreshes tokens before expiry.
If you need a non-expiring token (not recommended):
# Kubernetes v1.24+ does NOT auto-create service account secrets
# Manually create one:
kubectl create secret generic <sa-token-secret> \\
--from-literal=token=$(kubectl create token <sa-name>) \\
-n <namespace>
# Or use a manual secret approach (not auto-rotated):
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: my-sa-secret
annotations:
kubernetes.io/service-account.name: my-sa
type: kubernetes.io/service-account-token
EOFWarning: Non-expiring tokens reduce security.
For kubectl access with expired OIDC tokens:
# Backup your kubeconfig first
cp ~/.kube/config ~/.kube/config.backup
# Edit and remove expired token fields
kubectl config unset users.<user-name>.user.auth-provider.config.access-token
kubectl config unset users.<user-name>.user.auth-provider.config.refresh-token
# Or manually edit
vim ~/.kube/config
# Remove the access-token and refresh-token fields
# Next kubectl command will trigger re-auth
kubectl get nodesIf cluster is inaccessible due to expired admin tokens:
# Allow kube-apiserver to extend token lifetime (recovery only)
# Add to kube-apiserver manifests:
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
# Add flag:
# - --service-account-extend-token-expiration=true
# Restart apiserver
kubectl delete pod -n kube-system kube-apiserver-$(hostname)
# This grants 1-year tokens temporarily (use for migration only)Kubernetes v1.20+ introduced projected service account tokens with automatic rotation (kubelet rotates when 80% of expiration reached). Kubernetes v1.24 disabled automatic generation of non-expiring secret-based tokens for security. OIDC flows require handling both id_token (short-lived, ~1hr) and refresh_token (long-lived). For CI/CD, use TokenRequest API or create tokens with --duration flags rather than storing static tokens. Service account tokens for pods should use projected volumes. Admin kubeconfig tokens from Azure/GCP/AWS use OIDC, which auto-refreshes; explicitly clear kubeconfig if refresh fails. For on-prem/kubeadm clusters, manage token lifetime via kube-apiserver flags. Never rely on non-expiring tokens; enable rotation instead.
Failed to connect to server: connection refused (HTTP/2)
How to fix "HTTP/2 connection refused" error in Kubernetes
missing request for cpu in container
How to fix "missing request for cpu in container" in Kubernetes HPA
error: invalid configuration
How to fix "error: invalid configuration" in Kubernetes
etcdserver: cluster ID mismatch
How to fix "etcdserver: cluster ID mismatch" in Kubernetes
running with swap on is not supported
How to fix "running with swap on is not supported" in kubeadm