Service account tokens in Kubernetes expire after a default period (1 hour for most tokens), causing authentication failures and pod communication breakdowns. When a token expires, the API server rejects requests with "invalid bearer token" errors. Fix by refreshing tokens, rotating service accounts, or using projected volumes for automatic token refresh.
Fixes Token expired
# Create a token review to check validity
kubectl create -f - <<EOF
apiVersion: authentication.k8s.io/v1
kind: TokenReview
metadata:
name: token-check
spec:
token: <your-token-here>
EOF
# Check response - if authenticated is false, token is expired or invalid
kubectl get tokenreview token-check -o jsonpath='{.status.authenticated}'# Create a token review to check validitykubectl create -f - <<EOFapiVersion: authentication.k8s.io/v1kind: TokenReviewmetadata:name: token-checkspec:token: <your-token-here>EOF# Check response - if authenticated is false, token is expired or invalidkubectl get tokenreview token-check -o jsonpath='{.status.authenticated}'Token expired
Kubernetes uses bearer tokens for authentication between components, pods, and the API server. Service account tokens have expiration dates to improve security. When a token expires, the API server rejects all requests using that token with an authentication error. This breaks pod-to-API communication, pod-to-pod service account authentication, and kubectl access if using token-based auth.
Verify if a token is actually expired:
# Create a token review to check validity
kubectl create -f - <<EOF
apiVersion: authentication.k8s.io/v1
kind: TokenReview
metadata:
name: token-check
spec:
token: <your-token-here>
EOF
# Check response - if authenticated is false, token is expired or invalid
kubectl get tokenreview token-check -o jsonpath='{.status.authenticated}'If using OIDC (Azure, Google, etc.):
# For Azure (AKS)
az aks get-credentials --resource-group <group> --name <cluster>
# For GKE
gcloud container clusters get-credentials <cluster-name> --zone <zone>
# For AWS (EKS)
aws eks update-kubeconfig --name <cluster> --region <region>This refreshes the OIDC token and updates ~/.kube/config.
Create a short-lived token with automatic rotation:
# Generate a token valid for 1 hour (3600 seconds)
kubectl create token <service-account-name> -n <namespace> --duration=3600s
# Or longer (max 168 hours / 7 days)
kubectl create token <service-account-name> -n <namespace> --duration=168h
# Script can call TokenRequest API directly for automatic rotation:
curl -X POST https://kubernetes.default/api/v1/namespaces/<ns>/serviceaccounts/<sa>/token \\
-H "Authorization: Bearer $BEARER_TOKEN" \\
-d '{"apiVersion":"authentication.k8s.io/v1","kind":"TokenRequest","spec":{"expirationSeconds":3600}}'Configure pods to auto-rotate tokens using volume projection:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
serviceAccountName: my-service-account
containers:
- name: app
image: myapp:latest
volumeMounts:
- mountPath: /var/run/secrets/tokens
name: token
readOnly: true
volumes:
- name: token
projected:
sources:
- serviceAccountToken:
path: token
expirationSeconds: 3600 # Kubelet auto-rotates 80% through lifespan
audience: https://kubernetes.defaultKubelet automatically refreshes tokens before expiry.
If you need a non-expiring token (not recommended):
# Kubernetes v1.24+ does NOT auto-create service account secrets
# Manually create one:
kubectl create secret generic <sa-token-secret> \\
--from-literal=token=$(kubectl create token <sa-name>) \\
-n <namespace>
# Or use a manual secret approach (not auto-rotated):
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: my-sa-secret
annotations:
kubernetes.io/service-account.name: my-sa
type: kubernetes.io/service-account-token
EOFWarning: Non-expiring tokens reduce security.
For kubectl access with expired OIDC tokens:
# Backup your kubeconfig first
cp ~/.kube/config ~/.kube/config.backup
# Edit and remove expired token fields
kubectl config unset users.<user-name>.user.auth-provider.config.access-token
kubectl config unset users.<user-name>.user.auth-provider.config.refresh-token
# Or manually edit
vim ~/.kube/config
# Remove the access-token and refresh-token fields
# Next kubectl command will trigger re-auth
kubectl get nodesIf cluster is inaccessible due to expired admin tokens:
# Allow kube-apiserver to extend token lifetime (recovery only)
# Add to kube-apiserver manifests:
sudo vim /etc/kubernetes/manifests/kube-apiserver.yaml
# Add flag:
# - --service-account-extend-token-expiration=true
# Restart apiserver
kubectl delete pod -n kube-system kube-apiserver-$(hostname)
# This grants 1-year tokens temporarily (use for migration only)Kubernetes v1.20+ introduced projected service account tokens with automatic rotation (kubelet rotates when 80% of expiration reached). Kubernetes v1.24 disabled automatic generation of non-expiring secret-based tokens for security. OIDC flows require handling both id_token (short-lived, ~1hr) and refresh_token (long-lived). For CI/CD, use TokenRequest API or create tokens with --duration flags rather than storing static tokens. Service account tokens for pods should use projected volumes. Admin kubeconfig tokens from Azure/GCP/AWS use OIDC, which auto-refreshes; explicitly clear kubeconfig if refresh fails. For on-prem/kubeadm clusters, manage token lifetime via kube-apiserver flags. Never rely on non-expiring tokens; enable rotation instead.