How to fix AKS virtual node error in Kubernetes

KubernetesINTERMEDIATEMEDIUMUpdated Dec 6, 2025· 96 views

AKS virtual node errors occur when pods fail to run on Azure Container Instances (ACI) due to networking, permissions, or quota issues. These errors typically involve authentication failures, container group quota limits, or misconfigured subnets.

Quick fix

Fixes AKS virtual node error

az aks show --resource-group <rg> --name <cluster-name> --query networkProfile.networkPlugin
  1. Verify AKS cluster uses advanced networking (Azure CNI)
  2. Verify virtual node was created via managed addon
  3. Check aci-connector-linux pod for errors
  4. Verify subnet configuration for ACI
kubernetes — error
az aks show --resource-group <rg> --name <cluster-name> --query networkProfile.networkPlugin
AKS virtual node error

What this error means

Azure AKS virtual nodes enable you to run Kubernetes pods on Azure Container Instances (ACI) without managing underlying virtual machines. However, virtual nodes require careful configuration of networking, managed identities, and quotas. When virtual node errors occur, pods fail to start, hang in "Creating" status, or encounter authentication issues communicating with the AKS API server and Azure services. Virtual node errors differ from regular Kubernetes pod errors because they involve coordination between the ACI connector (a Virtual Kubelet pod running in your cluster) and Azure Container Instances APIs. Misconfiguration at the Azure networking or IAM level often cascades to pod deployment failures.

Typical symptoms

  • Pods scheduled on virtual node remain stuck in "Pending" or "Creating" state indefinitely
  • aci-connector-linux pod crashes repeatedly with "CrashLoopBackOff" status
  • Pod events show "ProviderFailed" or "ContainerInstanceError" messages
  • Error message: "ContainerGroupQuotaReached" with usage at 100/100 Container Instances
  • Virtual node unable to authenticate to ACI: credential or identity errors in pod logs
  • Pod deployment fails with "authentication failed" when using Managed Identity or registry credentials
  • Succeeded container instances accumulate and are never cleaned up from ACI
  • Virtual Kubelet pod logs show network profile or subnet lookup failures

Common causes

  1. Subnet misconfiguration: ACI subnet shares a route table or contains other resources conflicting with container group creation
  2. Managed Identity permissions insufficient: ACI connector lacks Network Contributor or appropriate roles on the target subnet
  3. Container Instance quota reached (limit 100 per region): old succeeded jobs not cleaned up automatically
  4. Network connectivity broken: AKS API server endpoint unreachable from ACI container instances
  5. Image registry credentials in wrong format: ImageRegistryCredential requires specific field names for ACI vs. standard Kubernetes
  6. AKS cluster not using advanced networking (Azure CNI): virtual nodes only work with Azure CNI, not kubenet
  7. ACI virtual node not created with managed addon: manually installed Virtual Kubelet pods not automatically upgraded and may become stale
  8. Pod spec incompatibility: ACI does not support providing args without explicit command field

How to fix "AKS virtual node error"

1Verify AKS cluster uses advanced networking (Azure CNI)

Virtual nodes only work with Azure CNI networking. Check your AKS cluster:

bash
az aks show --resource-group <rg> --name <cluster-name> --query networkProfile.networkPlugin

Output should be "azure". If "kubenet":
- Virtual nodes cannot be enabled on kubenet clusters
- You must create a new AKS cluster with Azure CNI or migrate workloads

When creating AKS cluster with virtual nodes:

bash
az aks create --resource-group <rg> --name <cluster> --enable-addons virtual-node --vnet-subnet-id <subnet-id> --network-plugin azure
2Verify virtual node was created via managed addon

Check if virtual node is managed by AKS:

bash
kubectl get nodes
kubectl describe node virtual-node-aci-linux  # Or your virtual node name

If the virtual node appears, check if it's from the managed addon:

bash
az aks addon list --resource-group <rg> --name <cluster>
az aks addon show --resource-group <rg> --name <cluster> --addon virtual-node

If virtual nodes addon is disabled, enable it:

bash
az aks enable-addons --resource-group <rg> --name <cluster> --addons virtual-node --subnet-name <subnet-name>
3Check aci-connector-linux pod for errors

The virtual kubelet runs as aci-connector-linux pod:

bash
kubectl get pods -n kube-system | grep aci-connector
kubectl logs -n kube-system -l app=aci-connector-linux -f
kubectl describe pod -n kube-system -l app=aci-connector-linux

Common error patterns:
- "subnet lookup failed" → subnet misconfigured or aci-connector identity lacks Network Contributor
- "authentication failed" → Managed Identity not properly assigned
- "CrashLoopBackOff" → Missing role assignments or network policy blocking connector

For identity issues, verify the aci-connector has correct Managed Identity:

bash
az aks show --resource-group <rg> --name <cluster> --query addonProfiles.virtualNode.identity
4Verify subnet configuration for ACI

The subnet used by ACI must be clean and properly configured:

bash
az network vnet subnet show --resource-group <rg> --vnet-name <vnet> --name <subnet-name> --query "routeTable, delegations"

Requirements:
- No attached route table (routeTable should be null or managed by Azure)
- No other resources should exist in subnet except ACI container groups
- For manual setup, verify subnet was created with ACI connector

If route table is attached:

bash
az network vnet subnet update --resource-group <rg> --vnet-name <vnet> --name <subnet-name> --remove routeTable
5Check Container Instance quota and cleanup

If you see "ContainerGroupQuotaReached" error, the 100 container instance limit per region is hit:

bash
az container list --resource-group <rg> --query "[].{name:name, state:containers[0].instanceView.currentState.state}" -o table

Delete succeeded container instances:

bash
az container delete --resource-group <rg> --name <container-name> --yes

Or cleanup all completed instances:

bash
for container in $(az container list --resource-group <rg> --query "[?containers[0].instanceView.currentState.state=='Succeeded'].name" -o tsv); do
  az container delete --resource-group <rg> --name "$container" --yes
done

For batch job pods, add appropriate cleanup policies in your Kubernetes manifests.

6Fix image registry credentials for ACI

Standard Kubernetes image pull secrets don't work with ACI. Use correct ACI format in pod spec:

yaml
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  imagePullSecrets:
  - name: regcred  # This won't work for ACI
  containers:
  - name: mycontainer
    image: myregistry.azurecr.io/myimage:tag

Instead, configure credentials in the pod spec directly if using private registry:

yaml
spec:
  containers:
  - name: mycontainer
    image: myregistry.azurecr.io/myimage:tag
    imagePullPolicy: IfNotPresent

For Azure Container Registry (ACR), use AKS integration:

bash
az aks update --resource-group <rg> --name <cluster> --attach-acr <acr-name>

This allows ACI virtual nodes to pull images without explicit secrets.

7Ensure pod spec is ACI-compatible

ACI has stricter requirements than regular Kubernetes. Verify:

1. If you provide args, you must also provide command:

yaml
spec:
  containers:
  - name: mycontainer
    image: myimage:tag
    command: ["/bin/sh"]
    args: ["-c", "echo hello"]

2. Check for unsupported fields (ACI may not support some Kubernetes features):
- SecurityContext (runAsUser, fsGroup, seLinuxOptions)
- Workload Identity / Managed Identity (not supported on virtual nodes)
- Daemon sets (cannot run on virtual nodes)
- StatefulSets with PVC (limited support)

3. Use nodeSelector to target virtual node explicitly:

yaml
spec:
  nodeSelector:
    kubernetes.io/hostname: virtual-node-aci-linux  # Or your virtual node name
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists
8Verify managed identity and role assignments

The aci-connector identity must have correct Azure roles:

bash
# Find the aci-connector managed identity
az identity list --resource-group <node-rg> --query "[?contains(name, 'aci-connector')].{name:name, id:id}"

# Verify Network Contributor role on the ACI subnet
az role assignment list --assignee <identity-principal-id> --scope <subnet-id> --query "[].roleDefinitionName"

If missing, assign the role:

bash
az role assignment create --assignee <identity-principal-id> --role "Network Contributor" --scope <subnet-id>

For ACR access (if using private images):

bash
az role assignment create --assignee <identity-principal-id> --role "AcrPull" --scope <acr-id>

Role assignments can take a few minutes to propagate.

Advanced notes

Azure AKS virtual nodes are built on Virtual Kubelet, a project that abstracts container orchestration platforms as Kubernetes nodes. Virtual nodes in AKS specifically use Azure Container Instances (ACI) as the underlying compute platform. Key advanced considerations: (1) Virtual nodes and regular node pools operate independently—pods scheduled to virtual nodes do not count toward your VM resource utilization, making them ideal for burst workloads, but costs scale with actual container runtime. (2) ACI does not support certain Kubernetes features: no daemon sets, limited persistent volume support, no init containers with args/command combinations, and no privileged security contexts. (3) Metrics scraping (Prometheus, monitoring agents) does not work on virtual nodes due to API limitations. (4) For production workloads with strict compliance requirements (SELinux, AppArmor, advanced networking policies), avoid virtual nodes or use them for stateless, temporary jobs only. (5) Container instance cleanup is critical for cost management—use finalizers or Kubernetes Job backoffLimit + ttlSecondsAfterFinished to auto-clean succeeded/failed container instances. (6) Multi-tenant scenarios: ACI container groups in the same subnet/region can theoretically leak information; isolate sensitive workloads using separate vNets or disable virtual nodes if not needed. (7) For CI/CD and batch processing, virtual nodes excel; use them for build jobs, testing, and ephemeral workloads rather than long-running services.

Frequently asked questions

What does "AKS virtual node error" mean?
Azure AKS virtual nodes enable you to run Kubernetes pods on Azure Container Instances (ACI) without managing underlying virtual machines. However, virtual nodes require careful configuration of networking, managed identities, and quotas. When virtual node errors occur, pods fail to start, hang in "Creating" status, or encounter authentication issues communicating with the AKS API server and Azure services.
How do I fix "AKS virtual node error"?
AKS virtual node errors occur when pods fail to run on Azure Container Instances (ACI) due to networking, permissions, or quota issues. These errors typically involve authentication failures, container group quota limits, or misconfigured subnets.
What causes "AKS virtual node error"?
Subnet misconfiguration: ACI subnet shares a route table or contains other resources conflicting with container group creation Managed Identity permissions insufficient: ACI connector lacks Network Contributor or appropriate roles on the target subnet Container Instance quota reached (limit 100 per region): old succeeded jobs not cleaned up automatically
Is the "AKS virtual node error" error serious?
It is a medium-severity error — more of a blocker than a risk. The steps above resolve it.

Official resources & further reading