How to fix AKS virtual node error in Kubernetes

KubernetesINTERMEDIATEMEDIUM

AKS virtual node errors occur when pods fail to run on Azure Container Instances (ACI) due to networking, permissions, or quota issues. These errors typically involve authentication failures, container group quota limits, or misconfigured subnets.

What this error means

Azure AKS virtual nodes enable you to run Kubernetes pods on Azure Container Instances (ACI) without managing underlying virtual machines. However, virtual nodes require careful configuration of networking, managed identities, and quotas. When virtual node errors occur, pods fail to start, hang in "Creating" status, or encounter authentication issues communicating with the AKS API server and Azure services. Virtual node errors differ from regular Kubernetes pod errors because they involve coordination between the ACI connector (a Virtual Kubelet pod running in your cluster) and Azure Container Instances APIs. Misconfiguration at the Azure networking or IAM level often cascades to pod deployment failures.

How to fix "AKS virtual node error"

1Verify AKS cluster uses advanced networking (Azure CNI)

Virtual nodes only work with Azure CNI networking. Check your AKS cluster:

bash

az aks show --resource-group <rg> --name <cluster-name> --query networkProfile.networkPlugin

Output should be "azure". If "kubenet":
- Virtual nodes cannot be enabled on kubenet clusters
- You must create a new AKS cluster with Azure CNI or migrate workloads

When creating AKS cluster with virtual nodes:

bash

az aks create --resource-group <rg> --name <cluster> --enable-addons virtual-node --vnet-subnet-id <subnet-id> --network-plugin azure

2Verify virtual node was created via managed addon

Check if virtual node is managed by AKS:

bash

kubectl get nodes
kubectl describe node virtual-node-aci-linux  # Or your virtual node name

If the virtual node appears, check if it's from the managed addon:

bash

az aks addon list --resource-group <rg> --name <cluster>
az aks addon show --resource-group <rg> --name <cluster> --addon virtual-node

If virtual nodes addon is disabled, enable it:

bash

az aks enable-addons --resource-group <rg> --name <cluster> --addons virtual-node --subnet-name <subnet-name>

3Check aci-connector-linux pod for errors

The virtual kubelet runs as aci-connector-linux pod:

bash

kubectl get pods -n kube-system | grep aci-connector
kubectl logs -n kube-system -l app=aci-connector-linux -f
kubectl describe pod -n kube-system -l app=aci-connector-linux

Common error patterns:
- "subnet lookup failed" → subnet misconfigured or aci-connector identity lacks Network Contributor
- "authentication failed" → Managed Identity not properly assigned
- "CrashLoopBackOff" → Missing role assignments or network policy blocking connector

For identity issues, verify the aci-connector has correct Managed Identity:

bash

az aks show --resource-group <rg> --name <cluster> --query addonProfiles.virtualNode.identity

4Verify subnet configuration for ACI

The subnet used by ACI must be clean and properly configured:

bash

az network vnet subnet show --resource-group <rg> --vnet-name <vnet> --name <subnet-name> --query "routeTable, delegations"

Requirements:
- No attached route table (routeTable should be null or managed by Azure)
- No other resources should exist in subnet except ACI container groups
- For manual setup, verify subnet was created with ACI connector

If route table is attached:

bash

az network vnet subnet update --resource-group <rg> --vnet-name <vnet> --name <subnet-name> --remove routeTable

5Check Container Instance quota and cleanup

If you see "ContainerGroupQuotaReached" error, the 100 container instance limit per region is hit:

bash

az container list --resource-group <rg> --query "[].{name:name, state:containers[0].instanceView.currentState.state}" -o table

Delete succeeded container instances:

bash

az container delete --resource-group <rg> --name <container-name> --yes

Or cleanup all completed instances:

bash

for container in $(az container list --resource-group <rg> --query "[?containers[0].instanceView.currentState.state=='Succeeded'].name" -o tsv); do
  az container delete --resource-group <rg> --name "$container" --yes
done

For batch job pods, add appropriate cleanup policies in your Kubernetes manifests.

6Fix image registry credentials for ACI

Standard Kubernetes image pull secrets don't work with ACI. Use correct ACI format in pod spec:

yaml

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  imagePullSecrets:
  - name: regcred  # This won't work for ACI
  containers:
  - name: mycontainer
    image: myregistry.azurecr.io/myimage:tag

Instead, configure credentials in the pod spec directly if using private registry:

yaml

spec:
  containers:
  - name: mycontainer
    image: myregistry.azurecr.io/myimage:tag
    imagePullPolicy: IfNotPresent

For Azure Container Registry (ACR), use AKS integration:

bash

az aks update --resource-group <rg> --name <cluster> --attach-acr <acr-name>

This allows ACI virtual nodes to pull images without explicit secrets.

7Ensure pod spec is ACI-compatible

ACI has stricter requirements than regular Kubernetes. Verify:

1. If you provide args, you must also provide command:

yaml

spec:
  containers:
  - name: mycontainer
    image: myimage:tag
    command: ["/bin/sh"]
    args: ["-c", "echo hello"]

2. Check for unsupported fields (ACI may not support some Kubernetes features):
- SecurityContext (runAsUser, fsGroup, seLinuxOptions)
- Workload Identity / Managed Identity (not supported on virtual nodes)
- Daemon sets (cannot run on virtual nodes)
- StatefulSets with PVC (limited support)

3. Use nodeSelector to target virtual node explicitly:

yaml

spec:
  nodeSelector:
    kubernetes.io/hostname: virtual-node-aci-linux  # Or your virtual node name
  tolerations:
  - key: virtual-kubelet.io/provider
    operator: Exists

8Verify managed identity and role assignments

The aci-connector identity must have correct Azure roles:

bash

# Find the aci-connector managed identity
az identity list --resource-group <node-rg> --query "[?contains(name, 'aci-connector')].{name:name, id:id}"

# Verify Network Contributor role on the ACI subnet
az role assignment list --assignee <identity-principal-id> --scope <subnet-id> --query "[].roleDefinitionName"

If missing, assign the role:

bash

az role assignment create --assignee <identity-principal-id> --role "Network Contributor" --scope <subnet-id>

For ACR access (if using private images):

bash

az role assignment create --assignee <identity-principal-id> --role "AcrPull" --scope <acr-id>

Role assignments can take a few minutes to propagate.

How to fix AKS virtual node error in Kubernetes

What this error means

Typical symptoms

Common causes

How to fix "AKS virtual node error"

Advanced notes

Related errors

Official resources & further reading