How to fix Error response from daemon: failed to get GCP credentials in Docker

DockerINTERMEDIATEHIGH

This error occurs when Docker's gcplogs logging driver cannot authenticate with Google Cloud Logging. The most common cause is that the Docker daemon itself lacks access to Google Cloud credentials, which must be set via environment variables in the daemon's systemd configuration, not in the container or shell environment.

What this error means

When you configure Docker to use the gcplogs logging driver, the Docker daemon attempts to authenticate with Google Cloud Logging to send container logs. This authentication requires valid Google Cloud credentials. The "failed to get GCP credentials" error indicates that Docker cannot locate or use valid authentication credentials. This happens because: 1. **The Docker daemon needs credentials, not the container**: A common misconception is that setting `GOOGLE_APPLICATION_CREDENTIALS` in your shell, container environment, or docker-compose.yml is sufficient. However, the gcplogs driver runs within the Docker daemon process itself, so the daemon needs access to the credentials. 2. **Automatic credential discovery fails**: When running on Google Cloud (GCE/GKE), Docker discovers credentials from the instance metadata service. Outside of Google Cloud, you must explicitly provide a service account key file and point the daemon to it. 3. **Service account permissions missing**: Even with valid credentials, the service account must have the "Logs Writer" role (`roles/logging.logWriter`) or equivalent permissions to write to Cloud Logging. This is a critical error because containers configured to use gcplogs will fail to start until authentication is properly configured.

How to fix "Error response from daemon: failed to get GCP credentials"

1Verify the gcplogs driver is the cause

First, confirm that the gcplogs driver is causing the issue by checking your logging configuration:

Check container logging configuration:

bash

# Check Docker daemon default logging driver
docker info | grep "Logging Driver"

# Check specific container configuration
docker inspect <container_name> | grep -A 10 "LogConfig"

Check docker-compose.yml:

Look for gcplogs driver configuration:

yaml

services:
  myapp:
    image: myapp:latest
    logging:
      driver: gcplogs
      options:
        gcp-project: "my-project-id"

Check daemon.json:

bash

cat /etc/docker/daemon.json

If it contains "log-driver": "gcplogs", this confirms the configuration.

2Create a Google Cloud service account key

If you don't already have a service account key file, create one:

Create a service account:

bash

# Create service account
gcloud iam service-accounts create docker-logging \
  --description="Service account for Docker gcplogs driver" \
  --display-name="Docker Logging"

# Grant Logs Writer permission
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:docker-logging@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/logging.logWriter"

Generate the key file:

bash

gcloud iam service-accounts keys create /etc/docker/gcp-logging-key.json \
  --iam-account=docker-logging@YOUR_PROJECT_ID.iam.gserviceaccount.com

Set proper permissions on the key file:

bash

sudo chmod 600 /etc/docker/gcp-logging-key.json
sudo chown root:root /etc/docker/gcp-logging-key.json

Important: Do not use the Owner role for the service account. The "Logs Writer" role is sufficient and follows the principle of least privilege.

3Configure Docker daemon with credentials (systemd)

The key fix is to set the GOOGLE_APPLICATION_CREDENTIALS environment variable for the Docker daemon itself, not for your shell or containers.

Create a systemd override file:

bash

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo nano /etc/systemd/system/docker.service.d/gcp-credentials.conf

Add the following content:

ini

[Service]
Environment="GOOGLE_APPLICATION_CREDENTIALS=/etc/docker/gcp-logging-key.json"

Reload systemd and restart Docker:

bash

sudo systemctl daemon-reload
sudo systemctl restart docker

Verify the environment variable is set:

bash

sudo systemctl show docker --property=Environment

You should see:

bash

Environment=GOOGLE_APPLICATION_CREDENTIALS=/etc/docker/gcp-logging-key.json

4Configure gcplogs driver options

When running outside of Google Cloud, you must also specify the project ID:

Set as default in daemon.json:

bash

sudo nano /etc/docker/daemon.json

json

{
  "log-driver": "gcplogs",
  "log-opts": {
    "gcp-project": "your-gcp-project-id",
    "gcp-meta-name": "docker-host-name"
  }
}

Or per container in docker-compose.yml:

yaml

services:
  myapp:
    image: myapp:latest
    logging:
      driver: gcplogs
      options:
        gcp-project: "your-gcp-project-id"
        gcp-log-cmd: "true"

Or at runtime:

bash

docker run -d \
  --log-driver=gcplogs \
  --log-opt gcp-project=your-gcp-project-id \
  your-image

Restart Docker after daemon.json changes:

bash

sudo systemctl restart docker

5Test the gcplogs driver

Verify that the gcplogs driver works correctly:

Run a test container:

bash

docker run --rm \
  --log-driver=gcplogs \
  --log-opt gcp-project=your-gcp-project-id \
  alpine echo "Test log message to GCP"

If the container runs successfully, the gcplogs driver is working.

Check Google Cloud Console:

1. Go to [Cloud Logging](https://console.cloud.google.com/logs)
2. Select your project
3. Filter by:
- Resource type: "Global"
- Log name: Search for logs with your container name

Use gcloud to verify logs:

bash

gcloud logging read "resource.type=global" --limit=5 --format=json

You should see the "Test log message to GCP" entry.

6Fix GCE/GKE credential issues

If you're running on Google Cloud infrastructure but still getting this error, check the VM's service account configuration:

Check VM service account and scopes:

bash

# From inside the GCE VM
curl -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email

curl -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/scopes

Required scopes:

The VM needs one of these OAuth scopes:
- https://www.googleapis.com/auth/logging.write
- https://www.googleapis.com/auth/cloud-platform

Update VM scopes if needed:

bash

# Stop the VM first
gcloud compute instances stop INSTANCE_NAME --zone=ZONE

# Set the required scope
gcloud compute instances set-service-account INSTANCE_NAME \
  --zone=ZONE \
  --scopes=logging-write,monitoring-write

# Start the VM
gcloud compute instances start INSTANCE_NAME --zone=ZONE

Verify service account has Logs Writer role:

bash

gcloud projects get-iam-policy YOUR_PROJECT_ID \
  --flatten="bindings[].members" \
  --filter="bindings.role:roles/logging.logWriter"

7Configure Container-Optimized OS (cos)

On Container-Optimized OS (used by GKE nodes and GCE container VMs), the /etc directory doesn't persist across reboots. Use cloud-init instead:

Create a cloud-init config:

yaml

#cloud-config
write_files:
- path: /etc/docker/daemon.json
  content: |
    {
      "log-driver": "gcplogs"
    }
runcmd:
- systemctl restart docker

Set via instance metadata:

bash

gcloud compute instances add-metadata INSTANCE_NAME \
  --zone=ZONE \
  --metadata-from-file user-data=cloud-init.yaml

For new instances:

bash

gcloud compute instances create INSTANCE_NAME \
  --zone=ZONE \
  --image-family=cos-stable \
  --image-project=cos-cloud \
  --metadata-from-file user-data=cloud-init.yaml \
  --scopes=logging-write

Note: The VM needs to be rebooted for cloud-init changes to take effect.

8Temporary workaround: use json-file driver

If you need containers running immediately while fixing the gcplogs authentication, switch to the default json-file driver:

Override per container:

bash

docker run --log-driver=json-file myimage

Override in docker-compose.yml:

yaml

services:
  myapp:
    image: myapp:latest
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Change daemon default:

bash

sudo nano /etc/docker/daemon.json

json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

bash

sudo systemctl restart docker

Important: This is a temporary workaround. Return to gcplogs once authentication is properly configured for centralized logging.

Advanced notes

### Common Mistakes When Configuring gcplogs

Mistake 1: Setting credentials in the wrong place

yaml

# WRONG - This sets the variable for the container, not the Docker daemon
services:
  myapp:
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/key.json
    volumes:
      - ./key.json:/key.json

The gcplogs driver runs in the Docker daemon process, not in the container. The daemon doesn't see environment variables set in container configuration.

Mistake 2: Setting credentials in your shell only

bash

# WRONG - Only affects your shell session, not the Docker daemon
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json
docker run --log-driver=gcplogs myimage  # Still fails

The Docker daemon is a separate process (usually started by systemd) and doesn't inherit your shell's environment.

### Alternative: Use Docker Desktop on macOS/Windows

Docker Desktop for macOS and Windows handles gcplogs differently. You can set the environment variable in Docker Desktop settings:

1. Open Docker Desktop
2. Go to Settings/Preferences
3. Navigate to Docker Engine
4. The daemon configuration is shown as JSON

For credentials, you may need to mount the credentials file in a shared location and reference it in Docker Desktop's daemon configuration.

### Debugging Credential Issues

Check if the key file is valid:

bash

# Test authentication with the key file
export GOOGLE_APPLICATION_CREDENTIALS=/etc/docker/gcp-logging-key.json
gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud logging write test-log "Test from CLI"

Check Docker daemon logs:

bash

sudo journalctl -u docker.service -f

Look for authentication-related errors when starting containers.

### Using Workload Identity on GKE

On GKE, Workload Identity is the preferred method for authenticating to Google Cloud services:

bash

# Enable Workload Identity on the cluster
gcloud container clusters update CLUSTER_NAME \
  --zone=ZONE \
  --workload-pool=PROJECT_ID.svc.id.goog

# Configure the Kubernetes service account
kubectl annotate serviceaccount default \
  iam.gke.io/gcp-service-account=GSA_NAME@PROJECT_ID.iam.gserviceaccount.com

However, note that gcplogs runs at the node level, not pod level, so Workload Identity doesn't directly apply. Nodes still use their node service account for logging.

### Performance Considerations

The gcplogs driver sends logs synchronously by default. For high-throughput containers, consider:

json

{
  "log-driver": "gcplogs",
  "log-opts": {
    "gcp-project": "your-project",
    "mode": "non-blocking",
    "max-buffer-size": "4m"
  }
}

Warning: In non-blocking mode, logs may be dropped if the buffer fills up faster than logs can be sent to Cloud Logging.

### Verifying Logs in Cloud Logging

Use this filter in Cloud Logging console to find Docker logs:

bash

resource.type="global"
jsonPayload.container.name="/your-container-name"

Or via gcloud:

bash

gcloud logging read 'resource.type="global"' \
  --project=YOUR_PROJECT \
  --limit=10 \
  --format="table(timestamp,jsonPayload.container.name,jsonPayload.data)"

How to fix Error response from daemon: failed to get GCP credentials in Docker

What this error means

Typical symptoms

Common causes

How to fix "Error response from daemon: failed to get GCP credentials"

Advanced notes

Related errors

Official resources & further reading