How to fix Error response from daemon: error while creating fluentd logger: dial tcp: connection refused in Docker

DockerINTERMEDIATEMEDIUM

This error occurs when Docker cannot establish a connection to the Fluentd logging service. The Fluentd daemon may not be running, not yet ready, or the connection address is misconfigured. Containers using the fluentd logging driver will fail to start until the connection succeeds.

What this error means

The "error while creating fluentd logger: dial tcp: connection refused" error indicates that Docker's logging driver is unable to connect to a Fluentd daemon for log forwarding. When you configure a container to use the fluentd logging driver, Docker must establish a TCP connection to Fluentd before the container can start. This error commonly appears in several scenarios: - **Docker Compose environments**: When containers with fluentd logging start before the Fluentd service is ready to accept connections - **Microservices architectures**: When application containers depend on Fluentd for centralized logging but the timing is not properly managed - **CI/CD pipelines**: When Fluentd is deployed as a sidecar or service container that takes time to initialize - **Kubernetes and Swarm**: When log aggregation services are not yet available when application pods/services start The key difference from other logging drivers is that, by default, Docker requires a successful connection to Fluentd before starting the container. This design ensures no logs are lost but creates a hard dependency on Fluentd availability.

How to fix "Error response from daemon: error while creating fluentd logger: dial tcp: connection refused"

1Verify Fluentd is running and listening

First, confirm that Fluentd is actually running and accepting connections on port 24224:

Check if Fluentd container is running:

bash

docker ps | grep fluentd

Test connectivity to Fluentd:

bash

# From the host
nc -zv localhost 24224
curl -v telnet://localhost:24224

# Check what's listening on 24224
ss -tlnp | grep 24224
netstat -tlnp | grep 24224

If Fluentd is not running, start it:

bash

# Using official Fluentd Docker image
docker run -d \
  --name fluentd \
  -p 24224:24224 \
  -p 24224:24224/udp \
  fluent/fluentd:v1.16-1

2Enable async connection mode (recommended)

The most reliable solution is to enable asynchronous connection mode. This allows Docker to buffer logs and retry connecting to Fluentd instead of failing immediately:

Docker run command:

bash

docker run -d \
  --log-driver=fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt fluentd-async=true \
  --log-opt fluentd-retry-wait=1s \
  --log-opt fluentd-max-retries=30 \
  myimage

Docker Compose:

yaml

services:
  myapp:
    image: myimage
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        fluentd-async: "true"
        fluentd-retry-wait: "1s"
        fluentd-max-retries: "30"
        tag: "docker.{{.Name}}"

Important: In Docker Compose, logging options must be strings (wrapped in quotes), including boolean values like "true".

3Expose Fluentd ports and use correct address

The Docker daemon itself runs outside your container network and cannot resolve container service names. You must expose Fluentd's port and use a reachable address:

Docker Compose - Correct configuration:

yaml

services:
  fluentd:
    image: fluent/fluentd:v1.16-1
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    volumes:
      - ./fluent.conf:/fluentd/etc/fluent.conf

  myapp:
    image: myimage
    depends_on:
      - fluentd
    logging:
      driver: fluentd
      options:
        # Use localhost because port is published to host
        fluentd-address: localhost:24224
        fluentd-async: "true"

Common mistake:

yaml

# WRONG - Docker daemon cannot resolve 'fluentd' service name
fluentd-address: fluentd:24224

# CORRECT - Use localhost with published port
fluentd-address: localhost:24224

# OR use specific IP address
fluentd-address: 192.168.1.100:24224

4Start Fluentd before other services

The depends_on directive only ensures container start order, not that Fluentd is ready to accept connections. Use multiple strategies:

Strategy 1: Start Fluentd first manually

bash

# Start Fluentd and wait for it to be ready
docker-compose up -d fluentd
sleep 5  # Give Fluentd time to initialize

# Then start other services
docker-compose up -d

Strategy 2: Use depends_on with condition (Compose v2.1+)

yaml

services:
  fluentd:
    image: fluent/fluentd:v1.16-1
    ports:
      - "24224:24224"
    healthcheck:
      test: ["CMD", "nc", "-z", "localhost", "24224"]
      interval: 5s
      timeout: 3s
      retries: 10

  myapp:
    image: myimage
    depends_on:
      fluentd:
        condition: service_healthy
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224

Strategy 3: Use a startup script with wait loop

bash

#!/bin/bash
# wait-for-fluentd.sh
echo "Waiting for Fluentd..."
until nc -z localhost 24224; do
  echo "Fluentd not ready, retrying..."
  sleep 2
done
echo "Fluentd is ready!"
docker-compose up -d myapp

5Configure Docker daemon defaults (optional)

If all your containers use Fluentd logging, configure the Docker daemon to use it by default with async mode:

Edit /etc/docker/daemon.json:

json

{
  "log-driver": "fluentd",
  "log-opts": {
    "fluentd-address": "localhost:24224",
    "fluentd-async": "true",
    "fluentd-retry-wait": "1s",
    "fluentd-max-retries": "30",
    "fluentd-buffer-limit": "1048576",
    "tag": "docker.{{.Name}}"
  }
}

Restart Docker to apply:

bash

sudo systemctl restart docker

Note: Log options in daemon.json must be strings, so boolean and numeric values must be quoted (e.g., "true", "30").

6Handle container startup failures gracefully

Configure Docker to not fail container startup if Fluentd is temporarily unavailable:

Use the fail-on-startup-error option:

bash

docker run -d \
  --log-driver=fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt fluentd-async=true \
  --log-opt fail-on-startup-error=false \
  myimage

Docker Compose:

yaml

services:
  myapp:
    image: myimage
    logging:
      driver: fluentd
      options:
        fluentd-address: localhost:24224
        fluentd-async: "true"
        fail-on-startup-error: "false"

With fail-on-startup-error: false, the container will start even if Fluentd is unavailable. Docker will buffer logs locally and attempt to reconnect. However, be aware that if the buffer fills up while Fluentd is down, logs may be lost.

7Debug Fluentd configuration issues

If Fluentd is running but still refusing connections, check its configuration:

Basic fluent.conf for accepting Docker logs:

xml

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match docker.**>
  @type stdout
</match>

Check Fluentd logs for errors:

bash

docker logs fluentd

Common Fluentd configuration issues:
- Missing @type forward source configuration
- Wrong port number (not 24224)
- Binding to 127.0.0.1 instead of 0.0.0.0
- Missing required plugins

Verify Fluentd is listening on the correct interface:

bash

docker exec fluentd netstat -tlnp | grep 24224

8Consider alternative logging approaches

If Fluentd reliability is a concern, consider alternative approaches that don't block container startup:

Option 1: Use json-file driver with log scrapers

yaml

services:
  myapp:
    image: myimage
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

  promtail:
    image: grafana/promtail
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro

Option 2: Use sidecar container for log forwarding

yaml

services:
  myapp:
    image: myimage
    volumes:
      - app-logs:/var/log/app

  log-forwarder:
    image: fluent/fluent-bit
    volumes:
      - app-logs:/var/log/app:ro
    depends_on:
      - fluentd

Option 3: Use Docker's local driver with dual logging (Enterprise)

Docker Enterprise supports dual logging, writing to both local files and remote destinations. In Docker Community Edition, consider the approaches above.

Advanced notes

Understanding Docker logging driver architecture:

When you specify a logging driver like fluentd, the Docker daemon (not the container) is responsible for collecting stdout/stderr from the container and forwarding it to the logging backend. This is why:

1. The Docker daemon must be able to reach Fluentd - container-to-container networking doesn't apply
2. Service names from docker-compose.yml cannot be used - the daemon doesn't participate in the container network
3. You must expose Fluentd's port to the host and use localhost or the host's IP

Buffer management:

The fluentd driver uses an in-memory buffer when fluentd-async is enabled:
- Default buffer limit: 1,048,576 events (configurable via fluentd-buffer-limit)
- If the buffer fills up, new log messages will be dropped
- When Fluentd reconnects, buffered messages are sent

Performance considerations:

- Async mode adds minimal CPU overhead but uses memory for buffering
- Synchronous mode (default) has lower memory usage but creates a hard dependency
- For high-throughput applications, tune fluentd-buffer-limit based on expected log volume during Fluentd outages

Troubleshooting DNS resolution:

If you see "lookup fluentd: i/o timeout" instead of "connection refused", it's a DNS issue:

bash

# The Docker daemon cannot resolve the service name
# This won't work:
--log-opt fluentd-address=fluentd:24224

# Use IP address or localhost instead:
--log-opt fluentd-address=192.168.1.100:24224
--log-opt fluentd-address=localhost:24224

Kubernetes considerations:

In Kubernetes, consider using a DaemonSet for Fluentd that collects logs from the node's /var/log/containers directory rather than using Docker's fluentd driver. This approach:
- Doesn't block pod startup
- Works with any container runtime (Docker, containerd, CRI-O)
- Is the recommended pattern for Kubernetes logging

Sub-second precision limitation:

Note that Docker's fluentd logging driver doesn't support sub-second timestamp precision. If you need precise timestamps, consider:
- Adding timestamps within your application
- Using filter_parser in Fluentd to extract timestamps from log messages

How to fix Error response from daemon: error while creating fluentd logger: dial tcp: connection refused in Docker

What this error means

Typical symptoms

Common causes

How to fix "Error response from daemon: error while creating fluentd logger: dial tcp: connection refused"

Advanced notes

Related errors

Official resources & further reading