This error occurs when Elasticsearch operations exceed their configured timeout limits, causing tasks to be cancelled. Common causes include slow queries, resource contention, network issues, or insufficient cluster resources. The timeout prevents operations from hanging indefinitely but requires tuning timeouts or optimizing queries.
The "ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task" error indicates that an Elasticsearch operation has exceeded its maximum allowed execution time and has been forcibly terminated. This is a protective mechanism that prevents operations from consuming resources indefinitely when they become stuck or excessively slow. This timeout can occur at multiple levels in Elasticsearch: 1. **Search timeouts**: When search queries take too long to execute 2. **Indexing timeouts**: When document indexing operations exceed time limits 3. **Bulk operation timeouts**: When bulk API requests don't complete within the allotted time 4. **Cluster operation timeouts**: During shard allocation, recovery, or snapshot operations 5. **Transport layer timeouts**: For network communication between nodes The timeout is controlled by various configuration settings throughout the Elasticsearch stack, from client-side request timeouts to server-side operation timeouts. When this error appears, it means an operation was taking longer than the system's configured patience threshold, and Elasticsearch decided to abort it rather than let it continue indefinitely.
First, identify which timeout is being exceeded and increase it appropriately:
# Increase search timeout at request level
curl -X GET "localhost:9200/my-index/_search" -u "username:password" -H 'Content-Type: application/json' -d'
{
"query": {
"match_all": {}
},
"timeout": "60s" # Increase from default 30s
}
'
# Set timeout in application code (Java example)
SearchRequest searchRequest = new SearchRequest("my-index");
searchRequest.source(new SearchSourceBuilder()
.query(QueryBuilders.matchAllQuery())
.timeout(TimeValue.timeValueSeconds(60)));
# For bulk operations
BulkRequest bulkRequest = new BulkRequest();
bulkRequest.timeout(TimeValue.timeValueMinutes(2));
# Check current cluster settings
curl -X GET "localhost:9200/_cluster/settings?include_defaults=true&filter_path=**.timeout" -u "username:password"
# Update cluster-level timeout settings
curl -X PUT "localhost:9200/_cluster/settings" -u "username:password" -H 'Content-Type: application/json' -d'
{
"persistent": {
"search.default_search_timeout": "60s",
"indices.memory.index_buffer_size": "10%"
}
}
'Note: Increasing timeouts is a temporary fix. Investigate why operations are slow rather than just increasing timeouts indefinitely.
Identify and optimize queries that are causing timeouts:
# Enable slow query logging
curl -X PUT "localhost:9200/my-index/_settings" -u "username:password" -H 'Content-Type: application/json' -d'
{
"index.search.slowlog.threshold.query.warn": "10s",
"index.search.slowlog.threshold.query.info": "5s",
"index.search.slowlog.threshold.query.debug": "2s",
"index.search.slowlog.threshold.query.trace": "500ms",
"index.search.slowlog.level": "info"
}
'
# Check slow query logs
curl -X GET "localhost:9200/_nodes/hot_threads" -u "username:password"
curl -X GET "localhost:9200/_cat/thread_pool?v" -u "username:password"
# Profile a slow query to see where time is spent
curl -X GET "localhost:9200/my-index/_search" -u "username:password" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"content": "search term"
}
},
"profile": true
}
'
# Common query optimizations:
# 1. Use filters instead of queries for yes/no conditions
# 2. Limit aggregation size with "size": 0 for count-only aggregations
# 3. Use "terminate_after" to limit total hits scanned
# 4. Avoid expensive joins (nested/ parent-child) when possible
# 5. Use keyword fields for exact matches instead of text analysis
# Example optimized query
curl -X GET "localhost:9200/my-index/_search" -u "username:password" -H 'Content-Type: application/json' -d'
{
"query": {
"bool": {
"filter": [
{"term": {"status": "active"}},
{"range": {"timestamp": {"gte": "now-7d/d"}}}
],
"must": [
{"match": {"title": "important"}}
]
}
},
"aggs": {
"category_count": {
"terms": {
"field": "category.keyword",
"size": 10 # Limit aggregation buckets
}
}
},
"terminate_after": 10000, # Stop after 10k documents
"timeout": "30s"
}
'Ensure your cluster has adequate resources and proper configuration:
# Check cluster health and resource usage
curl -X GET "localhost:9200/_cluster/health?pretty" -u "username:password"
curl -X GET "localhost:9200/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m" -u "username:password"
curl -X GET "localhost:9200/_cat/indices?v&h=index,docs.count,store.size,pri.store.size" -u "username:password"
# Adjust thread pool settings for better concurrency
curl -X PUT "localhost:9200/_cluster/settings" -u "username:password" -H 'Content-Type: application/json' -d'
{
"persistent": {
"thread_pool.search.size": 20,
"thread_pool.search.queue_size": 1000,
"thread_pool.write.size": 10,
"thread_pool.write.queue_size": 500,
"thread_pool.get.size": 10,
"thread_pool.get.queue_size": 1000
}
}
'
# Increase circuit breaker limits if hitting memory limits
curl -X PUT "localhost:9200/_cluster/settings" -u "username:password" -H 'Content-Type: application/json' -d'
{
"persistent": {
"indices.breaker.total.limit": "70%",
"indices.breaker.fielddata.limit": "40%",
"indices.breaker.request.limit": "60%"
}
}
'
# Optimize index settings for better performance
curl -X PUT "localhost:9200/my-index/_settings" -u "username:password" -H 'Content-Type: application/json' -d'
{
"index": {
"refresh_interval": "30s", # Reduce refresh frequency for indexing-heavy workloads
"number_of_replicas": 1, # Adjust based on read/write ratio
"translog.durability": "async", # For better indexing performance
"translog.sync_interval": "5s"
}
}
'
# Consider adding more nodes or upgrading hardware if consistently resource-bound
# - Add data nodes for storage/IO capacity
# - Add coordinating nodes for query processing
# - Increase heap size (but not beyond 50% of total RAM)
# - Use SSDs for better disk performanceAdd robust timeout handling and retry logic in your application:
// JavaScript/Node.js example with exponential backoff
const { Client } = require('@elastic/elasticsearch');
const client = new Client({ node: 'http://localhost:9200' });
async function searchWithRetry(index, query, maxRetries = 3) {
let lastError;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
const response = await client.search({
index: index,
body: query,
requestTimeout: 60000, // 60 seconds
});
return response;
} catch (error) {
lastError = error;
// Check if it's a timeout error
if (error.meta?.body?.error?.type === 'elasticsearch_timeout_exception' ||
error.meta?.body?.error?.type === 'timeout_exception') {
console.warn(`Search timeout (attempt ${attempt}/${maxRetries}): ${error.message}`);
if (attempt < maxRetries) {
// Exponential backoff: 1s, 2s, 4s, etc.
const delay = Math.pow(2, attempt - 1) * 1000;
await new Promise(resolve => setTimeout(resolve, delay));
continue;
}
} else {
// Non-timeout error, rethrow immediately
throw error;
}
}
}
throw new Error(`Search failed after ${maxRetries} attempts: ${lastError.message}`);
}
// Java example with retry template
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.retry.annotation.Backoff;
import org.springframework.retry.annotation.Retryable;
@Service
public class SearchService {
@Retryable(
value = {ElasticsearchTimeoutException.class},
maxAttempts = 3,
backoff = @Backoff(delay = 1000, multiplier = 2)
)
public SearchResponse searchWithRetry(SearchRequest request) throws IOException {
return client.search(request, RequestOptions.DEFAULT);
}
}
// Python example with tenacity library
from elasticsearch import Elasticsearch
from tenacity import retry, stop_after_attempt, wait_exponential
client = Elasticsearch(["localhost:9200"])
@retry(
stop=stop_after_attempt(3),
wait=wait_exponential(multiplier=1, min=1, max=10)
)
def search_with_retry(index, query):
try:
return client.search(index=index, body=query, request_timeout=60)
except Exception as e:
if "timeout" in str(e).lower() or "TimeoutException" in str(e):
raise
else:
# Non-timeout errors shouldn't trigger retry
raise Exception("Non-retryable error") from eSet up monitoring to detect timeout patterns before they become critical:
# Monitor timeout metrics
curl -X GET "localhost:9200/_nodes/stats/indices,thread_pool?pretty" -u "username:password"
# Check for rejected threads (indicates queue saturation)
curl -X GET "localhost:9200/_cat/thread_pool?v&h=name,active,queue,rejected,completed" -u "username:password"
# Monitor search latency percentiles
curl -X GET "localhost:9200/_nodes/stats/indices/search?pretty" -u "username:password"
# Set up alerting rules (example using Elasticsearch Watcher)
curl -X PUT "localhost:9200/_watcher/watch/search_timeout_alert" -u "username:password" -H 'Content-Type: application/json' -d'
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"indices": [".monitoring-es-*"],
"body": {
"query": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "now-5m"
}
}
},
{
"term": {
"type": "search_slowlog"
}
}
]
}
},
"aggs": {
"timeout_count": {
"value_count": {
"field": "search_slowlog.total_time"
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.aggregations.timeout_count.value": {
"gt": 10
}
}
},
"actions": {
"send_email": {
"email": {
"to": ["[email protected]"],
"subject": "High search timeout rate detected",
"body": "{{ctx.payload.aggregations.timeout_count.value}} search timeouts detected in the last 5 minutes."
}
}
}
}
'
# Use APM or application performance monitoring
# - Track 95th/99th percentile response times
# - Set alerts when latency exceeds thresholds
# - Monitor error rates for timeout exceptions
# - Track resource utilization (CPU, memory, disk I/O)
# Common monitoring thresholds:
# - Search latency > 5s (warning), > 10s (critical)
# - Indexing latency > 2s (warning), > 5s (critical)
# - Thread pool queue > 50% capacity
# - JVM GC pauses > 1s
# - Disk utilization > 80%## Advanced Timeout Management
### Timeout Hierarchy in Elasticsearch
Elasticsearch has multiple layers of timeouts that can be configured:
1. Transport layer timeouts (network communication):
- transport.tcp.connect_timeout: 30s default
- transport.ping_schedule: -1 (disabled by default)
- transport.tcp.keep_alive: true
2. Discovery and recovery timeouts:
- discovery.zen.join_timeout: 60s
- indices.recovery.max_bytes_per_sec: 40mb
- cluster.service.slow_task_logging_threshold: 30s
3. Search-specific timeouts:
- search.default_search_timeout: No global default (uses request timeout)
- Request-level timeout parameter
- terminate_after: Maximum documents to collect
4. Indexing timeouts:
- Bulk API timeout parameter
- Index creation timeout
- Refresh interval affects visibility latency
### Timeout Best Practices
Production Recommendations:
- Set reasonable timeouts based on SLA requirements (e.g., search: 10s, bulk: 30s)
- Use shorter timeouts for user-facing requests, longer for background jobs
- Implement circuit breakers at application level to fail fast
- Use async operations with callbacks for long-running tasks
- Consider splitting large operations into smaller batches
Troubleshooting Timeout Sources:
# Identify which phase is timing out
curl -X GET "localhost:9200/_search/profile" -u "username:password" -H 'Content-Type: application/json' -d'
{
"query": {
"match": {
"message": "test"
}
},
"profile": true
}
'
# Check for long GC pauses
curl -X GET "localhost:9200/_nodes/stats/jvm?pretty" -u "username:password"
# Monitor disk latency
curl -X GET "localhost:9200/_nodes/stats/fs?pretty" -u "username:password"
# Check for network issues
curl -X GET "localhost:9200/_nodes/hot_threads?type=wait&interval=500ms" -u "username:password"### Performance Tuning for High-Throughput Clusters
1. Query Optimization:
- Use filter context for cacheable conditions
- Limit aggregation precision with precision_threshold
- Use docvalue_fields instead of _source for stored fields
- Enable index.fielddata.cache for frequently accessed fields
2. Index Design:
- Right-size shards (aim for 10-50GB per shard)
- Use index sorting for better compression and query performance
- Implement index lifecycle management (ILM)
- Use data streams for time-series data
3. Hardware Considerations:
- SSDs for data nodes (NVMe preferred)
- Separate master and data nodes in large clusters
- Adequate RAM (50% heap, 50% OS cache)
- Fast network (10Gbps+ for large clusters)
### Timeout in Distributed Systems Context
Remember that in distributed systems like Elasticsearch:
- Timeouts are necessary to prevent cascading failures
- A single slow node can affect the entire cluster
- Network partitions can cause false timeout positives
- Always design for partial failure and implement retry logic with idempotent operations
QueryShardException: No mapping found for [field] in order to sort on
How to fix "QueryShardException: No mapping found for field in order to sort on" in Elasticsearch
IndexNotFoundException: no such index [index_name]
How to fix "IndexNotFoundException: no such index [index_name]" in Elasticsearch
DocumentMissingException: [index][type][id]: document missing
DocumentMissingException: Document missing
ParsingException: Unknown key for a START_OBJECT in [query]
How to fix "ParsingException: Unknown key for a START_OBJECT in [query]" in Elasticsearch
AggregationExecutionException: Aggregation [agg_name] does not support sampling
How to fix "AggregationExecutionException: Aggregation [agg_name] does not support sampling" in Elasticsearch