How to fix DocumentMissingException: [index][type][id]: document missing in Elasticsearch

ElasticsearchINTERMEDIATEMEDIUM

This error occurs when attempting to update, delete, or retrieve a document that does not exist in the Elasticsearch index. It commonly happens during concurrent operations or when update requests target non-existent documents.

What this error means

The DocumentMissingException is thrown by Elasticsearch when an operation requires an existing document but cannot find it in the specified index. Unlike index operations that create or overwrite documents, update and delete operations expect the document to already exist at the given ID. This exception is Elasticsearch's way of signaling that your application attempted to modify or access a document that either never existed, was already deleted, or was removed between the time you checked for its existence and when you tried to operate on it. The error includes the index name, type (deprecated in newer versions), and document ID to help identify which document caused the issue. Understanding this error is crucial for building resilient Elasticsearch applications, as it often indicates race conditions, stale references, or logic errors in document lifecycle management.

How to fix "DocumentMissingException: [index][type][id]: document missing"

1Verify the document exists before updating

Use a GET request to check if the document exists before attempting to update it:

bash

# Check if document exists
curl -X GET "localhost:9200/my_index/_doc/doc_id?pretty"

Or in your application code:

javascript

const { body: exists } = await client.exists({
  index: 'my_index',
  id: 'doc_id'
});

if (exists) {
  // Proceed with update
  await client.update({
    index: 'my_index',
    id: 'doc_id',
    body: {
      doc: { field: 'value' }
    }
  });
}

This prevents attempting updates on non-existent documents.

2Use upsert to insert if document does not exist

Add an upsert clause to your update request to create the document if it doesn't exist:

bash

curl -X POST "localhost:9200/my_index/_update/doc_id?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "doc": {
    "field": "updated_value",
    "counter": 1
  },
  "upsert": {
    "field": "initial_value",
    "counter": 1,
    "created_at": "2024-01-01T00:00:00Z"
  }
}
'

In JavaScript/Node.js:

javascript

await client.update({
  index: 'my_index',
  id: 'doc_id',
  body: {
    doc: {
      field: 'updated_value',
      counter: 1
    },
    upsert: {
      field: 'initial_value',
      counter: 1,
      created_at: new Date().toISOString()
    }
  }
});

The upsert field defines what to insert if the document doesn't exist.

3Use doc_as_upsert for simpler upsert operations

Set doc_as_upsert to true to use the doc content as both update and insert:

bash

curl -X POST "localhost:9200/my_index/_update/doc_id?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "doc": {
    "field": "value",
    "counter": 1
  },
  "doc_as_upsert": true
}
'

In Python:

python

from elasticsearch import Elasticsearch

es = Elasticsearch(['localhost:9200'])

es.update(
    index='my_index',
    id='doc_id',
    body={
        'doc': {
            'field': 'value',
            'counter': 1
        },
        'doc_as_upsert': True
    }
)

Note: There are known edge cases where doc_as_upsert can still throw DocumentMissingException in high-concurrency scenarios.

4Handle the exception gracefully in application code

Wrap update operations in try-catch blocks to handle missing documents:

javascript

try {
  await client.update({
    index: 'my_index',
    id: 'doc_id',
    body: {
      doc: { field: 'value' }
    }
  });
} catch (error) {
  if (error.meta?.statusCode === 404) {
    console.log('Document not found, creating new document');
    await client.index({
      index: 'my_index',
      id: 'doc_id',
      body: { field: 'value' }
    });
  } else {
    throw error;
  }
}

In Java:

java

try {
    UpdateRequest request = new UpdateRequest("my_index", "doc_id")
        .doc("field", "value");
    client.update(request, RequestOptions.DEFAULT);
} catch (ElasticsearchException e) {
    if (e.status() == RestStatus.NOT_FOUND) {
        IndexRequest indexRequest = new IndexRequest("my_index")
            .id("doc_id")
            .source("field", "value");
        client.index(indexRequest, RequestOptions.DEFAULT);
    } else {
        throw e;
    }
}

5Use index operation instead of update for idempotent writes

If you want to overwrite the entire document regardless of existence, use the index API:

bash

curl -X PUT "localhost:9200/my_index/_doc/doc_id?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "field": "value",
  "counter": 1,
  "updated_at": "2024-01-01T00:00:00Z"
}
'

In application code:

javascript

await client.index({
  index: 'my_index',
  id: 'doc_id',
  body: {
    field: 'value',
    counter: 1,
    updated_at: new Date().toISOString()
  }
});

Index operations create the document if it doesn't exist or replace it if it does, never throwing DocumentMissingException.

6Add retry_on_conflict for concurrent update scenarios

Configure retry logic to handle version conflicts that might precede document deletion:

bash

curl -X POST "localhost:9200/my_index/_update/doc_id?retry_on_conflict=3&pretty" \
  -H 'Content-Type: application/json' -d'
{
  "doc": {
    "field": "value"
  },
  "doc_as_upsert": true
}
'

In client code:

javascript

await client.update({
  index: 'my_index',
  id: 'doc_id',
  retry_on_conflict: 3,
  body: {
    doc: { field: 'value' },
    doc_as_upsert: true
  }
});

This helps handle concurrent updates but won't prevent DocumentMissingException if the document is truly gone.

7Review bulk operation error handling

When using bulk operations, check individual item responses for errors:

javascript

const { body: bulkResponse } = await client.bulk({
  refresh: true,
  body: [
    { update: { _index: 'my_index', _id: 'doc1' } },
    { doc: { field: 'value1' }, doc_as_upsert: true },
    { update: { _index: 'my_index', _id: 'doc2' } },
    { doc: { field: 'value2' }, doc_as_upsert: true },
  ]
});

if (bulkResponse.errors) {
  const erroredDocuments = [];
  bulkResponse.items.forEach((action, i) => {
    const operation = Object.keys(action)[0];
    if (action[operation].error) {
      erroredDocuments.push({
        status: action[operation].status,
        error: action[operation].error,
        operation: body[i * 2],
        document: body[i * 2 + 1]
      });
    }
  });

  console.log('Failed documents:', erroredDocuments);
}

This allows you to handle missing documents individually without failing the entire bulk operation.

Advanced notes

Race Conditions and Eventual Consistency

DocumentMissingException is often a symptom of race conditions in distributed systems. If you have multiple services or processes operating on the same documents, consider implementing optimistic locking using the _seq_no and _primary_term fields:

javascript

// Get document with sequence number
const { body: doc } = await client.get({
  index: 'my_index',
  id: 'doc_id'
});

// Update with optimistic locking
await client.update({
  index: 'my_index',
  id: 'doc_id',
  if_seq_no: doc._seq_no,
  if_primary_term: doc._primary_term,
  body: {
    doc: { field: 'new_value' }
  }
});

This ensures the document hasn't changed between read and write operations.

Known Issues with doc_as_upsert

There are documented cases where doc_as_upsert can still produce DocumentMissingException in high-concurrency scenarios, particularly during retries after version conflicts. This appears to be related to timing issues in Elasticsearch's internal retry mechanism. If you encounter this, consider using explicit upsert fields instead of doc_as_upsert, or implement application-level retry logic with exponential backoff.

Scripted Updates and Missing Documents

When using scripted updates, you can detect missing documents within the script using ctx._source:

bash

curl -X POST "localhost:9200/my_index/_update/doc_id?pretty" \
  -H 'Content-Type: application/json' -d'
{
  "script": {
    "source": "ctx._source.counter += params.count",
    "params": {
      "count": 1
    }
  },
  "upsert": {
    "counter": 1
  }
}
'

The script runs only if the document exists; otherwise, the upsert document is inserted.

Index Refresh and Visibility

If you're experiencing DocumentMissingException immediately after creating a document, the issue might be index refresh timing. By default, Elasticsearch refreshes indexes every second. For immediate visibility:

javascript

// Index with immediate refresh
await client.index({
  index: 'my_index',
  id: 'doc_id',
  refresh: true,  // or 'wait_for'
  body: { field: 'value' }
});

However, using refresh on every operation impacts performance and should be used sparingly.

Monitoring Document Lifecycle

For production systems experiencing frequent DocumentMissingExceptions, implement audit logging to track document creation, updates, and deletions:

javascript

const auditLog = {
  timestamp: new Date().toISOString(),
  operation: 'update',
  index: 'my_index',
  doc_id: 'doc_id',
  success: false,
  error: 'document_missing_exception'
};

await logAuditEvent(auditLog);

This helps identify patterns like which documents are frequently missing and whether the issue is systematic or isolated.

How to fix DocumentMissingException: [index][type][id]: document missing in Elasticsearch

What this error means

Typical symptoms

Common causes

How to fix "DocumentMissingException: [index][type][id]: document missing"

Advanced notes

Related errors

Official resources & further reading