The shard iterator used to read DynamoDB Streams has expired after 15 minutes of inactivity. You need to obtain a fresh shard iterator before making subsequent GetRecords requests.
The ExpiredIteratorException occurs when you attempt to use a shard iterator that is no longer valid. In DynamoDB Streams, a shard iterator is a pointer that allows you to read records from a specific shard and expires 15 minutes after it is created. If you don't call GetRecords within this window, or if there's a 15-minute gap between calls, the iterator becomes invalid. This is a stream consumption issue that typically occurs in applications that process DynamoDB stream events asynchronously, have intermittent traffic, or experience processing delays. When the iterator expires, you cannot retrieve records until you obtain a new iterator via GetShardIterator.
DynamoDB Streams shard iterators expire 15 minutes after creation. A new iterator is returned in the NextShardIterator field of every GetRecords response. You should always use this NextShardIterator for the next call.
The timeline looks like:
- Time 0: Call GetShardIterator → receive shardIterator (valid until Time 15)
- Time 5: Call GetRecords(shardIterator) → receive NextShardIterator (valid until Time 20)
- Time 20: Call GetRecords(NextShardIterator) → receive new NextShardIterator (valid until Time 35)
If you wait 20 minutes before the next GetRecords call using the original iterator, it will expire.
The most common fix is to ensure you're using the iterator returned by GetRecords, not reusing the original iterator.
In Node.js with AWS SDK v3:
import { DynamoDBStreamsClient, GetShardIteratorCommand, GetRecordsCommand }
from "@aws-sdk/client-dynamodb-streams";
const client = new DynamoDBStreamsClient({ region: "us-east-1" });
// Initial iterator
const iteratorResponse = await client.send(new GetShardIteratorCommand({
StreamArn: "arn:aws:dynamodb:region:account:table/my-table/stream/...",
ShardId: "shard-id",
ShardIteratorType: "TRIM_HORIZON"
}));
let shardIterator = iteratorResponse.ShardIterator;
// Loop: always use the NextShardIterator returned
while (true) {
const recordsResponse = await client.send(new GetRecordsCommand({
ShardIterator: shardIterator
}));
// Process records...
console.log("Records:", recordsResponse.Records);
// THIS IS CRITICAL: Use NextShardIterator, not the original
shardIterator = recordsResponse.NextShardIterator;
if (!shardIterator) break; // Shard has closed
}The key point: Always assign recordsResponse.NextShardIterator back to your iterator variable before the next call.
If you do catch ExpiredIteratorException, get a new iterator immediately. Catch the exception and call GetShardIterator again:
async function getRecordsWithRetry(streamArn, shardId, currentIterator) {
try {
const response = await client.send(new GetRecordsCommand({
ShardIterator: currentIterator
}));
return response;
} catch (error) {
if (error.name === "ExpiredIteratorException") {
console.warn("Iterator expired, obtaining fresh iterator...");
// Get a new iterator starting from LATEST (don't replay)
const newIterator = await client.send(new GetShardIteratorCommand({
StreamArn: streamArn,
ShardId: shardId,
ShardIteratorType: "LATEST" // Skip expired records
}));
// Retry with new iterator
return await client.send(new GetRecordsCommand({
ShardIterator: newIterator.ShardIterator
}));
}
throw error;
}
}Note: Using "LATEST" means you skip any records that expired during processing delays. Use "TRIM_HORIZON" only if you need to replay from the earliest available record.
If using Lambda with DynamoDB Streams, the iterator is typically managed by Lambda. However, if your handler takes too long to process records, the next invocation might receive an expired iterator.
Optimization strategies:
1. Process records asynchronously in batches:
exports.handler = async (event) => {
// Don't wait for all processing to complete
const promises = event.Records.map(record => {
return processRecordAsync(record).catch(err => {
console.error("Record processing failed:", err);
// Still return success to avoid re-processing expired records
});
});
// Use Promise.allSettled to not block on one slow record
await Promise.allSettled(promises);
};2. Increase Lambda timeout to allow processing within 15 minutes.
3. Use SQS batch window instead of Lambda's default concurrent invocations to reduce backlog.
Beyond ExpiredIteratorException, you might also encounter TrimmedDataAccessException. DynamoDB Streams retains records for 24 hours. If you don't consume records for over 24 hours, they're deleted (trimmed).
If you see TrimmedDataAccessException, your consumer is too far behind. Solutions:
- Increase consumer throughput
- Add more Lambda concurrency
- Use dedicated stream consumer application (not Lambda-based polling)
- Consider switching to Kinesis if you need longer retention
Iterator Lifecycle Details: When you call GetShardIterator, AWS returns a unique iterator string. This iterator is bound to a specific shard and becomes valid the moment it's returned. The 15-minute timer starts immediately. This is different from Kinesis, where shard iterators expire after 5 minutes.
ShardIteratorType Options:
- TRIM_HORIZON: Start from oldest available record (use with caution - old records might be trimmed)
- LATEST: Start from newest record (skip everything in between)
- AT_SEQUENCE_NUMBER: Resume from a specific record sequence number
- AFTER_SEQUENCE_NUMBER: Resume after a specific record sequence number
- AT_TIMESTAMP: Resume from a specific timestamp
Stream Data Retention: DynamoDB Streams retain records for exactly 24 hours. After 24 hours, records are automatically deleted. This is non-configurable. If your consumer falls more than 24 hours behind, you'll lose records permanently.
Using Kinesis Client Library (KCL): If you're building a custom consumer, consider using the Kinesis Client Library for DynamoDB Streams, which handles iterator refresh and shard management automatically.
Production Patterns:
- Always wrap GetRecords calls in try-catch for ExpiredIteratorException
- Use exponential backoff before retrying with a fresh iterator
- Log iterator refresh events for monitoring
- Set CloudWatch alarms on ExpiredIteratorException count
- For critical applications, use DynamoDB export snapshots + Streams for point-in-time recovery
ValidationException: The provided key element does not match the schema
How to fix "ValidationException: The provided key element does not match the schema" in DynamoDB
UnrecognizedClientException: The security token included in the request is invalid
How to fix "UnrecognizedClientException: The security token included in the request is invalid" in DynamoDB
TransactionCanceledException: Transaction cancelled
How to fix "TransactionCanceledException: Transaction cancelled" in DynamoDB
RequestLimitExceeded: Throughput exceeds the current throughput limit for your account
How to fix "RequestLimitExceeded: Throughput exceeds the current throughput limit for your account" in DynamoDB
InternalServerError: Internal Server Error
How to fix "InternalServerError" in DynamoDB