How to fix Error: Error creating Kinesis Stream: LimitExceededException in Terraform

TerraformINTERMEDIATEHIGHUpdated Dec 31, 2025· 91 views

This error occurs when AWS Kinesis exceeds account-level or shard-level limits. Common causes include reaching maximum concurrent stream creation, shard quota limits, or API rate limits. Resolving it requires requesting quota increases, optimizing stream configuration, or adjusting Terraform parallelism.

Quick fix

Fixes Error: Error creating Kinesis Stream: LimitExceededException

aws kinesis list-streams --region us-east-1
aws kinesis describe-stream --stream-name your-stream-name --region us-east-1
  1. Check current stream creation status
  2. Check your shard quota
  3. Request AWS shard quota increase
  4. Reduce Terraform parallelism
terraform — error
aws kinesis list-streams --region us-east-1
aws kinesis describe-stream --stream-name your-stream-name --region us-east-1
Error: Error creating Kinesis Stream: LimitExceededException

What this error means

AWS Kinesis enforces strict quotas and limits to maintain service stability. A LimitExceededException indicates you have exceeded one of these boundaries. This can happen at several levels: creating too many streams simultaneously (max 5 in CREATING state), exceeding your account's total shard limit, hitting API rate limits (5 calls per second for CreateStream), or triggering throughput limits on individual shards (1 MB/s write or 1,000 records/s). Terraform makes rapid API calls, which can trigger these limits faster than manual configuration.

Typical symptoms

  • Terraform apply fails when creating Kinesis streams
  • Error message contains "LimitExceededException"
  • Works on first apply but fails on subsequent operations
  • Multiple Terraform operations running in parallel cause errors

Common causes

  1. More than 5 streams are in CREATING state simultaneously (API allows only 5 concurrent creations)
  2. Your AWS account has reached the shard limit quota for the region (default limit can be low)
  3. CreateStream API is being called more than 5 times per second
  4. Terraform parallelism setting is too high, causing too many concurrent API calls
  5. Previous streams never finished initializing and are still blocking new creations

How to fix "Error: Error creating Kinesis Stream: LimitExceededException"

1Check current stream creation status

First, verify how many streams are currently in the CREATING state using the AWS CLI:

bash
aws kinesis list-streams --region us-east-1
aws kinesis describe-stream --stream-name your-stream-name --region us-east-1

Look for streams with status "CREATING". If more than 4 are in this state, wait for them to reach ACTIVE status before creating new streams.

2Check your shard quota

Use the DescribeLimits API to check your current shard quota:

bash
aws kinesis describe-limits --region us-east-1

This shows OpenShardCount (current shards) and ShardLimit (your quota). If you're at or near the limit, you need to request a quota increase.

3Request AWS shard quota increase

If you've reached your shard limit, request an increase through AWS Service Quotas:

bash
# Open Service Quotas console and navigate to Kinesis
# Search for "Kinesis Data Streams"
# Select "Shard quota"
# Click "Request quota increase"
# Enter your desired shard count
# Submit (usually approved within hours)

Alternatively, contact AWS Support for faster processing of quota increases.

4Reduce Terraform parallelism

Terraform by default parallelizes resource creation. Since Kinesis has a 5 calls/second limit for CreateStream, reduce parallelism:

bash
terraform apply -parallelism=1

Or permanently set it in your Terraform code:

hcl
terraform {
  required_version = ">= 1.0"
}

Then apply with explicit parallelism or use the command flag.

5Configure AWS provider retry settings

Add retry configuration to your AWS Terraform provider to automatically retry rate-limited requests:

hcl
provider "aws" {
  region       = "us-east-1"
  retry_mode   = "standard"
  max_retries  = 10
}

The standard retry mode includes exponential backoff and will retry rate-limited requests automatically.

6Consider using On-Demand capacity mode

Instead of provisioned mode, use On-Demand for variable workloads:

hcl
resource "aws_kinesis_stream" "example" {
  name             = "my-stream"
  stream_mode_details {
    stream_mode = "ON_DEMAND"
  }
}

On-Demand mode automatically scales and reduces quota conflicts. It starts with 4 shards and scales automatically, supporting up to 2 GB/s write throughput.

Advanced notes

The 5 concurrent stream creation limit is a hard AWS API constraint that cannot be changed per account. To create many streams, implement a creation queue in your Terraform code using for_each with explicit depends_on to serialize creation. For very large-scale deployments, consider whether Kinesis is the right fit or if you should use a managed streaming service with higher quotas. If you're hitting shard limits repeatedly, audit your partition key distribution to ensure even load across shards, which may reduce your required shard count.

Frequently asked questions

What does "Error: Error creating Kinesis Stream: LimitExceededException" mean?
AWS Kinesis enforces strict quotas and limits to maintain service stability. A LimitExceededException indicates you have exceeded one of these boundaries. This can happen at several levels: creating too many streams simultaneously (max 5 in CREATING state), exceeding your account's total shard limit, hitting API rate limits (5 calls per second for CreateStream), or triggering throughput limits on individual shards (1 MB/s write or 1,000 records/s).
How do I fix "Error: Error creating Kinesis Stream: LimitExceededException"?
This error occurs when AWS Kinesis exceeds account-level or shard-level limits. Common causes include reaching maximum concurrent stream creation, shard quota limits, or API rate limits. Resolving it requires requesting quota increases, optimizing stream configuration, or adjusting Terraform parallelism.
What causes "Error: Error creating Kinesis Stream: LimitExceededException"?
More than 5 streams are in CREATING state simultaneously (API allows only 5 concurrent creations) Your AWS account has reached the shard limit quota for the region (default limit can be low) CreateStream API is being called more than 5 times per second
Is the "Error: Error creating Kinesis Stream: LimitExceededException" error serious?
Yes. This is a high-severity error that usually blocks work until it is fixed, but the steps above resolve it.

Official resources & further reading