Introduction
In event-driven Kubernetes architectures, CPU and memory utilization often fail to reflect real system pressure. A worker pod may sit idle from a CPU perspective while thousands of messages pile up in an Amazon SQS queue. In other cases, pods may continue running long after a traffic spike has passed.
For asynchronous, queue-based workloads, backlog is the true scaling signal – not infrastructure utilization.
In this article, you’ll learn:
- Why Amazon SQS queue depth is a better autoscaling metric than CPU for event-driven workers
- How to scale workloads on Amazon EKS using KEDA with AWS pod identity
- How KEDA calculates desired replicas from queue depth
- How to tune queueLength, activationQueueLength, cooldowns, and HPA behavior
- How to validate scaling behavior and troubleshoot common issues
Why this matters:
In queue-driven systems, delayed message processing directly impacts users and downstream systems. Scaling based on SQS depth aligns autoscaling with actual demand, enabling faster burst handling and lower idle costs when queues are empty.
1. Prerequisites
Before implementing KEDA-based autoscaling with Amazon SQS, ensure you have:
- An Amazon EKS cluster running a supported Kubernetes version
- KEDA installed in the cluster (for example, via the kedacore Helm chart)
- An AWS authentication method selected (IRSA or EKS Pod Identity)
- An existing Amazon SQS queue
- A worker deployment designed to consume messages from the queue
2. Architecture and Request Flow
This architecture relies on KEDA observing Amazon SQS queue metrics and managing a Kubernetes Horizontal Pod Autoscaler (HPA) based on backlog.
Request flow:
- Producers send messages to the Amazon SQS queue
- KEDA polls queue attributes to determine backlog
- KEDA updates the target metrics in the HPA
- The HPA scales the worker Deployment
- Kubernetes schedules additional Pods to process messages
- As the queue drains, replicas scale back down
This model keeps autoscaling decisions tied directly to outstanding work.
3. Implementation Steps
Install KEDA via Helm
The recommended way to install KEDA is via Helm:
# Add the KEDA Helm repository
helm repo add kedacore https://kedacore.github.io/charts
helm repo update
# Install KEDA into a dedicated namespace
helm install keda kedacore/keda \
--namespace keda \
--create-namespaceWhat this does:
Deploys the KEDA operator and metrics API server, along with CRDs such as ScaledObject and TriggerAuthentication.
Deploy the Worker with AWS Identity Attached
An SQS consumer typically requires permissions such as:
- GetQueueAttributes
- GetQueueUrl
- ReceiveMessage
- DeleteMessage
- ChangeMessageVisibility
Example IAM policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ConsumeFromQueue",
"Effect": "Allow",
"Action": [
"sqs:GetQueueAttributes",
"sqs:GetQueueUrl",
"sqs:ReceiveMessage",
"sqs:DeleteMessage",
"sqs:ChangeMessageVisibility"
],
"Resource": "arn:aws:sqs:us-east-1:123456789012:orders-queue"
}
]
}What this does:
Enforces least-privilege access by granting permissions only for the required SQS queue.
Configure KEDA Authentication and ScaledObject
KEDA connects the deployment to the SQS queue using TriggerAuthentication and ScaledObject. Using queueURLFromEnv avoids hardcoding the queue URL.
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: sqs-processor-auth
namespace: workers
spec:
podIdentity:
provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: sqs-processor
namespace: workers
spec:
scaleTargetRef:
name: sqs-processor
pollingInterval: 10
cooldownPeriod: 120
minReplicaCount: 0
maxReplicaCount: 30
advanced:
horizontalPodAutoscalerConfig:
behavior:
scaleDown:
stabilizationWindowSeconds: 60
triggers:
- type: aws-sqs-queue
authenticationRef:
name: sqs-processor-auth
metadata:
queueURLFromEnv: QUEUE_URL
awsRegion: us-east-1
queueLength: "10"
activationQueueLength: "1"What this does:
Defines how KEDA authenticates with AWS and how replicas are calculated. Each pod targets 10 messages, and scale-to-zero is enabled when the queue is empty.
4. Replica Calculation Logic
KEDA calculates outstanding work as:
ApproximateNumberOfMessages
+ ApproximateNumberOfMessagesNotVisible
(+ delayed messages, if enabled)
Replica calculation:
desired replicas = ceil(outstanding messages / queueLength)
Example (queueLength: “10”):
- 0 messages → 0 pods
- 8 messages → 1 pod
- 25 messages → 3 pods
- 95 messages → 10 pods
Final replica counts are bounded by minReplicaCount and maxReplicaCount.
5. Verification
After applying the manifests:
1. Send 50 messages to the SQS queue
2. Inspect the ScaledObject:
kubectl get scaledobject sqs-processor -n workers
3. Watch pod scaling:
kubectl get pods -n workers -w
4. Confirm pods scale back to zero after processing completes
6. Troubleshooting Common Issues
No scale-out
- Cause: Incorrect queue URL or IAM permissions
- Check: kubectl describe scaledobject, KEDA operator logs
Too many pods
- Cause: In-flight messages counted
- Check: scaleOnInFlight, SQS visibility timeout
Never scales to zero
- Cause: Delayed or unacknowledged messages
- Check: Queue attributes and application behavior
HPA exists but no scaling
- Cause: Authentication or metric fetch errors
- Check: KEDA logs and fallback status
7. Production Tuning Tips
- Choose queueLength based on throughput, not guesswork
- Tune cooldown and HPA behavior separately – they control different scale-down paths
- Enable fallback replicas for resilience if metrics become unavailable
- Decide whether in-flight messages should count based on visibility timeout behavior
Conclusion
For queue-driven workloads, CPU-based autoscaling is often too indirect. KEDA enables Kubernetes to scale directly on Amazon SQS backlog – the metric that actually reflects demand.
The result is faster reaction to bursts, lower idle cost, and scaling behavior that aligns closely with real workload pressure. For event-driven systems, scaling on queue depth is often the autoscaling model teams needed all along.
Check out more of our blog posts here.
