Synchronous communication between services is a liability at scale. One slow dependency makes every service that calls it slow. One failed dependency makes every service that depends on it fail. When traffic spikes, the pressure propagates instantly across every service in the call chain.
SQS and SNS give you the building blocks for asynchronous, decoupled communication that absorbs load spikes, survives partial failures, and scales independently. They are among the most fundamental AWS services, and yet I regularly see teams use them in ways that leave significant reliability on the table.
In this article I will walk you through SQS and SNS individually, the fan-out pattern that combines them, and the production configuration decisions that actually matter: dead letter queues, visibility timeouts, message ordering, and observability.
SQS: What It Is and What You Need to Know
SQS is a managed message queue. Producers write messages to a queue. Consumers read and process them. If the consumer is busy or unavailable, messages wait in the queue. If traffic spikes, messages queue up and get processed as capacity allows. The producer never knows or cares whether the consumer is slow, fast, or temporarily down.
SQS has two queue types. Standard queues deliver messages at least once in an approximately ordered fashion and support essentially unlimited throughput. FIFO queues deliver messages exactly once in strict order but are limited to 3,000 messages per second with batching. Use FIFO queues when order and exactly-once processing are strict requirements. Use Standard queues for everything else.
The visibility timeout is the most important configuration parameter to understand. When a consumer reads a message, it becomes invisible to other consumers for the visibility timeout duration. If the consumer successfully processes the message and deletes it, it is gone. If the consumer crashes or fails to delete within the visibility timeout, the message becomes visible again and gets redelivered. Set your visibility timeout to at least 6 times your average processing time to avoid unnecessary redeliveries.
Setting Up SQS with Terraform
resource "aws_sqs_queue" "order_processing_dlq" {
name = "order-processing-dlq"
message_retention_seconds = 1209600
kms_master_key_id = aws_kms_key.sqs.id
tags = {
Environment = "production"
Purpose = "dead-letter-queue"
}
}
resource "aws_sqs_queue" "order_processing" {
name = "order-processing"
visibility_timeout_seconds = 300
message_retention_seconds = 86400
max_message_size = 262144
delay_seconds = 0
receive_wait_time_seconds = 20
kms_master_key_id = aws_kms_key.sqs.id
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.order_processing_dlq.arn
maxReceiveCount = 3
})
tags = {
Environment = "production"
Purpose = "order-processing"
}
}
resource "aws_sqs_queue_policy" "order_processing" {
queue_url = aws_sqs_queue.order_processing.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Principal = { AWS = aws_iam_role.order_service.arn }
Action = ["sqs:SendMessage"]
Resource = aws_sqs_queue.order_processing.arn
},
{
Effect = "Allow"
Principal = { AWS = aws_iam_role.fulfillment_service.arn }
Action = ["sqs:ReceiveMessage", "sqs:DeleteMessage", "sqs:GetQueueAttributes"]
Resource = aws_sqs_queue.order_processing.arn
}
]
})
}receive_wait_time_seconds = 20 enables long polling. Without long polling, each ReceiveMessage call returns immediately even if the queue is empty, and you burn API calls at a high rate checking an empty queue. Long polling holds the connection open for up to 20 seconds waiting for a message to arrive. For Lambda triggers, SQS handles this automatically. For polling consumers you build yourself, always enable long polling.
maxReceiveCount = 3 in the redrive policy means a message gets delivered up to 3 times before it is sent to the dead letter queue. Choose this number based on your processing logic. If failures are typically transient, 3 to 5 retries gives them time to recover. If failures indicate genuinely bad messages, keep it low so bad messages do not consume processing capacity through repeated retries.
Processing SQS Messages with Lambda
resource "aws_lambda_event_source_mapping" "order_processing" {
event_source_arn = aws_sqs_queue.order_processing.arn
function_name = aws_lambda_function.process_order.arn
batch_size = 10
maximum_batching_window_in_seconds = 5
function_response_types = ["ReportBatchItemFailures"]
}
The ReportBatchItemFailures response type is critical for Lambda SQS processing. Without it, if your Lambda processes 10 messages and 2 fail, all 10 go back to the queue for retry. With it, you report which specific messages failed and only those messages are retried. Here is how to implement it:
import json
import logging
logger = logging.getLogger()
def lambda_handler(event, context):
failed_message_ids = []
for record in event["Records"]:
message_id = record["messageId"]
try:
body = json.loads(record["body"])
process_order(body)
logger.info(f"Processed order {body.get('orderId')} successfully")
except Exception as e:
logger.error(f"Failed to process message {message_id}: {e}")
failed_message_ids.append({"itemIdentifier": message_id})
return {"batchItemFailures": failed_message_ids}
def process_order(order: dict):
order_id = order["orderId"]
validate_order(order)
charge_payment(order)
trigger_fulfillment(order_id)SNS: Pub/Sub at Scale
SNS is a pub/sub messaging service. Publishers send messages to a topic. SNS delivers those messages to every subscriber simultaneously. Subscribers can be SQS queues, Lambda functions, HTTP endpoints, email addresses, or SMS numbers.
The key difference from SQS is fan-out. One SNS message can trigger 10 different processing pipelines simultaneously. This is how you build event-driven architectures where a single business event like an order being placed triggers inventory updates, email notifications, analytics events, and fulfillment processing all in parallel.
resource "aws_sns_topic" "order_events" {
name = "order-events"
kms_master_key_id = aws_kms_key.sns.id
tags = {
Environment = "production"
}
}
resource "aws_sqs_queue" "fulfillment_queue" {
name = "fulfillment-queue"
visibility_timeout_seconds = 300
kms_master_key_id = aws_kms_key.sqs.id
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.fulfillment_dlq.arn
maxReceiveCount = 3
})
}
resource "aws_sqs_queue" "analytics_queue" {
name = "analytics-queue"
visibility_timeout_seconds = 120
kms_master_key_id = aws_kms_key.sqs.id
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.analytics_dlq.arn
maxReceiveCount = 5
})
}
resource "aws_sns_topic_subscription" "order_to_fulfillment" {
topic_arn = aws_sns_topic.order_events.arn
protocol = "sqs"
endpoint = aws_sqs_queue.fulfillment_queue.arn
raw_message_delivery = true
filter_policy = jsonencode({
eventType = ["ORDER_PLACED", "ORDER_UPDATED"]
})
}
resource "aws_sns_topic_subscription" "order_to_analytics" {
topic_arn = aws_sns_topic.order_events.arn
protocol = "sqs"
endpoint = aws_sqs_queue.analytics_queue.arn
raw_message_delivery = true
}
resource "aws_sqs_queue_policy" "allow_sns_fulfillment" {
queue_url = aws_sqs_queue.fulfillment_queue.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Principal = { Service = "sns.amazonaws.com" }
Action = "sqs:SendMessage"
Resource = aws_sqs_queue.fulfillment_queue.arn
Condition = {
ArnEquals = {
"aws:SourceArn" = aws_sns_topic.order_events.arn
}
}
}]
})
}The filter_policy on the fulfillment subscription means only ORDER_PLACED and ORDER_UPDATED events reach the fulfillment queue. The analytics queue has no filter and receives every event type. SNS evaluates these filters before delivery, so your downstream queues and Lambda functions are not burdened with messages they do not care about.
raw_message_delivery = true strips the SNS envelope from the message before delivering to SQS. Without it, your consumer receives a JSON object with SNS metadata wrapping the actual payload. With raw delivery, your consumer receives the original message body directly, which simplifies your processing code.
Publishing Events
import boto3
import json
import os
sns = boto3.client("sns", region_name="us-east-1")
def publish_order_event(order_id: str, event_type: str, order_data: dict):
message = {
"orderId": order_id,
"eventType": event_type,
"timestamp": datetime.utcnow().isoformat(),
"data": order_data
}
sns.publish(
TopicArn=os.environ["ORDER_EVENTS_TOPIC_ARN"],
Message=json.dumps(message),
MessageAttributes={
"eventType": {
"DataType": "String",
"StringValue": event_type
}
}
)
def place_order(order_data: dict) -> dict:
order = create_order_in_database(order_data)
publish_order_event(
order_id = order["id"],
event_type = "ORDER_PLACED",
order_data = order
)
return orderDead Letter Queue Monitoring
A message landing in the dead letter queue means your processing logic failed consistently. This is always worth an alert. Do not let DLQs accumulate silently.
resource "aws_cloudwatch_metric_alarm" "order_dlq_messages" {
alarm_name = "order-processing-dlq-messages"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 1
metric_name = "ApproximateNumberOfMessagesVisible"
namespace = "AWS/SQS"
period = 60
statistic = "Sum"
threshold = 0
alarm_description = "Messages have arrived in the order processing dead letter queue"
treat_missing_data = "notBreaching"
alarm_actions = [aws_sns_topic.alerts.arn]
dimensions = {
QueueName = aws_sqs_queue.order_processing_dlq.name
}
}
resource "aws_cloudwatch_metric_alarm" "order_queue_depth" {
alarm_name = "order-queue-depth-high"
comparison_operator = "GreaterThanThreshold"
evaluation_periods = 3
metric_name = "ApproximateNumberOfMessagesVisible"
namespace = "AWS/SQS"
period = 300
statistic = "Average"
threshold = 1000
alarm_description = "Order processing queue depth exceeded 1000 messages"
alarm_actions = [aws_sns_topic.alerts.arn]
dimensions = {
QueueName = aws_sqs_queue.order_processing.name
}
}The DLQ alarm threshold is 0, meaning any message arriving in the DLQ triggers an alert. The main queue depth alarm at 1000 messages tells you when your consumers cannot keep up with producers, which could indicate a processing bottleneck or a consumer outage.
Closing Thoughts
SQS and SNS are the backbone of event-driven architecture on AWS. Used together in the SNS fan-out pattern, a single business event can trigger multiple independent downstream processes without any of them knowing about each other. Each service scales independently, fails independently, and deploys independently.
The configuration details that matter most are the visibility timeout, the dead letter queue, and the redrive policy. Get those right and your messaging infrastructure will handle failure and load gracefully. Ignore them and you will spend time debugging mysterious duplicate processing and messages that vanish without explanation.
Enjoy the cloud.
Osama
#AWS #SQS #SNS #EventDrivenArchitecture #CloudArchitecture #Microservices #ServerlessArchitecture #Terraform #InfrastructureAsCode #AWSLambda #CloudNative #AmazonWebServices #SolutionsArchitect #CloudComputing #BackendEngineering #SystemDesign #DevOps #TechBlog #MessageQueue #CloudInfrastructure
Technology Geek 
Leave a comment