AWS - SQS

Amazon SQS is a fully managed message queuing service that enables decoupling and scaling of microservices, distributed systems, and serverless applications. It acts as a buffer between components of a system that communicate asynchronously.

---

Why Use SQS?

Imagine you have a system with multiple services (e.g., a web app and a background worker). You don't want your web server to wait for long tasks (like image processing or email sending). With SQS:

• The web server sends a message to the queue.
• The worker processes messages from the queue asynchronously.
• If the worker fails, the message stays in the queue until it’s successfully processed (or moved to a dead-letter queue after repeated failures).

---

Core Concepts

Concept	Description
Queue	A buffer where messages are stored until they're processed.
Message	A unit of data sent between components (can include up to 256 KB of data).
Producer	The component that sends messages to the queue.
Consumer	The component that receives and processes messages.
Visibility Timeout	Time during which a message is invisible after being read (to avoid duplication).
Dead-Letter Queue	Stores messages that couldn't be processed successfully after retries.

---

Types of SQS Queues

1. Standard Queue
• Default type
• At-least-once delivery (possible duplicates)
• Best-effort ordering
• High throughput (nearly unlimited)
2. FIFO Queue (First-In-First-Out)
• Exactly-once processing (no duplicates)
• Preserved order
• Lower throughput compared to Standard (up to 3000 messages/sec with batching)

---

Key Features

• Fully managed: No infrastructure to manage
• Scalable: Handles millions of messages per second
• Secure: Supports IAM policies, encryption (SSE)
• Durable: Messages stored redundantly across multiple AZs
• Long polling: Reduces empty responses by waiting up to 20 seconds for messages

---

Common Use Cases

• Decoupling microservices
• Buffering requests (throttling)
• Asynchronous task processing (email, video transcoding)
• Distributed workloads
• Order processing systems (FIFO)

---

Sample Workflow

1. Producer sends a message to the queue:

   aws sqs send-message --queue-url <QUEUE_URL> --message-body "ProcessOrder123"
   

2. Consumer polls and processes:

   aws sqs receive-message --queue-url <QUEUE_URL> --max-number-of-messages 1
   

3. Deletes message after processing:

   aws sqs delete-message --queue-url <QUEUE_URL> --receipt-handle <HANDLE>
   

---

Security and Access Control

• Use IAM policies to restrict who can send/receive/delete messages.
• Enable server-side encryption with SSE-SQS or SSE-KMS.
• Use VPC endpoints for private access.

---

Integrations

• AWS Lambda: Trigger Lambda functions from SQS messages (FIFO supported).
• Amazon SNS: SNS can fan-out messages to SQS queues.
• Step Functions: Use SQS for wait and queue-based orchestration.
• EC2, ECS, EKS: Poll SQS from worker nodes.

#!/bin/bash

# ==============================

# CONFIGURATION

# ==============================

REGION="us-east-1"

QUEUE_NAME="my-cli-demo-queue"

PROFILE="default"

echo "Creating SQS queue..."

# ==============================

# CREATE QUEUE

# ==============================

QUEUE_URL=$(aws sqs create-queue \

  --queue-name $QUEUE_NAME \

  --region $REGION \

  --profile $PROFILE \

  --query 'QueueUrl' \

  --output text)

echo "Queue URL: $QUEUE_URL"

echo "-------------------------------------"

# ==============================

# GET QUEUE ARN

# ==============================

QUEUE_ARN=$(aws sqs get-queue-attributes \

  --queue-url $QUEUE_URL \

  --attribute-names QueueArn \

  --region $REGION \

  --profile $PROFILE \

  --query 'Attributes.QueueArn' \

  --output text)

echo "Queue ARN: $QUEUE_ARN"

echo "-------------------------------------"

# ==============================

# SEND MESSAGE

# ==============================

echo "Sending message..."

aws sqs send-message \

  --queue-url $QUEUE_URL \

  --message-body "Hello from SQS CLI bundle" \

  --region $REGION \

  --profile $PROFILE

echo "-------------------------------------"

# ==============================

# RECEIVE MESSAGE

# ==============================

echo "Receiving message..."

RECEIVE_OUTPUT=$(aws sqs receive-message \

  --queue-url $QUEUE_URL \

  --max-number-of-messages 1 \

  --wait-time-seconds 10 \

  --region $REGION \

  --profile $PROFILE)

echo "$RECEIVE_OUTPUT"

echo "-------------------------------------"

# ==============================

# EXTRACT RECEIPT HANDLE

# ==============================

RECEIPT_HANDLE=$(echo "$RECEIVE_OUTPUT" | jq -r '.Messages[0].ReceiptHandle')

if [ "$RECEIPT_HANDLE" != "null" ]; then

  echo "Deleting message..."

  aws sqs delete-message \

    --queue-url $QUEUE_URL \

    --receipt-handle "$RECEIPT_HANDLE" \

    --region $REGION \

    --profile $PROFILE

else

  echo "No message received."

fi

echo "-------------------------------------"

# ==============================

# LIST QUEUES

# ==============================

echo "Listing queues..."

aws sqs list-queues \

  --region $REGION \

  --profile $PROFILE

echo "-------------------------------------"

# ==============================

# DELETE QUEUE

# ==============================

echo "Deleting queue..."

aws sqs delete-queue \

  --queue-url $QUEUE_URL \

  --region $REGION \

  --profile $PROFILE

echo "Queue deleted."

echo "Done."