SQS

What is SQS?

Amazon SQS is a web service that gives you access to a message queue that can be used to store mesages while waiting for a computer to process them.

Amazon SQS is a distributed queue system that enables web service applicationst o quickly and reliably queue messages that one component in the application generates to be consumed by another component. A queue is a temporary repository for messages that are awaiting processing.

Meme Website

  1. We have a user that wants to upload a photo.
  2. Website will store photo in S3
  3. When it is finished uploading, trigger lambda to send all data about the image to SQS
  4. Send data as to what the top of meme should say, bottom of meme should say, data about S3 bucket location, data around user
  5. Sits in SQS as a message waiting to be processed
  6. Fleet of EC2 instances that pull from the SQS
  7. Find message, create meme & encode
  8. EC2 instance writes it to S3 bucket
  9. EC2 goes back to the SQS looking for more jobs

Can scale EC2 instances off of how many jobs in the queue.

Travel Website

  1. User looking for package holiday for best flights
  2. Query in browser
  3. Hits EC2 instance
  4. Take what the user is looking for, send as message to SQS queue
  5. EC2 pulls the messages from SQS looking for a job
  6. Get job, process, find airline flights and send back to web server

If we didn't have SQS and we passed the message to an application service, then app to airlines. If we lose the application service, we don't lose messages in queue. Instead it stays until the visibility timeout. When visibility timeout is up, it is placed in the queue again and lets another application service grab the job.

What is SQS pt. 2

Using Amazon SQS, you can decouple the components of an application so they run independently, easing message management between components.

Any component of a dstributed application can store messages in the queue. Messages can contain up to 256 KB of text in any format. Any component can later retrieve the messages programmatically using the Amazon SQS API

The queue acts as a buffer between the component producing and saving data, and the component receiving the data for processing. This means the queue resolves issues that arise if the producer is producing work faster than the consumer can process it, or if the producer or consumer are only intermittently connected to the network.

If we only have 2 EC2 instances pulling the SQS Queue and we suddenly get a large influx of messages, we can configure auto scaling groups to monitor the SQS queue. If it goes over a certain number of messages in the queue, start provisioning additional EC2 instaces.

If the number of messages drops below a certain point, we can scale down as well.

Queue Types

Two types of queue:

  • Standard queues (default)
  • FIFO Queues

Standard Queues

Amazon SQS offers standard as the default queue type. A standard queue lets you have a nearly-unlimited number of transactions per second. Standard queues guarantee that a message is delivered at least once. However, occasionally (because of the highly-distributed architecture that allows high throughput), more than one copy of a message might be delivered out of order. Standard queues provide best-effort ordering which ensures that messages are generally delivered in the same order as they are sent, but cannot be guaranteed

FIFO Queues

The FIFO queue complements the standard queue. The most important feature of this queue type are FIFO (first-in first-out) delivery and exactly-once processing: The order in which messages are sent and received is strictly preserved and a message is delivered once and remains available until a consumer processes and deletes it; duplicates are not introduced into the queue. FIFO queues also support message groups that allow multiple ordered message groups within a single queue. FIFO queues are limited to 300 transactions per second (TPS), but have all the capabilites of standard queues.

SQS Key Facts

  • SQS is pull-based, not pushed-based
  • Messages are 256 KB in size
  • Messages can be kept in the queue from 1 minute to 14 days
  • Default retention period is 4 days
  • SQS guarantees that your messages will be processed at least once

SQS Visibility Timeout

  • The Visibility Timeout is the amount of time that the message is invisible in the SQS queue after a reader picks up that message. Provided the job is processed before the visibility time out expires, the message will then be deleted from the queue. If the job is not processed within tha ttime, the message will become visible again and another reader will process it. This could result in the same message being delivered twice.
  • Default Visibility Timeout is 30 seconds
  • Increase it if your task takes > 30 seconds
  • Maximum is 12 hours
  • API call is ChangeMessageVisibility

Long Polling

  • Amazon SQS Long Polling is a way to retrieve messages from your Amazon SQS queues.
  • While the regular short polling returns immediately (even if the message queue being polled is empty), long polling doesn't return a response until a message arrives in the message queue, or the long poll times out.
  • As such, long polling can save you money because you're not constantly polling an empty queue. Instead of the EC2 instance constantly trying to get messages, the long poll will return saying its empty.
  • When ReceiveMessageWaitTimeSeconds is set to a value greater than zero, long poling is enabled.
    • Long polling allows Amazon SQS to wait until a message is available in the queue before sending a response.
    • Short polling continuously pools a queue and can have false positives.
    • Enabling long polling reduces the number of poll requests, false positives, and empty responses.

Exam Tips

  • SQS is a distributed message queueing system
  • Allows you to decouple the components of an application so that they are independent
  • Pull-based, not push-based
  • Standard queues (default)
    • Best effort ordering; message delivered atleast once
  • FIFO Queues (First in First Out)
    • Ordering strictly preserved, message delivered once, no duplicates.
      • e.g. good for banking transactions which need to happen in strict order
    • FIFO SQS Queues will end in the .fifo suffix
  • Visibility Timeout
    • Default is 30 seconds; increase this if your task takes > 30 seconds to complete
    • Max is 12 hours
    • ChangeMessageVisibility
  • Short polling
    • Returned immediately even if no messages are in the queue
  • Long polling - polls the queue periodically and only returns a response when a message is in the queue or the timeout is reached
    • maximum 20 second timeout

Delay Queues

Delay Queues let you postpone the delivery of new messages to a queue for a number of seconds. If you create a delay queue, any messages that you send to the queue remain invisible to consumers for the duration of the delay period. The default (minimum) delay for a queue is 0 seconds. The maximum is 15 minutes.

Delay queues are similar to visibility timeouts becaouse both features make messages unavailable to consumers for a specific period of time. The difference between the two is that, for delay queues, a message is hidden when it is first added to the queue, whereas for visibility timeouts a emssage is hidden only after it is consumed from the queue.

To set delay seconds on individual messages rather than on an entire queue, use message timeres to allow Amazon SQS to use the message timer's DelaySeconds value instead of the delay queue's DelaySeconds value.

Extras

  • "Use the ChangeMessageVisibility API at the Consumer Level": Here, the question is saying that it takes an unpredictably long time to process messages. Therefore, the consumer must, in case of long processing happening, use the ChangeMessageVisibility API to extend their message timeout and process the message until the end while making sure other consumers won't see that message.
    • Since we can't predict the processing time, the queue can't be set up for a longer visibility timeout. We have to manage it in the consumer just on a message by message bassis.
  • Amazon SQS supports dead-letter queues, which other queues (source queues) can target for messages that cannot be processed (consumed) successfully. Dead-letter queues are useful for debugging your application or messaging system because they let you isolate problematic messages to determine why their processing doesn't succeed.
  • Dead Letter Queues
    • Used for errors processing items in the queue
    • use this to separate them to see what went wrong
    • Things that will put a message into a DLQ after being processed by Lambda
      • Invocation Failed
      • Invocation was asynchronous
        • Any Lambda function invoked asynchronously is retried twice before the event is discarded
        • In addition, you need to add permissions to the execution role of your Lambda function for access to the SQS.
  • To delete messages in the Queue, use PurgeQueue
    • BatchDelete does not exist
    • https://docs.aws.amazon.com/cli/latest/reference/sqs/purge-queue.html
  • ReceiveMessage API call gets the message
  • ChangeMessageVisibility API call changes the visibility timeout
  • ReceiveMessageWaitTimeSeconds set to a value greater than zero means long polling is enabled
  • The minimum visibility timeout for an SQS message is 0 seconds.
    • There is no way to decrease that value.
  • an SQS request can contain up to 10 individual messages
    • Total size of the request cannot exceed 256KB
  • FIFO SQS Queues will end in the .fifo suffix
  • SQS Messages are billed to your AWS account in increments of 64KB
    • Messages can be up to 256KB
    • Would be billed as four units of 64KB each
  • GetQueueAttributes that can always be gotten:
    • ReceiveMessageWaitTimeSeconds
    • DelaySeconds
    • VisibilityTimeout
      • For FIFO Queues, you can get ContentBasedDeduplication
  • A message is considered to be in flight after it's received from a queue by a consumer, but not yet deleted from the queue.
  • In addition to XML, JSON, and unformatted text, the following Unicode characters are allowed:
  • "#x9" | #xA | #xD | #x20 to #xD7FF | #xE000 to #xFFFD | #x10000 to #x10FFFF
  • Set the parameter to use to order by using the MessageGroupId parameter
    • The message group ID is the tag that specifies that a message belongs to a specific message group
    • Messages that belong to the same message group are always processed one by one, in a strict order relative to the message group
      • However, msesages that belong to different message groups might be processed out of order
  • Message Deduplication ID
    • In FIFO queues, this token is used for deduplication of sent messages
    • If a message with a particular message deduplication ID is sent successfully, any messages sent with the same message dedplication ID is sent successfully
    • Any messages sent with the same message dedplucation ID are accepted successfully but aren't delivered during the 5-minute deduplication interval

CHANGEMESSAGEVISABILITY IS THE API CALL TO CHANGE THE MESSAGE'S VISIBILITY TIMEOUT