1 of 74

Module 10: Developing with Messaging Services

AWS Academy Cloud Developing

2 of 74

Section 1: Introduction

Module 10: Developing with Messaging Services

3 of 74

Module objectives

At the end of this module, you should be able to do the following:

Illustrate how messaging services, including queues, pub/sub messaging, and streams, support asynchronous processing
Describe Amazon Simple Queue Service (Amazon SQS)
Send messages to an SQS queue
Describe Amazon Simple Notification Service (Amazon SNS)
Subscribe an SQS queue to an SNS topic
Describe how Amazon Kinesis can be used for real-time analytics

3

4 of 74

Module overview

Sections

Introduction
Processing requests asynchronously
Introducing Amazon SQS
Working with Amazon SQS messages
Configuring Amazon SQS queues
Introducing Amazon SNS
Developing with Amazon SNS
Introducing Kinesis Data Streams

Demonstration

Working with Amazon Messaging Services

Lab

Implementing a Messaging System Using Amazon SNS and Amazon SQS

4

Knowledge check

5 of 74

Café business requirement

Currently, updating the coffee inventory on the website is a manual process. Sofía would like to find an easier way to keep the inventory updated. She considers setting up a messaging system to automatically receive and process inventory updates.

5

6 of 74

Messaging services as part of developing a cloud application

6

7 of 74

Section 2: Processing requests asynchronously

Module 10: Developing with Messaging Services

8 of 74

Synchronous processing

8

Tightly coupled

Client

Service A

Service B

Response/result

Request

In a synchronous model, the client (producer) generates a request message to the consumer. Then, the client waits until the message is processed and the client gets a response from the consumer. This is a relatively simple interaction common to create, read, update, and delete (CRUD) API actions.

In the example of the coffee shop, this is similar to ordering a coffee at the counter and getting the cup of coffee immediately before you walk away. This action is complete before the next person orders.

In the example diagrammed on the slide, the client makes a request to service A. Service A then calls service B. Service A waits for service B to complete its actions and respond to service A. That response then goes back to the client.

In this model, strong interdependency exists between the producer and the consumer. This interdependency creates a tightly coupled system. The disadvantage of a tightly coupled system is that it is not fault tolerant. This means that if any component of the system fails, then the entire system will fail. If the consumer fails while processing a message, then the producer will be forced to wait until that message gets processed (assuming that the message is not lost). Additionally, if new consumer instances are launched to recover from a failure or to keep up with an increased workload, the producer must be explicitly made aware of the new consumer instances. In this scenario, the producer is tightly coupled with the consumer, and the coupling is prone to brittleness.

9 of 74

Asynchronous processing

9

Client

Service A

Service B

Acknowledgement

Request

The result might be returned to the client through another process

In contrast to a synchronous model is asynchronous processing. In this design, the client (producer) sends a request and might get an acknowledgement that the event was received. However, the client doesn’t receive a response that includes the results of the request.

In the coffee shop example, this is similar to placing an order that is delivered later. You get your order number, and then the next person in line can order without waiting for your order to be fulfilled. Someone brings you the order when it is ready. A problem with completing your order does not prevent the person behind you in line from ordering.

In the example diagrammed on the slide, when the client makes a request to service A, service A surfaces the event to its consumers (in this example, service B) and moves on.

The disadvantage is that service B does not have a channel to pass back results to service A. You need to write your applications to get the results and deal with errors asynchronously. Although you might be used to writing code that has explicit coupling, the coupling is not actually needed for the type of processing you are doing.

The advantage of an asynchronous approach is that you reduce the dependencies on downstream activities, and this reduction improves responsiveness back to the client. This means that you don’t need to put logic into your code to deal with long wait times or handle errors that might occur downstream. After your client has successfully handed off the request, you can move on.

The producer does not need to wait for the first message to be read. Instead, the producer can continue to generate messages whether the consumer is processing them or not. The producer is not impacted if the consumer fails to process the message. This means much less interdependency between the producer and the consumer, which makes the system more loosely coupled with greater fault tolerance.

10 of 74

Asynchronous processing�with messaging services

Message queues

Pub/sub messaging

Data streams

10

Amazon SQS

Amazon SNS

Amazon Kinesis�Data Streams

In this module, you'll learn about three types of messaging services that you can use for asynchronous processing:

Message queues
Publish/subscribe (pub/sub) messaging
Data streams

The type of messaging service that you choose depends on your use case. The next few slides provide an overview of these three types of messaging services. In each type, producers send requests (messages) to a service and do not wait for a result. The messaging service provides an interim location from which consumers get messages. This provides an asynchronous, decoupled design between your components.

You'll learn about three fully managed services that you can use to implement each type of messaging service: Amazon Simple Queue Service (Amazon SQS) for message queues, Amazon Simple Notification Service (Amazon SNS) for pub/sub messaging, and Amazon Kinesis Data Streams for data streams.

11 of 74

Message queues overview

11

Queue

Producers

Consumer

Poll for messages

Add messages to the queue

Process each retrieved message

Delete the processed message from the queue

1

2

3

4

12 of 74

Pub/sub messaging overview

12

Topic

Producers

Subscriber A

Push messages to subscribers

Subscriber B

Subscriber C

Add messages to a topic

Process messages of interest

1

2

3

13 of 74

Data streams overview

13

Stream

Producers

Consumer

Poll for messages

Add messages to the stream

Aggregate and analyze large numbers of messages

1

2

3

Data streams are similar to queues in that they provide a temporary repository for producer messages, and consumers poll the stream to check for new messages.

Streams are different from queues in that the rate of messages is typically much higher. Rather than processing each individual message on the queue, streams are typically used to analyze high volumes of data in near-real time.

Another distinction from queues is that multiple consumers might process the same messages and do entirely different actions with them. Consumers all view the same data (similar to looking through the same window) and act on the messages for their own purposes. Consumers do not delete messages from a stream.

Streams are helpful for use cases where you need to handle rapid and continuous data intake and aggregation. For example, you might use a stream to process IT infrastructure log data, social media data, market data feeds, or web clickstream data.

14 of 74

Section 2 key takeaways

An asynchronous design reduces interdependencies and can improve responsiveness to the client.
Messaging services are good for creating asynchronous designs and include queues, pub/sub messaging, and streams.
AWS provides fully managed messaging services including Amazon SQS, Amazon SNS, and Kinesis Data Streams.

14

15 of 74

Section 3: Introducing Amazon SQS

Module 10: Developing with Messaging Services

16 of 74

Amazon SQS

Fully managed message queuing service that eliminates the complexity and overhead associated with managing and operating message queues

Limits administrative overhead
Reliably delivers messages at very high volume and throughput
Scales dynamically
Can keep data secure
Offers standard and First-In-First-Out (FIFO) options

16

Amazon Simple Queue Service (Amazon SQS)

Amazon Simple Queue Service (Amazon SQS) is a fully managed message queuing service. With Amazon SQS, you can decouple and scale microservices, distributed systems, and serverless applications. The service eliminates the complexity and overhead associated with managing and operating message-oriented services, so that you can focus on the business logic of your application.

You can use Amazon SQS to transmit any volume of data at any level of throughput, without losing messages or requiring other services to be available. With Amazon SQS, you can decouple application components so that they run and fail independently, which increases the overall fault tolerance of the system. Multiple copies of every message are stored redundantly across multiple Availability Zones so that they are available when they are needed.

Amazon SQS uses the AWS Cloud to dynamically scale based on demand. Amazon SQS scales elastically with your application so that you don’t have to worry about capacity planning and pre-provisioning. The number of messages per queue is not limited.

You can use Amazon SQS to exchange sensitive data between applications using server-side encryption (SSE) to encrypt each message body. Amazon SQS SSE integration with AWS Key Management Service (AWS KMS) provides the ability to centrally manage the keys that protect Amazon SQS messages along with keys that protect your other AWS resources. AWS KMS logs every use of your encryption keys to AWS CloudTrail to help meet your regulatory and compliance needs.

Amazon SQS offers both standard and First-In-First-Out (FIFO) queues to support different processing priorities.

17 of 74

Standard and FIFO queue types

17

Standard (default)	FIFO
Best effort ordering, at-least-once delivery. Nearly unlimited throughput.	Message order is preserved, exactly-once delivery. More limited throughput.
Use cases	Use cases
Let users upload media while resizing or encoding it	Ensure that a deposit is recorded before a bank withdrawal happens
Process a high number of credit card validation requests	Ensure that a credit card transaction is not processed more than once
Schedule multiple entries to be added to a database that are not order dependent	Prevent a student from enrolling in a course before they register for an account

Amazon SQS supports two types of message queues:

Standard queues are the default queue type. Standard queues provide best-effort ordering, which ensures that messages are generally delivered in the same order that they are sent. Standard queues support at-least-once message delivery. However, occasionally more than one copy of a message might be delivered out of order. Standard queues also support a nearly unlimited number of transactions per second (TPS) per action.

First-In-First-Out (FIFO) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated. FIFO queues also provide exactly-once processing, but they have a limited number of TPS. The next section provides more information about queue type distinctions.

You can use standard message queues in many scenarios, as long as your application can process messages that arrive more than once and out of order. For example, you can use a standard queue for the following:

Decouple live user requests from intensive background work: Enable users to upload media while resizing or encoding it.
Allocate tasks to multiple worker nodes: Process a high number of credit card validation requests.
Batch messages for future processing: Schedule multiple entries to be added to a database that are not order dependent and for which idempotency is not an issue.

You use FIFO queues for applications when the order of operations and events is critical, or in cases where duplicates can't be tolerated. For example, you can use a FIFO queue for the following:

Bank transactions: Ensure that a deposit is recorded before a bank withdrawal happens.
Credit card transactions: Ensure that a credit card transaction is not processed more than once.
Course enrollment: Prevent a student from enrolling in a course before they register for an account.

18 of 74

Amazon SQS architecture example

18

AWS Cloud

Auto Scaling group

EC2 instances

User

S3 bucket

SQS queue

Lambda function

Sends message about image to SQS queue

Processes image and sends back to S3 bucket

Polls SQS queue

Invokes Lambda function

Uploads photo to S3 bucket

You can integrate Amazon SQS with other AWS services to make applications more reliable and scalable.

This slide illustrates an example of using an SQS queue to decouple components in an image processing application.

In this example, a user uploads a photo to an Amazon Simple Storage Service (Amazon S3) bucket, which invokes an AWS Lambda function. The Lambda function sends a message containing information about the image to an SQS queue. A processing server that is part of an Amazon EC2 Auto Scaling group polls the SQS queue for messages to process. After processing the photo, the processing server sends the processed photo back to the S3 bucket.

Note that the Auto Scaling group scales based on the size of the SQS queue. The processing servers keep working until the queue is empty and then keep holding until messages appear. If the EC2 instances become unavailable, the Lambda function can keep putting messages on the queue when an upload is made. When the Auto Scaling group is ready to start consuming messages again, it polls the queue for messages.

19 of 74

Section 4: Working with Amazon SQS messages

Module 10: Developing with Messaging Services

20 of 74

Amazon SQS message lifecycle

20

Producer 1

Consumer 1

A

SQS queue �(distributed on SQS servers)

A

SendMessage�Producer delivers message(s) to a queue

Amazon SQS redundantly stores the message across multiple Amazon SQS servers.

ReceiveMessage�Consumer polls the queue and retrieves messages

Amazon SQS sets the visibility timeout on the retrieved message(s) in the queue

DeleteMessage�After processing the message successfully, the consumer deletes it from the queue

1

2

3

Producer 2

B

A

B

Consumer 2

The lifecycle of an Amazon SQS message is as follows:

A producer component sends a message to the SQS queue. Amazon SQS redundantly stores the message across multiple Amazon SQS servers.
A consumer component retrieves the message from the queue, and Amazon SQS starts the visibility timeout period. During the visibility timeout period, no other consumers can pick the message up from the queue.
The consumer component processes the message and then deletes it from the queue during the visibility timeout period. Note that if the message is not processed and deleted from the queue before the visibility timeout period expires, another consumer might pick up and process the same message.

You can perform these actions with the SendMessage, ReceiveMessage, and DeleteMessage API calls, respectively, either programmatically or from the Amazon SQS console. A variety of configuration options affect cost, message processing, and message retention.

21 of 74

The SendMessage operation

21

Producer

aws sqs send-message �--queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue �--message-body "My first message"

{ �"MD5OfMessageBody": "51b0a325...39163aa0", �"MessageId": "d6790f8d-d575-4f01-bc51-40122EXAMPLE" �}

SQS queue

Example Amazon SQS response with MD5 hash of body and MessageID

Example AWS CLI command for SendMessage operation

The SendMessage operation delivers a message to a specific queue. This operation takes the following required request parameters:

MessageBody (required): The message to send. The maximum string size is 256 KB.
QueueUrl (required): The URL of the Amazon SQS queue where a message is sent.

Optionally you might also include the following:

MessageAttribute (optional): You can include structured metadata (such as timestamps, geospatial data, signatures, and identifiers) by using message attributes. Each message can have up to 10 attributes. Message attributes are separate from the message body; however, they are sent alongside it. Each message attribute consists of a name, type, and value.
DelaySeconds (optional): The length of time, in seconds, to delay a specific message.

The slide illustrates how you might use the AWS Command Line Interface (AWS CLI) to call the SendMessage operation.

After you send a message, the Amazon SQS response includes a Message-Digest algorithm 5 (MD5) digest of the message body string, which you can use to verify that Amazon SQS received the message correctly. The response also includes a system-assigned MessageId, which is useful for identifying messages. Additional elements might be included in the response depending on the message parameters and the type of queue.

22 of 74

The ReceiveMessage operation

22

aws sqs receive-message

--queue-url https://sqs.us-east1.amazonaws.com/80398EXAMPLE/MyQueue

--max-number-of-messages 10

"Messages": [

{

"Body": "My first message.",

"ReceiptHandle": "AQEBzbVv...fqNzFw==",

"MD5OfBody": "51b0a325...39163aa0",

"MessageId": "d6790f8d-d575-4f01-bc51-40122EXAMPLE",

"Attributes": {

…

},

}

]

}

Consumer

Example AWS CLI command for ReceiveMessage operation

Example Amazon SQS response with messages. Each message has a ReceiptHandle.

SQS queue

23 of 74

ReceiveMessage and visibility timeout

23

SQS queue

A

Time

ReceiveMessage�Consumer 1

Message A �returned

Message A not�returned

ReceiveMessage�Consumer 2

ReceiveMessage�Consumer 3

ReceiveMessage�Consumer 4

Visibility timeout (default = 30 seconds)

A

Consumer 1 should process and delete message A from the queue before it becomes visible again

Message A not�returned

A

Immediately after a message is received using the ReceiveMessage operation, Amazon SQS sets a visibility timeout. The visibility timeout is a period during which other consumers cannot receive and process the message.

Before the visibility timeout expires, the consumer is expected to process AND then delete the message from the queue. Amazon SQS doesn't automatically delete the message because, as a distributed system, there's no guarantee that the consumer actually received and handled the message (for example, because of a connectivity issue or because of an issue in the consumer application).

For the duration of the visibility timeout, Amazon SQS temporarily stops returning the message as part of any ReceiveMessage requests.

The visibility timeout defaults to 30 seconds. The minimum is 0 seconds, and the maximum is 12 hours. You can use the VisibilityTimeout parameter on the ReceiveMessage operation to modify the visibility timeout for a ReceiveMessage request.

If your consumer doesn't delete the message before the visibility timeout expires, the message becomes visible to other consumers, and Amazon SQS flags the message as being returned to the queue. Amazon SQS increments its counter each time the message becomes visible on the queue. By default, Amazon SQS will increment the counter and make the message available to consumers until the retention period for that message expires. You can modify this behavior using a dead-letter queue with a redrive policy.

24 of 74

Configuring a dead-letter queue

24

Source SQS queue

A

Time

ReceiveMessage�

Visibility timeout

A

1

2

A

Visibility timeout

A

Consumer fails to complete processing

A

Consumer fails to complete processing

Dead-letter queue

A

3

Producer 1

A

maxReceiveCount= 2

For messages that cannot be processed (and deleted by the consumer), configure a dead-letter queue in Amazon SQS. Dead-letter queues can help you troubleshoot incorrect transmission operations and stop your queue from continually trying to process a corrupted message.

The dead-letter queue can receive messages from the source queue after the maximum number of processing attempts (maxReceiveCount) has been reached. The maxReceiveCount is set as part of the source queue's redrive policy. Messages can be sent to and received from a dead-letter queue just like any other SQS queue. You can create a dead-letter queue from the Amazon SQS API or the Amazon SQS console.

To configure the dead-letter queue from the console, first create a queue to be used as your dead-letter queue. Then, on the source queue, choose to configure a redrive policy. As part of the redrive policy, select the queue to use as the dead-letter queue, and set the maximum receives option to tell Amazon SQS how many processing attempts to make before sending the message to the dead-letter queue.

In the example, on the slide the source queue is set with a redrive policy with a maxReceiveCount equal to two. Because of the policy, message A is moved to the dead-letter queue after the second consumer fails to process the message. Rather than making the message visible to consumers for a third time, Amazon SQS moves message A to the dead-letter queue, and the message is no longer available in the source queue.

For more information, see the following resources in the Amazon SQS Developer Guide:

Amazon SQS Dead-Letter Queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
Configuring a Dead-Letter Queue (Console): https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-configure-dead-letter-queue.html

25 of 74

ReceiveMessage and short polling

25

SQS queue

(distributed on SQS servers)

A

D

E

A

C

D

B

Messages

received from

sampled servers

ReceiveMessage request with WaitTimeSeconds = 0

B

C

E

C

D

A

C

A

E

B

C

D

A

B

Amazon SQS samples the servers

Consumer 1

Consumer 2

As noted earlier, consumers use a polling model to retrieve messages from an SQS queue. By default, when you make a ReceiveMessage API call, Amazon SQS performs short polling, in which it samples a subset of SQS servers (based on a weighted random distribution).

Amazon SQS immediately returns only the messages that are on the sampled servers. In the example, short polling returns messages A, B, C, and D, but it does not return message E. Short polling occurs when the WaitTimeSeconds parameter of a ReceiveMessage call is set to 0.

If the number of messages in the queue is small (fewer than 1,000), you will most likely get fewer messages than you requested per ReceiveMessage call. If the number of messages in the queue is extremely small, you might not receive any messages from a particular ReceiveMessage call. If you keep requesting ReceiveMessage, Amazon SQS will sample all of the servers and you will eventually receive all of your messages.

26 of 74

ReceiveMessage and long polling

26

SQS queue

(distributed on SQS servers)

A

D

E

A

C

D

B

Messages

Received from all servers

ReceiveMessage request with WaitTimeSeconds > 0

B

C

E

C

D

A

C

D

A

B

Amazon SQS queries all servers

E

A

E

B

C

D

Consumer 1

Consumer 2

27 of 74

ReceiveMessage long polling example

27

https://sqs.us-east-1.amazonaws.com/123456789012/testQueue/

?Action=ReceiveMessage

&WaitTimeSeconds=10

&MaxNumberOfMessages=5

&VisibilityTimeout=15

&AttributeName=All;

&Expires=2019-04-18T22%3A52%3A43PST

&Version=2012-11-05

&AUTPARAMS

Retrieve messages from "testQueue" at this URL

Use long polling with a wait time of 10 seconds

Retrieve up to 5 messages at a time

Make these messages invisible to other consumers for 15 seconds

28 of 74

DeleteMessage operation

28

aws sqs delete-message

--queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue

--receipt-handle AQEBRXTo...q2doVA==

Consumer

Example AWS CLI command for DeleteMessage operation

SQS queue

Amazon SQS automatically deletes messages that remain in queue beyond the queue's retention period.

Amazon SQS deletes the message with the specified receipt handle. If you use the DeleteMessageBatch option, Amazon SQS responds with a list of successes and failures.

AQEBRXTo...q2doVA==

To prevent a message from being received and processed again when the visibility timeout expires, the consumer must delete the message. You can use the DeleteMessage operation to delete a specific message from a specific queue, or you can use DeleteMessageBatch to delete up to 10 messages. To select the message to delete, use the receipt handle of the message. To use the DeleteMessageBatch operation, provide the list of receipt handles to be deleted.

The ReceiptHandle is associated with a specific instance of receiving a message. If you receive a message more than once, the ReceiptHandle is different each time that you receive a message. When you use the DeleteMessage action, you must provide the most recently received ReceiptHandle for the message.

Amazon SQS automatically deletes messages that have been in a queue longer than the queue’s configured message retention period. The default message retention period is 4 days. However, you can set the message retention period to a value from 60 seconds to 1,209,600 seconds (14 days) by using the SetQueueAttributes action.

29 of 74

Amazon SQS message lifecycle operations summary

29

Producer 1

Consumer 1

A

SQS queue �(distributed on SQS servers)

A

aws sqs send-message �--queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue �--message-body "My first message"

aws sqs receive-message

--queue-url https://sqs.us-east1.amazonaws.com/80398EXAMPLE/MyQueue

--max-number-of-messages 1

1

2

3

Producer 2

B

A

B

Consumer 2

aws sqs delete-message

--queue-url https://sqs.us-east-1.amazonaws.com/80398EXAMPLE/MyQueue

--receipt-handle AQEBRXTo...q2doVA==

The lifecycle of an Amazon SQS message is as follows:

A producer component sends a message to the SQS queue. Amazon SQS redundantly stores the message across multiple Amazon SQS servers.
A consumer component retrieves the message from the queue, and Amazon SQS starts the visibility timeout period. During the visibility timeout period, no other consumers can pick the message up from the queue.
The consumer component processes the message and then deletes it from the queue during the visibility timeout period. Note that if the message is not processed and deleted from the queue before the visibility timeout period expires, another consumer might pick up and process the same message.

You can perform these actions with the SendMessage, ReceiveMessage, and DeleteMessage API calls, respectively, either programmatically or from the Amazon SQS console. A variety of configuration options affect cost, message processing, and message retention.

30 of 74

Section 4 key takeaways

Developers use the SendMessage, ReceiveMessage, and DeleteMessage API operations to add, retrieve, and delete queue messages.
Amazon SQS makes a retrieved message invisible on the queue for the duration of the visibility timeout.
Amazon SQS can send failed records to a dead-letter queue for separate processing.
Short polling samples Amazon SQS servers for messages, while long polling queries all servers.

30

31 of 74

Section 5: Configuring Amazon SQS queues

Module 10: Developing with Messaging Services

32 of 74

Standard SQS queues

32

Duplicates are possible

Messaging order is not guaranteed

Nearly unlimited throughput

Producer 1

Consumer 1

A

SQS queue �(distributed on SQS servers)

A

B

Consumer 2

B

C

A

C

B

C

App database

Standard queues are the default queue type. Standard queues support a nearly unlimited number of transactions per second (TPS) per action.

Because of the queue's highly distributed architecture (which allows this high throughput), you need to account for the following two potential conditions related to message delivery to your consumer(s).

Messaging order is not guaranteed. Standard queues provide best-effort ordering. This ensures that messages are generally delivered in the same order as they are sent, but some might be delivered out of order. If you use a standard queue and have a target application that needs to process messages in the order received, you need to include logic in your consumer application to reorder messages before further processing; for example, using the timestamp of each message to compare and order them.�
More than one copy of a message might be delivered. Standard queues support at-least-once delivery, not exactly-once delivery. To avoid processing the same message more than once, your application code needs to check for duplicates and only process a message the first time it is received. For example, you might add code that checks your database for a unique value in the incoming event and ignores it if it already exists in the database. MessageID is a unique identifier that Amazon SQS generates. If you want to verify the uniqueness of the record on the SQS queue, you could use the messageID attribute. Another scenario might be that you actually need to verify the uniqueness of the payload. For example, if an upstream producer sent the same record to Amazon SQS twice, each event would have a unique messageID in Amazon SQS but would be the same payload from your application’s perspective. You can avoid processing the same payload twice using the md5OfBody attribute in the Amazon SQS event. This field represents a unique hash of the payload.

33 of 74

FIFO SQS queues

33

Messaging order is preserved

Producer 1

Consumer 1

A

SQS queue �(distributed on SQS servers)

A

B

Consumer 2

B

C

A

B

C

App database

More limited throughput

Messages are delivered exactly once

34 of 74

Creating and configuring queues

34

API operation	Description
CreateQueue	Creates an SQS queue including queue attributes. If you don't provide a value for an attribute, the queue is created with the default value for the attribute.
GetQueueAttributes	Retrieves attributes of a queue including those that help you estimate the resources required to process queue messages; for example, the approximate number of messages available to retrieve and how many are currently invisible.
SetQueueAttributes	Sets attributes on an existing queue.
GetQueueUrl	Retrieves the URL of an SQS queue.
ListQueues	Retrieves a list of your queues.
DeleteQueue	Deletes a queue, regardless of whether the queue is empty. When a queue is deleted, any messages in the queue are no longer available.

You can use the AWS Management Console to create and configure your SQS queues, and you might find it helpful to use the console to send test messages to your queue.

You can also create and configure queues using API operations, and set queue parameters similar to some of the message parameters that you just learned about. For example, you can set the WaitTimeSeconds on the queue itself rather than an individual ReceiveMessage request.

The following are the basic queue options:

CreateQueue: Create an SQS queue including queue attributes.
SetQueueAttributes and GetQueueAttributes: Set and retrieve the attributes of a queue, respectively. These options help you estimate the resources required to process Amazon SQS messages.
GetQueueUrl: Retrieve the URL of an SQS queue.
ListQueues: Retrieve a list of your queues.
DeleteQueue: Deletes a queue, regardless of whether the queue is empty. When a queue is deleted, any messages in the queue are no longer available.

For more information, see the Amazon SQS API Reference at https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/Welcome.html.

35 of 74

Create SQS queue example

35

aws sqs create-queue --queue-name MyQueue --attributes file://create-queue.json

Create a queue using the attributes in create-queue.json

create-queue.json

{

"RedrivePolicy": "{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:80398EXAMPLE:MyDeadLetterQueue\",\"maxReceiveCount\":\"1000\"}",

"MessageRetentionPeriod": "259200"

}

Set the dead-letter queue destination

Set the message retention period to 3 days

Set the maximum receive count to 1,000

36 of 74

Three types of SQS queue security

AWS Identity and Access Management (IAM) and Amazon SQS policies

Server-side encryption (SSE) with AWS Key Management Service (AWS KMS)

Amazon Virtual Private Cloud (Amazon VPC) endpoint

36

Data encryption

Internetwork traffic privacy

Identity and access management

There are three ways to secure your Amazon SQS resources:

For identity and access management: Access to Amazon SQS requires credentials that AWS can use to authenticate your requests. These credentials must have permissions to access AWS resources, such as SQS queues and messages. Use AWS Identity and Access Management (IAM) policies and Amazon SQS policies. Amazon SQS has its own resource-based permissions system that uses policies that are written in the same language that is used for IAM policies. Thus, you can achieve similar results with Amazon SQS policies and IAM policies, and you might use them in combination. An example is presented on the next slide. The main difference is that you MUST use SQS policies to grant permissions to other AWS accounts.��For more information about using IAM and Amazon SQS policies, see the following sections of the Amazon SQS Developer Guide:

Overview of Managing Access in Amazon SQS: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-overview-of-managing-access.html
Using Custom Policies with the Amazon SQS Access Policy Language: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-creating-custom-policies.html

For data encryption: Encrypt your data using server-side encryption (SSE). With SSE, you can transmit sensitive data in encrypted queues. SSE protects the contents of messages in SQS queues by using keys that are managed by AWS KMS. When you enable SSE on an SQS queue, Amazon SQS will encrypt an incoming message before storing it on the queue and unencrypt it upon delivery to the consumer. Note that the encryption occurs at a point in time and applies to messages arriving after that point. For example, if messages 1–10 are on the queue when you enable SSE, message 11 will be encrypted, but messages 1–10 will not be.��For more information, see the Encryption at Rest section of the Amazon SQS Developer Guide at https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-server-side-encryption.html.

For internetwork traffic privacy: If you use Amazon Virtual Private Cloud (Amazon VPC ) to host your AWS resources, you can establish a connection between your VPC and Amazon SQS. You can use this connection to send messages to your SQS queues without crossing the public internet.��For more information, see the Internetwork Traffic Privacy section of the Amazon SQS Developer Guide at https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-internetwork-traffic-privacy.html.

For more information about Amazon SQS security, see the Amazon SQS Developer Guide at https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-security.html.

37 of 74

Examples of IAM and Amazon SQS policies

IAM policy attached to IAM user Bob

Allow

Actions:

ReceiveMessage, SendMessage

Resource:

arn:aws:sqs:us-east-1:80398EXAMPLE:MyQueue

Amazon SQS policy attached to MyQueue

Allow who:

Bob

Actions:

ReceiveMessage, SendMessage

Resource:

arn:aws:sqs:us-east-1:80398EXAMPLE:MyQueue

37

Equivalent permissions to MyQueue

38 of 74

Section 5 key takeaways

Standard queues provide nearly unlimited throughput but don't preserve message order and might include duplicate messages.
FIFO queues preserve order and provide exactly-once delivery but have more limited throughput.
Queue security includes managing access to queue resources, protecting data stored in the queue, and limiting exposure of messages to the internet.

38

39 of 74

Section 6: Introducing Amazon SNS

Module 10: Developing with Messaging Services

40 of 74

Amazon SNS

Fully managed messaging service with pub/sub functionality for many-to-many messaging between distributed systems, microservices, and event-driven applications

Offloads message filtering logic from your subscriber systems and message routing logic from your publisher systems
Reliably delivers messages and provides failure management features (retries, dead-letter queues)
Scales dynamically
Offers a FIFO topics option, which works with FIFO SQS queues to preserve message order

40

Amazon Simple Notification Service (Amazon SNS)

41 of 74

Amazon SNS overview

41

SNS topic

Publisher

Subscribers

Endpoint types

Application-to-application messaging (A2A)

Lambda functions

SQS queues

Kinesis Data Firehose

HTTP(S)

Application-to-person messaging (A2P)

Email

Text messaging (SMS)

Mobile push notifications

Publisher

A topic acts as the logical access point

All messages are pushed to all subscriber endpoints

Publishers send messages�to a topic

An SNS topic is a logical access point, which acts as a communication channel. Instead of including a specific destination address in each message, a publisher sends a message to the topic. Amazon SNS matches the topic to a list of subscribers for that topic. The service then delivers the message to each subscriber.

Each topic has a unique name, which identifies the Amazon SNS endpoint for publishers to post messages and subscribers to register for notifications.

A publisher can send messages to topics that they have created or to topics they have permissions to publish to. When you create an SNS topic, you can control access to it by defining policies that determine which publishers and subscribers can communicate with the topic.

To receive messages that are published to a topic, you must subscribe an endpoint to the topic. When you subscribe an endpoint to a topic, the endpoint begins to receive messages published to the associated topic. Subscribers receive all messages that are published to the topics that they subscribe to, and all subscribers to a topic receive the same messages. Amazon SNS formats the message based on each endpoint type.

Amazon SNS supports application-to-application messaging (A2A) with Lambda functions, SQS queues, Amazon Kinesis Data Firehose delivery streams, and HTTP(S). Amazon SNS supports application-to-person messaging (A2P) using email, text messaging (also known as short message service or SMS), and mobile push notifications.

�For more information about using Amazon SNS for application-to-application (A2A) messaging, see the Amazon SNS Developer Guide at https://docs.aws.amazon.com/sns/latest/dg/sns-system-to-system-messaging.html.

42 of 74

Amazon SNS fanout pattern example

42

Subscribers

Publisher

SNS topic

“new-order”

Ordering system

Data warehouse SQS queue

EC2 instance

Fulfillment SQS queue

EC2�instance

“new order-ABCD”

message

Order fulfillment application

Data warehouse application

DEV queue

Lambda function

Dev database

When a message must be processed by more than one consumer, you can combine pub/sub messaging with a message queue in a fanout design pattern. In a fanout pattern, you use an SNS topic to receive a message that is then pushed out to multiple subscribers for parallel processing.

In the example on the slide, SQS queues subscribe to the "new-order" topic. When a message is pushed from that topic, each queue gets an identical message but performs its own asynchronous processing. In this example, one queue hands off messages to an order fulfillment application, and the other queue interacts with a data warehousing application.

Subscribers in a fanout pattern can be a combination of endpoint types including HTTP endpoints or email addresses.

You could also use fanout to replicate data that is sent to your production environment to your development environment. For example, you could subscribe yet another queue to the same topic for new incoming orders. Then, by attaching this new queue to your development environment, you could continue to improve and test your application by using data received from your production environment. In this example, the DEV queue is connected to a Lambda function that anonymizes data before writing it to a development Amazon DynamoDB database.

43 of 74

Image processing fanout example

43

Source �S3 bucket

Destination �S3 bucket

User

SNS topic

SQS queue�Generate thumbnail

SQS queue�Size image for mobile

SQS queue�Size image for web

Lambda�functions

44 of 74

Section 7: Developing with Amazon SNS

Module 10: Developing with Messaging Services

45 of 74

Amazon SNS message delivery lifecycle

45

SNS topic

Unavailable subscriber endpoint

A

Publisher

A

Return MessageID

Failed delivery

A

Retry per delivery policy

Dead-letter queue

A

Discard or send to dead-letter queue when retries are exhausted

46 of 74

Amazon SNS API operations

46

API operation	Description	Input	Output
CreateTopic	Creates a topic	Topic name	ARN of topic
Subscribe	Prepares to subscribe to an endpoint	Subscriber’s endpoint Protocol ARN of topic	ARN of topic
ConfirmSubscription	Verifies an endpoint owner's intent to receive messages	Token sent to endpoint
Publish	Sends a message to all of a topic's subscribed endpoints	Message Message attributes (optional) Message structure:json (optional) Subject (optional) ARN of topic	Message ID
DeleteTopic	Deletes a topic and all of its subscriptions	ARN of topic

Developers should be familiar with the following common API calls for Amazon SNS:

CreateTopic: Creates a topic where notifications can be published. If a requester already owns a topic with the specified name, that topic's Amazon Resource Name (ARN) is returned without creating a new topic.
Subscribe: Prepares to subscribe to an endpoint by sending a confirmation message to the endpoint. If the service was able to create a subscription immediately (without requiring endpoint owner confirmation), the response of the Subscribe request includes the ARN of the subscription. To actually create a subscription, the endpoint owner must call the ConfirmSubscription action with the token from the confirmation message.
ConfirmSubscription: Verifies an endpoint owner's intent to receive messages by validating the token that was sent to the endpoint by an earlier Subscribe action. If the token is valid, the action creates a new subscription and returns its ARN.
Publish: Sends a message to all of a topic's subscribed endpoints. When a message ID is returned, the message has been saved, and Amazon SNS will attempt to deliver it to the topic's subscribers.
DeleteTopic: Deletes a topic and all of its subscriptions. Deleting a topic might prevent some messages that were previously sent to the topic from being delivered to subscribers.

For more information, see the Amazon SNS API Reference: https://docs.aws.amazon.com/sns/latest/api/Welcome.html

47 of 74

Amazon SNS message filtering

47

Subscribers with filter policies

Ecommerce

website

Online buyer

SNS topic

Shopping

Events

SQS queue

Payment

Lambda

function�SearchIndex

Payment

gateway

Search

engine

event_type=

order_cancelled

event_type=

order_placed

event_type=

product_page_visited

Filter: �"event_type": [{"prefix": "order"}]

Filter:

"event_type": ["product_page_visited"]

By default, an SNS topic subscriber receives every message that is published to the topic. To receive a subset of the messages, a subscriber must assign a filter policy to the topic subscription. A filter policy is a simple JSON object that contains attributes that define which messages the subscriber receives. When you publish a message to a topic, Amazon SNS compares the message attributes to the attributes in the filter policy for each of the topic's subscriptions. If any of the attributes match, Amazon SNS sends the message to the subscriber. Otherwise, Amazon SNS skips the subscriber without sending the message. If a subscription doesn't have a filter policy, the subscription receives every message that is published to its topic.

In this example, an online buyer visits a website, cancels one order, and places another order. These messages are published to the SNS topic Shopping Events. The Payment SQS queue subscriber has a filter policy applied so it receives only the messages about the orders. The Lambda function subscriber has a filter policy to receive only the messages that the product page was visited.

For more information, see Amazon SNS Message Filtering in the Amazon SNS Developer Guide at https://docs.aws.amazon.com/sns/latest/dg/sns-message-filtering.html.

48 of 74

Subscription filter policy example

48

topic_arn = sns.create_topic(

Name=‘ShoppingEvents’

)['TopicArn'}

search_engine_subscription_arn = sns.subscribe(

TopicArn = topic_arn,

Protocol = 'lambda',

Endpoint = 'arn:aws:lambda:us-east-1:123456789012:function:SearchIndex'

)['SubscriptionArn']

sns.set_subscription_attributes(

SubscriptionArn = search_engine_subscription_arn,

AttributeName = 'FilterPolicy'

AttributeValue = '{"event_type": ["product_page_visited"]}'

)

SNS topic

Attribute in filter policy

Subscriber

49 of 74

Message with attributes example

49

message = '{"product": {"id": 1251, "status": "in_stock"},'\

' "buyer": {"id":4454}}'

sns.publish(

TopicArn = topic_arn,

Subject = "Product Visited #1251",

Message = message,

MessageAttributes = {

'event_type': {

'DataType': 'String',

'StringValue': 'product_page_visited'

}

Message

Message attribute that matches the attribute in the filter policy

50 of 74

Three types of Amazon SNS security

IAM and Amazon SNS policies

Server-side encryption (SSE) with AWS KMS

Amazon VPC endpoint

50

Data encryption

Internetwork traffic privacy

Identity and access management

You have detailed control over which endpoints a topic allows, who is able to publish to a topic, and under what conditions:

For identity and access management: Amazon SNS is integrated with IAM so that you can specify which Amazon SNS actions a user in your AWS account can perform with your Amazon SNS resources. You can specify a particular topic in the policy. For example, when you create an IAM policy, you can use variables that give certain users permissions to use the Publish action with specific topics in your AWS account. You can use an IAM policy to restrict your users' access to Amazon SNS actions and topics. An IAM policy can restrict access only to users within your AWS account, not to other AWS accounts. You can use an Amazon SNS policy with a particular topic to restrict who can work with that topic (for example, who can publish messages to it and who can subscribe to it). Amazon SNS policies can give access to other AWS accounts or to users within your own AWS account.

For data encryption: Encrypt your data using server-side encryption (SSE). SSE protects the contents of messages in SNS topics by using keys that are managed in AWS KMS.

For internetwork traffic privacy: If you use Amazon VPC to host your AWS resources, you can establish a private connection between your VPC and Amazon SNS. With this connection, you can publish messages to your SNS topics without sending them through the public internet.

For more information about Amazon SNS security, see the Amazon SNS Developer Guide at https://docs.aws.amazon.com/sns/latest/dg/sns-security.html.

51 of 74

Demonstration: Working with Amazon Messaging Services

51

52 of 74

Sections 6 and 7 key takeaways

Developers publish messages to SNS topics, and subscribers to that topic get copies of all published messages.
With attribute filtering, subscribers receive only relevant messages.
Subscriber endpoints include both application-to-application (A2A) and application-to-person (A2P) types.
Amazon SNS security includes managing access to SNS resources, protecting data sent to topics, and limiting exposure of messages to the internet.

52

53 of 74

Section 8: Introducing Kinesis Data Streams

Module 10: Developing with Messaging Services

54 of 74

Kinesis Data Streams

Continuously capture gigabytes of data per second
Source stream data from sources including website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events
Easily process data with built-in integrations with Lambda and other Kinesis services such as Kinesis Data Analytics and Kinesis Data Firehose

Fully managed, massively scalable and durable real-time data streaming service that you can use to ingest, buffer, and process streaming data in real time

54

Amazon Kinesis�Data Streams

Amazon Kinesis Data Streams is a fully managed data streaming service. Kinesis Data Streams can continuously capture gigabytes of data per second from hundreds of thousands of sources such as website clickstreams, database event streams, financial transactions, social media feeds, IT logs, and location-tracking events. The data collected is available in milliseconds to enable real-time analytics use cases.

Kinesis Data Streams reduces the overhead of building a streaming application with tools including the AWS SDK, the Kinesis Client Library (KCL), connectors, and agents. Kinesis Data Streams also has built-in integrations to AWS Lambda, Amazon Kinesis Data Analytics, Amazon Kinesis Data Firehose, and AWS Glue Schema Registry to simplify setting up a consumer to read records from the stream.

55 of 74

Kinesis Data Streams use cases

Log and event data collection

Real-time analytics

Mobile data capture

Gaming data feed

55

56 of 74

Kinesis Data Streams overview

56

Kinesis data stream

Producer

Consumers

Producer

Put messages on the stream

The stream stores data durably distributed on shards

Custom application written with KCL deployed on Amazon EC2

Lambda/�Lambda function

Kinesis Data Firehose stream connected to a destination like Amazon S3

Kinesis Data Analytics application connected to another stream or a Lambda function

Poll the stream

Shard 1

Shard 2

Records are available during the retention period

A producer collects data and puts it on to the stream; for example, a web server that sends log data to a Kinesis data stream is a producer. You can build producers for Kinesis Data Streams using the Kinesis Producer Library (KPL) or the AWS SDK for Java.

Data records also include a partition key and a data blob. The partition key is used to group data by shard within a stream. The sequence number is unique per partition key within its shard. For writes, a shard can support up to 1,000 records per second or up to a maximum of 1 MB per second. For reads, a shard can support up to 5 transactions per second or up to a maximum of 2 MB per second.

Kinesis Data Streams stores a unit of data as a data record. A data stream represents a group of data records. The data stream ingests a large amount of data in real time, durably stores the data, and makes the data available for consumption. The data records in a data stream are distributed into shards. After a record is added to the stream, the record is available for a specified retention period, which you can set per stream. The Kinesis Data Streams service adds shards to scale horizontally.

Each shard has a uniquely identified sequence of data records, and each data record has a sequence number that Kinesis assigns.

A partition key is used to group data by shard within a stream. When you create a stream, you specify the number of shards for the stream. The total capacity of a stream is the sum of the capacities of its shards.

A consumer is an application that polls the data stream and processes the data from a Kinesis data stream. You can have multiple consumers on a data stream. When you have multiple consumers, they share the read throughput of the stream among them. An option called enhanced fanout lets you give each consumer its own allotment of read throughput.

Create consumers using any of the following options:

Build your own application to process stream data using the Kinesis Consumer Library (KCL) and deploy it to Amazon EC2.
Configure a Lambda function as a consumer of the stream to process data serverlessly, and let Lambda handle aspects of consuming the stream.
Set up Kinesis Data Firehose as a consumer to deliver records to destinations including Amazon S3, Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk.
Send records to a Kinesis Data Analytics application to process and analyze data in a Kinesis data stream using SQL, Java, or Scala.

Note that, unlike queue consumers, stream consumers do not delete records from the stream. Instead, each consumer must maintain a pointer of where they are on the stream.

57 of 74

Other Kinesis data streaming services

Amazon Kinesis Data Firehose

Deliver streaming data to data stores without writing consumer applications for your stream
Automatically convert incoming data to open and standards based before the data is delivered

Amazon Kinesis Data Analytics

Perform real-time analysis on data in the stream using SQL queries before persisting the data
Use Apache Flink, Java, Scala, or Python for your analysis application

57

As noted in the previous slide, consumers of a stream might be other Kinesis streaming services such as Kinesis Data Firehose and Kinesis Data Analytics. These services were created to simplify common Kinesis Data Streams use cases and are often used in conjunction.

Amazon Kinesis Data Firehose is the easiest way to reliably load streaming data into data lakes, data stores, and analytics services. The service can capture, transform, and deliver streaming data to Amazon S3, Amazon Redshift, Amazon ES, generic HTTP endpoints, and service providers such as Datadog, New Relic, MongoDB, and Splunk.

With Kinesis Data Firehose, you can automatically convert the incoming data to open and standards based formats such as Apache Parquet and Apache ORC before the data is delivered.

With Kinesis Data Firehose, you don't need to write consumer applications or manage shards. You configure your data producers to send data to Kinesis Data Firehose, and the service automatically delivers the data to a destination that you specify.

With Kinesis Data Analytics, you can do real-time analysis by using SQL queries before persisting the data. The service is designed for near-real-time queries, and you can aggregate data across a sliding window.

With Kinesis Data Analytics, you write SQL statements or applications, and then upload them into a Kinesis Data Analytics application to perform analysis on the data in the stream. Kinesis Data Analytics also supports using a Lambda function to preprocess the data before your SQL runs.

Kinesis Data Analytics applications can enrich data by using reference sources, aggregate data over time, or use machine learning to find data anomalies. Then you can write the analysis results to another Kinesis data stream, a Kinesis Data Firehose delivery stream, or a Lambda function.

58 of 74

Kinesis Data Streams example

58

Amazon Kinesis

Data Streams

Amazon Kinesis

Data Firehose

Amazon Simple Storage Service (Amazon S3)

Amazon Elastic Compute Cloud (Amazon EC2)

AWS Lambda

Amazon Elastic Container Service

(Amazon ECS)

Amazon Kinesis

Data Analytics

Amazon Elasticsearch Service (Amazon ES)

Amazon Athena

Amazon CloudWatch

In this example architecture, customer interaction logs from multiple applications and microservices running across Amazon EC2, Amazon Elastic Container Service (Amazon ECS), and Lambda are all producers that add records to a Kinesis data stream. Consumers of the stream include Kinesis Data Firehose and Kinesis Data Analytics.

Kinesis Data Firehose delivers the streaming data to two storage destinations: Amazon ES and Amazon S3. The application owners can use Amazon ES for operational insights into their applications. Business users can use Amazon Athena or a similar query tool to interact with the data stored on Amazon S3 without building a complex Extract, Transform, and Load (ETL) processor.

At the same time, Kinesis Data Analytics consumes the stream and generates metrics, percentiles, and derived values. These values are then emitted to Amazon CloudWatch where they are part of an overall monitoring solution.

59 of 74

Kinesis API operations

59

Kinesis data stream

Producers

Consumers

Shard 1

Shard 2

CreateStream: Creates a stream. Requires ShardCount and StreamName.

DeleteStream: Deletes the named stream and all of its data shards.

DescribeStreamSummary: Provides summary information about a named stream.

UpdateShardcount: Modifies the number of shards on the stream.

MergeShards: Merges two shards to reduce the stream's capacity.

SplitShards: Splits a shard into two shards to increase the stream's capacity.

PutRecord/PutRecords: Writes a record or records to the stream. Includes a partition key that determines which shard the record(s) will be written to.

GetShardIterator: Gets the shard iterator, which tells you the shard position from which to start reading records.

GetRecords: Reads records from the stream, starting from a specified shard iterator.

60 of 74

Examples using the AWS CLI�for basic tasks

60

aws kinesis create-stream --stream-name MyStream --shard-count 1

aws kinesis describe-stream-summary --stream-name MyStream

aws kinesis put-record --stream-name MyStream --partition-key 123 --data testdata

Create a stream named MyStream with one shard

Check whether your new stream is ready to be used

Put a test record on the stream

Get the starting position to read the stream, then use the output value to get records

aws kinesis get-shard-iterator --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON --stream-name MyStream

aws kinesis get-records --shard-iterator AAAAAAAAAAHSywl…

61 of 74

Building producers and consumers

61

Feature	Kinesis Producer Library (KPL)	Kinesis Client Library (KCL)	Kinesis API with AWS SDK
Abstracts stream management tasks so you can focus on application logic	✔	✔
Handles producer tasks such as writing with automatic retries and aggregating records to optimize throughput	✔
Handles consumer subtasks such as load balancing and checkpointing processed records		✔
Provides the ability to write producer or consumer applications using direct interaction with the Kinesis API			✔
Requires your application code to handle stream management			✔

To build your producer and consumer applications, you would typically use the KPL and KCL, or possibly the AWS SDK.

Both the KPL and KCL abstract stream management tasks so that you can focus on application logic. If you need to build custom producer and consumer applications, use the KPL and KCL unless your use case has a specific need that the libraries cannot meet.

The KPL is a highly configurable library that helps you write to a Kinesis data stream. The library acts as an intermediary between your producer application code and the Kinesis Data Streams API actions. The KPL performs the following primary tasks:

Writes to one or more Kinesis data streams with an automatic and configurable retry mechanism
Collects records and uses PutRecords to write multiple records to multiple shards per request
Aggregates user records to increase payload size and improve throughput
Integrates seamlessly with the KCL to deaggregate batched records on the consumer
Submits CloudWatch metrics on your behalf to provide visibility into producer performance

The KCL helps you consume and process data from a Kinesis data stream by taking care of many of the complex tasks that are associated with distributed computing. These include load balancing across multiple consumer application instances, responding to consumer application instance failures, checkpointing processed records, and reacting to resharding.

You can build producer and consumer applications by using the AWS SDK to interact directly with the Kinesis Data Streams API. However, for most use cases, use the KPL or KCL to remove the complexities of handling all of the subtasks of stream processing so that you can focus on your application's business logic.

62 of 74

Comparison of queues and streams

62

	Queues	Streams
Data value	The value comes from processing individual messages.	The value comes from aggregating messages to get actionable data.
Message rate	The message rate is variable.	The message rate is continuous and high volume.
Message processing	Messages are deleted after a consumer successfully processes them.	Messages are available to multiple consumers to process in parallel, and each consumer maintains a pointer but does not delete records.
Example use cases	Financial transactions, product orders	Clickstream data, application logs

As noted earlier in this module, although both queues and streams are messaging services that use polling to retrieve messages from an interim store, queues and streams are suited to different types of data patterns, and each process the data a bit differently.

The following are characteristics of queues:

An individual message is the core entity where you apply some processing or compute. The message could be a financial transaction or a command to control an Internet of Things (IoT) device.
The occurrence of messages typically varies.
Messages must be deleted off the queue once they have been processed.

The following are characteristics of stream processing:

A stream of messages is the core entity. You can only get the value of each individual message by looking at many messages together.
The stream of messages is also constant in most cases.
Consumers keep track of where they are on the stream but do not delete records from the stream.
Clickstream data from mobile apps, application logs, and home security camera feeds are common examples where streams provide the responsiveness that you need to make data actionable.

63 of 74

Section 8 key takeaways

With Kinesis Data Streams, you can ingest, buffer, and process streaming data in real time.
Kinesis Data Firehose and Kinesis Data Analytics were designed to simplify common data streaming use cases.
Producers put records on to a stream where the records are stored in shards, and consumers get records off the stream for processing.
The Kinesis Producer Library (KPL) and Kinesis Client Library (KCL) abstract stream interactions for developers.

63

64 of 74

Lab 10.1: �Implementing a Messaging System Using Amazon SNS and Amazon SQS��

64

65 of 74

Lab: Tasks

Preparing the development environment
Configuring the Amazon SQS dead-letter queue
Configuring the Amazon SQS queue
Configuring the Amazon SNS topic
Linking Amazon SQS and Amazon SNS
Testing message publishing
Configuring the application to poll the queue

65

66 of 74

Lab: Final product

66

Elastic Beanstalk deployment

Coffee suppliers application

Aurora Serverless

database

Coffee suppliers

send inventory updates

Amazon SNS�topic

Amazon SQS�queue

Amazon SQS�dead-letter queue

67 of 74

Begin Lab 10.1: Implementing a Messaging System Using Amazon SNS and Amazon SQS

67

~ 90 minutes

68 of 74

Lab debrief: �Key takeaways

68

69 of 74

Module wrap-up

Module 10: Developing with Messaging Services

70 of 74

Module summary

In summary, in this module, you learned how to do the following:

Illustrate how messaging services, including queues, pub/sub messaging, and streams, support asynchronous processing
Describe Amazon SQS
Send messages to an SQS queue
Describe Amazon SNS
Subscribe an SQS queue to an SNS topic
Describe how Kinesis can be used for real-time analytics

70

71 of 74

Complete the knowledge check

71

72 of 74

Sample exam question

A developer wants to introduce an asynchronous connection in their application workflow that processes individual banking transactions including deposits and withdrawals. Transactions must be handled in the order they arrive. Which method would meet the developer's needs?

Send transaction messages to a standard SQS queue.
Put transaction messages on to a Kinesis data stream.
Send transaction messages to a FIFO SQS queue.
Put transactions on to a Kinesis Data Firehose delivery stream.

72

73 of 74

Thank you

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. This work may not be reproduced or redistributed, in whole or in part, without prior written permission from Amazon Web Services, Inc. Commercial copying, lending, or selling is prohibited. Corrections, feedback, or other questions? Contact us at https://support.aws.amazon.com/#/contacts/aws-training. All trademarks are the property of their owners.

74 of 74

Messaging services as part of developing a cloud application

74

Step Functions �retrieves data from RDS and posts report to S3 with presigned URL

Amazon Cognito provides token for report requests

Amazon S3�

Developers

AWS Cloud9

CloudShell

AWS CLI

SDK for Python

Console

�bucket hosts café website

Café website users

Webpage �requests

Bucket policy

DynamoDB database stores products table

API Gateway REST API

Lambda function �does database lookups

Café employees

Elastic Beanstalk

ECR hosts Docker image

EC2 instance with Docker container runs coffee supplier website

Aurora Serverless on RDS stores supplier database

AWS WAF �secures API endpoint

Café employees

Refresh cache per settings

CloudFront distribution

ElastiCache �for Memcached

AWS WAF �secures website

SNS

SQS

Suppliers

Coffee inventory

updates

Lambda