Application integration on AWS is a suite of services that enable communication between decoupled components within microservices, distributed systems, and serverless applications. Applications need to communicate with each other and can use a couple of patterns for application communication. There is synchronous communication, an application to application messaging, or an event-based option, which is asynchronous, using an application to queue to application architecture. Synchronous between applications can be problematic if there are sudden spikes of traffic. In that case, it’s better to decouple your applications using SQS or SNS. These services can scale independently from the actual application.
- AWS SQS (Simple Queue Service)
- AWS Kinesis Data Streams
- Amazon SNS (Simple Notification Service)
- Amazon MQ (managed message broker service)
AWS SQS (Simple Queue Service)
Amazon SQS is a message queue service used by distributed applications to exchange messages via a polling model, and can be used to decouple sending and receiving components. Amazon Simple Queue Service (SQS) is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. The AWS service that enables the decoupling and scaling of applications is Amazon Simple Queue Service (Amazon SQS).
• Fully managed service (~serverless), used to decouple applications • Scales from 1 message per second to 10,000s per second
• Default retention of messages: 4 days, maximum of 14 days
• No limit to how many messages can be in the queue
• Messages are deleted after consumers read them
• Low latency (<10 ms on publishing and receiving)
• Consumers share the work to read messages & scale horizontally
Amazon Simple Queue Service (Amazon SQS) can be used to decouple applications. Amazon Simple Queue Service (Amazon SQS) helps developers use loose coupling and reliable messaging between microservices. The ability to upgrade components independently is a benefit of decoupling an AWS Cloud architecture.
Amazon Simple Queue Service (Amazon SQS) and AWS Lambda can be used to build an operationally efficient solution for a company that needs to simultaneously process hundreds of requests from different users.
- Getting started with Amazon SQS – Amazon Simple (docs.aws.amazon.com)
- Getting Started with Amazon SQS | Message Queuing (aws.amazon.com)
- How to use AWS: SNS and SQS – Medium (medium.com)
- php – How to properly use AWS SQS – Stack Overflow (stackoverflow.com)
- How to use Amazon Lambda with Amazon SQS? – AWSMAG (awsmag.com)
- How to Manage & Design AWS SQS Java: 5 Easy Steps – Learn (hevodata.com)
- How to Instrument AWS Services with OpenTelemetry | Aspecto (www.aspecto.io)
- Using AWS SQS with Node Js application – Relinns Technologies (www.relinns.com)
- Using SQS with AWS Lambda and Serverless (serverless.com)
AWS Kinesis Data Streams
Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store data streams at any scale. While Kinesis and Apache Kafka offer similar functionality and cover the same use cases, they are unrelated. Kinesis Data Streams is a proprietary product developed by AWS and is not based on open-source Apache Kafka.
• Kinesis Data Streams: low latency streaming to ingest data at scale from hundreds of thousands of sources
• Kinesis Data Firehose: load streams into S3, Redshift, ElasticSearch, etc.
• Kinesis Data Analytics: perform real-time analytics on streams using SQL
• Kinesis Video Streams: monitor real-time video streams for analytics or ML
A company can use Amazon Kinesis Data Analytics to analyze streaming user data and respond to customer queries in real-time.
Amazon SNS (Simple Notification Service)
Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication. In simple terms, SNS – sends messages to the subscriber using a push mechanism without needing a pull. SQS – it is a message queue service used by distributed applications to exchange messages through a polling model and can be used to decouple sending and receiving components.
• The “event publishers” only send messages to one SNS topic
• As many “event subscribers” as we want to listen to the SNS topic notifications • Each subscriber to the topic will get all the messages
• Up to 12,500,000 subscriptions per topic, 100,000 topics limit
- Getting started with Amazon SNS – Amazon Simple (docs.aws.amazon.com)
- Getting Started with Amazon Simple Notification Service (aws.amazon.com)
- Amazon SNS Tutorial for Sending Email Notifications (mailtrap.io)
- How to use Amazon Simple Notification Service (SNS) (www.australtech.net)
- Quick Tips to use Amazon Simple Notification Service (www.whizlabs.com)
- How to use Amazon SNS – docs.deliveryapi.com (docs.deliveryapi.com)
- Amazon SNS (from AWS) (www.serverless.com)
- How to use IBM App Connect with Amazon SNS (www.ibm.com)
- How to Send a Mail With Amazon SNS? | AWS in Plain English (aws.plainenglish.io)
- Set up Amazon SNS | Deep Security – Trend Micro (help.deepsecurity.trendmicro.com)
- Amazon SNS Review 2022. Features, Pricing & Alternatives (www.semplaza.com)
Amazon MQ (managed message broker service)
Amazon MQ is a managed message broker service for Apache ActiveMQ and RabbitMQ that makes it easy to set up and operate message brokers on AWS. Amazon MQ reduces your operational responsibilities by managing the provisioning, setup, and maintenance of message brokers for you. Amazon MQ now supports RabbitMQ, a popular open-source message broker. This enables you to migrate your existing RabbitMQ message brokers to AWS without rewriting code. Amazon MQ is a fully managed service that provisions and contains open source message brokers like RabbitMQ and Apache ActiveMQ for you. Operational resilience and Business agility are benefits of migrating to the AWS Cloud.
• SQS and SNS are “cloud-native” services, and they’re using proprietary protocols from AWS.
• Traditional applications running from on-premise may use open protocols such as MQTT, AMQP, STOMP, Openwire, and WSS
• When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ
• Amazon MQ = managed Apache ActiveMQ
• Amazon MQ doesn’t “scale” as much as SQS / SNS
• Amazon MQ runs on a dedicated machine (not serverless)
• Amazon MQ has both queue feature (~SQS) and topic features (~SNS)
- Getting started with Amazon MQ – Amazon MQ (docs.aws.amazon.com)
- Amazon MQ FAQs | Managed Message Broker Service (aws.amazon.com)
- Getting started with Amazon MQ. How to use Amazon (medium.com)
- Getting Started with Amazon MQ – Managed Message Broker Service (www.youtube.com)
- Connecting to AmazonMQ (ActiveMQ) broker from .NET using (stackoverflow.com)
- Implement message sending and receiving with Spring Boot + Amazon MQ (kbilel.com)
- Using MuleSoft With Amazon MQ – dejim.com (dejim.com)
- How to Connect to an Amazon MQ service using the AMQP Adapter (blogs.sap.com)
AWS Cloud Integration Summary
SQS is a queue service in Amazon Web Services. There are several producers and messages are kept up to fourteen days. Multiple consumers share the read and delegate messages when done. SQS is used to decouple applications deployed to Amazon Web Services. SNS is a notification service in AWS. The subscribers can be Mobile, HTTP, SQS, Lambda, Email, or other options. If there are multiple subscribers, the messages are sent to all of them. In SNS there is no message retention. For applications that leverage real-time streaming, persistence, and analysis, one can choose AWS Kinesis. Finally, the managed Apache MQ services named Amazon MQ can offer both queue and topic features. If a company is considering a migration from on premises to the AWS Cloud and the company’s IT team needs to offload support of the workload, they should use AWS Managed Services to provision, run, and support the company infrastructure.