Example real-time reference architecture for AWS IoT

Many companies are leveraging physical devices to send, collect, and share data so it can be analyzed within seconds. The messages that come from these devices are high in velocity and volume and require a scalable cloud system that can process the messages. In order to query the freshest incoming data, a real-time analytics database is needed to handle low-latency analytics.

In this example, there’s an uptick in traffic. But, we’re unsure if it’s a malicious attack. How can we discover what the uptick in traffic is in real-time (in seconds) and possibly prevent any downtime? In this architecture design, we are leveraging AWS IoT Core so the devices can easily interact with each other and cloud apps. Since we need to analyze the data in seconds, we are using a real-time analytics database like Rockset.

There may be variations in the design, but the logic should still be the same.

1 | Device messages are transmitted and received in AWS IoT core via MQTT protocol. AWS IoT core lets you connect IoT devices and route messages to AWS services.

2 | Messages are routed to AWS SqS (queue) via AWS SNS or directly to AWS Kinesis Data Firehose. The publisher in AWS SNS topics fans out messages to subscriber systems, here being AWS SQS.

3a | AWS Kinesis Data Firehose delivers the streaming data to AWS S3.

3b | AWS Lambda writes the SQS messages to DynamoDB.

4 | You can analyze DynamoDB data using SQL in Rockset. Rockset continuously monitors and syncs changes from the DynamoDB table (inserts, deletes and updates) to a collection in a few seconds using DynamoDB streams. Data is mutable in Rockset- you can do inserts and updates. As data is added to a collection, it’s indexed automatically via Converged Index and can be rolled up. Converged Index is how Rockset SQL queries have millisecond response times.

5 | The query results are automatically sent to an AI/ML model.

6 | The i.e. security team or other teams can act on the results that come from the model.

In general, when you’re thinking about how to architect a data stack that is able to process and analyze IoT messages as fast as possible, you want to pick a scalable cloud system and a real-time analytics database that optimizes for data freshness, scalability, and query performance. This allows your team to iterate quickly and maximize their time in building a better product.

1 Like