AWS

Time series processing architecture in AWS
CK CK 3788 
Created at
Updated at  
 10,559 0 0

When data arrives as a succession of regular measurements, it is known as time series information. Processing of time series information poses systems scaling challenges that the elasticity of AWS services is uniquely positioned to address.
 

This elasticity is achieved by using Auto Scaling groups for ingest processing, AWS Data Pipeline for intersystem data orchestration, and Amazon Redshift for potentially massive-scale analysis. Key architectural throttle points involving Amazon SQS for sensor message buffering and less frequent AWS Data Pipeline scheduling keep the overall solution costs perdictable and controlled.

Time series processing architecture in AWS

From above diagram, regarding a Supervisory Control and Data Acquisition (SCADA), we created a flow of samples to or from Amazon DynamoDB to support additional cloud processing or other existing systems, respectively.

Using AWS Data Pipeline, create a pipeline with a regular Amazon Elastic MapReduce job that both calculates expensive sample processing and delivers samples and results.

The pipeline places results into Amazon Redshift for additional analysis, and exports historical week-oriented sample tables, from Amazon DynamoDB to Amazon Simple Storage Service (Amazon S3). The pipeline also optionally exports results in a format custom applications can accept.

Amazon Redshift optionally imports historic samples to reside with calculated results.

Tags AWS Architecture AWS Data Pipeline Amazon DynamoDB Amazon EC2 Amazon S3 Amazon SQS Auto Scaling Facebook X
Comments 0
Similar posts
  1. Web Application Hosting provides the reliable, scalable, secure, and high-performance infrastructure
    10,615
  2. Synchronous online gaming architecutre in AWS - Highly Available, Scalable & Elastic to support Millions of Players
    10,417
  3. Web front-end architecture in AWS for e-commerce website
    365
  4. Secure and Scalable LAMP Service Architecture in AWS
    10,660
  6. Live streaming / Static Content & Media Service based on AWS
    10,655
  7. Large scale messenger service architecture in AWS - Slack case study
    11,674
  8. Large scale computing based on huge data sets in AWS
    10,609
  9. Image Moderation Chatbot - a serverless chatbot on AWS that monitors chat channels and removes images suggestive or explicit image
    10,425
  10. Financial services grid computing in AWS
    10,354
  11. File synchronization service architecture in AWS
    10,376
  12. Fault Tolerance & High Availability based on multi Availability zones in AWS
    10,805
  13. Disaster recovery solution in AWS for local applications
    11,178
  14. Content Distribution for VOD streaming services in AWS
    10,355
  15. Checkout service architecture in AWS for e-commerce website
    11,009
  17. Batch processing composed of job manager, job store and worker based on AWS
    10,448
  18. AWS Architecture for collecting Click-Through Events on Advertisement serving
    10,374
  19. Teraform from the scratch
    1,355
  20. Product Marketing & Recommendation Architecture in AWS for e-commerce website
    10,736
  21. Architecting for Resilience: AWS High Availability Components Including API Gateway
    137
  22. Designing Resilient Infrastructure to Prevent Order Data Loss During Kafka Failures
    154
  23. Architecting Dynamic Scale-Out and Scale-In with Docker Containers
    133
  24. How to Set Up and Deploy Containerized Applications with AWS ECS
    10,673
  25. Predictable Ad Spend: Architecting Pacing at Hyperscale
    422
  26. Measuring Ad Viewability: AWS Architectures for Programmatic Accountability
    476
  27. Ad Tech Multi-Tenancy: A Spectrum of Isolation Architectures on AWS
    331
  28. Enterprise Data Lake Architecture based on AWS
    10,405
  30. Media sharing service architecture in AWS
    10,366
  31. A Python/PHP code to deregister anormalous instaces in Amazon ECS
    1,857
  32. A Python script to create an EC2 Spot instance using boto3
    1,602
  33. AWS System Design Example - Instagram Architecture
    1,917
More in AWS
  1. "502 Bad Gateway" Error easy to happen at the combination of nginx and node.js in AWS Elastic Beanstalk
    10,487
  2. How to Set Up and Deploy Containerized Applications with AWS ECS
    10,673
  3. Power of Containerized Deployment with AWS ECS
    10,597
  4. FAISS Already Does ANN... So Why Do Top Teams Still Pay for SageMaker?
    309
  5. Recommended Vector DB Structure for Amazon Bedrock Integration
    389
  6. Product Marketing & Recommendation Architecture in AWS for e-commerce website
    10,736
  7. What cause a 502 bad gateway error in AWS environment?
    444
  8. Building an AWS SAM(Serverless Application Model) Project
    898
  9. Teraform from the scratch
    1,355
  10. Check incomplete multipart uploads if AWS S3 storage size is abnormally big huge compared to the origin storage
    11,597
  12. Importing a wild card domain SSL certificate into AWS Certificate Manager
    10,348
  13. AWS Lambda Example based on Teraform
    900
  14. Simple AWS ECS example based on Teraform
    988
  15. Web front-end architecture in AWS for e-commerce website
    365
  16. Web Application Hosting provides the reliable, scalable, secure, and high-performance infrastructure
    10,615
  17. Synchronous online gaming architecutre in AWS - Highly Available, Scalable & Elastic to support Millions of Players
    10,417
  18. Secure and Scalable LAMP Service Architecture in AWS
    10,660
  19. Live streaming / Static Content & Media Service based on AWS
    10,655
  20. Large scale messenger service architecture in AWS - Slack case study
    11,674
  21. Large scale computing based on huge data sets in AWS
    10,609
  22. Image Moderation Chatbot - a serverless chatbot on AWS that monitors chat channels and removes images suggestive or explicit image
    10,425
  23. Financial services grid computing in AWS
    10,354
  24. File synchronization service architecture in AWS
    10,376
  25. Fault Tolerance & High Availability based on multi Availability zones in AWS
    10,805
  27. Disaster recovery solution in AWS for local applications
    11,178
  28. Content Distribution for VOD streaming services in AWS
    10,355
  29. Checkout service architecture in AWS for e-commerce website
    11,009
  30. Batch processing composed of job manager, job store and worker based on AWS
    10,448
  31. AWS Architecture for collecting Click-Through Events on Advertisement serving
    10,374
  32. Where to get AWS Access Key ?
    11,262
Recently updated
  1. Why 100% SEO Guarantees Better User Engagement and How to Achieve It in Your Service
    40
  2. Why SPF and DKIM Are Essential for Linux Email Servers?
    34
  3. How do I create animated background for Google Meet?
    20,611
  4. MaxMind GeoLite2 vs. DB-IP Lite vs. ip-api.com: Choosing the Right Free IP Geolocation Solution
    244
  6. Meet Valkey: Why Redis Changed Its License, Why Valkey is Better, and How to Use It as a Drop-in Replacement
    94
  7. How Rogue Web Scrapers Are Stealing My Content and Destroying My Server
    157
  8. Array of arguments passed to script in PHP
    90
  9. Invalid Auction Storms in Ad Tech
    98
  10. Building MRC-Compliant Instrumentation in Ad Tech: Implementation Steps and Key Challenges
    202
  11. The Two Clocks of Ad Server Architecture
    70
  12. Frequency Cap Architecture Overview
    514
  13. Collecting Proof-of-Play from 100,000 Screens
    146
  14. Vector Search in Relational Databases (RDBs): Trends & Usage Environments
    287
  15. What is RedisVL?
    109
  16. Designing a Skill Registry Architecture: Minimize Token Usage and Maximize AI Performance
    230
  17. TPU & GPU Cluster Management Tools
    243
  18. The Concept of Scaffolding in AI Agents
    233
  19. Beyond Static Pipelines: Architecting LLM Multi-Agent Systems for Autonomous Workflow Orchestration
    583
  21. AI-DLC vs. AI-SDLC: Model-Centric vs. System-Centric AI Development
    258
  22. Fine-Tuning Open-Weight LLMs: Training, Licensing, and Deploying Custom Models with Ollama
    339
  23. Inside the Next-Generation Streaming Architecture: Ultra-Low Latency, SGAI, and AI Codecs
    202
  24. Machine Learning: Training vs. Ranking
    221
  25. The Continuous Learning Machine: Designing a Loop That Actually Turns
    222
  26. The Next-Gen Low-Latency Standards: LL-HLS & MoQ
    195
  27. Designing a Data Curation System Around a Vector Database
    220
  28. Extracting Context from Live Video: The Architecture Got Simpler When We Stopped Building Features
    256
  29. The New Ad Stack Behind Live Sports Streaming: Inside SGAI, EABN, and the Standards Race to Fix Live Monetization
    656
  30. Designing MLOps Architectures to Handle Seasonal Data Drift in EV Fleet Models
    195
  31. From 15 to 40+ FPS: Accelerating PyTorch Object Detection on Automotive ECUs
    217
  32. Understanding Data Lineage
    260