AWS

Batch processing composed of job manager, job store and worker based on AWS
CK CK 3776 
Created at
Updated at  
 10,444 0 0

Batch processing on AWS allows for the on-demand provisioning of a multi-part job processing architecture that can be used for instantaneous or delayed deployment of a heterogeneous, scalable "grid" of worker nodes that can quickly crunch through large batch oriented applications in place today that can leverage this style of on-demand processing, including claims processing, large scale transformation, media transcoding and multi-part data processing work.
 

Batch processing architectures are often synonymous with highly variable usage patterns that have significant usage peaks (e.g. month-end processing) followed by significant periods of underutilzation.

There are numerous approaches to building a batch processing architecture. This document outlines a basic batch processing architecture that supports job scheduling, job status inspection, uploading raw data, outputting job results, gird management, and reporting job performance data.

 

Batch processing composed of job manager, job store and worker based on AWS

 

#

Description

1Users interact with the Job Manager application which is deployed on an Amazon Elastic Computer Cloud (EC2) instance. This component controls the process of accepting, scheduling, starting, managing, and completing batch jobs. It also provides access to the final results, job and worker statistics, and job progress information.
2Raw job data is updated to Amazon Simple Storage Service (S3), a highly-available and persistent data store.
3Individual job tasks are inserted by the Job Manager in an Amazon Simple Queue Service (SQS) input queue on the user's behalf.
4Worker nodes are Amazon EC2 instances deployed on an Auto Scaling group. This group is a container that ensures health and scalability of worker nodes. Worker nodes pick up job parts from the input queue automatically and perform single tasks that are part of the list of batch processing steps.
5Interim results from worker nodes are stored in Amazon S3.
6Progress information and statistics are stored on the analytics store. This component can be either an Amazon SimpleDB domain or a relational database such as an Amazon Relational Database Service (RDS) instance.
7Optionaly, completed tasks can be inserted in an Amazon SQS queue for chaining to a second processing stages.
Tags AWS Architecture AWS Batch Processing Batch Processing Grid Management Media Transcoding Transcoding Facebook X
Comments 0
Similar posts
  1. AWS Architecture for collecting Click-Through Events on Advertisement serving
    10,371
  2. Checkout service architecture in AWS for e-commerce website
    11,004
  3. Content Distribution for VOD streaming services in AWS
    10,353
  4. Disaster recovery solution in AWS for local applications
    11,176
  5. Fault Tolerance & High Availability based on multi Availability zones in AWS
    10,799
  7. File synchronization service architecture in AWS
    10,374
  8. Financial services grid computing in AWS
    10,354
  9. Image Moderation Chatbot - a serverless chatbot on AWS that monitors chat channels and removes images suggestive or explicit image
    10,422
  10. Large scale computing based on huge data sets in AWS
    10,605
  11. Large scale messenger service architecture in AWS - Slack case study
    11,673
  12. Live streaming / Static Content & Media Service based on AWS
    10,650
  13. Secure and Scalable LAMP Service Architecture in AWS
    10,657
  14. Synchronous online gaming architecutre in AWS - Highly Available, Scalable & Elastic to support Millions of Players
    10,412
  15. Time series processing architecture in AWS
    10,556
  16. Web Application Hosting provides the reliable, scalable, secure, and high-performance infrastructure
    10,613
  18. Web front-end architecture in AWS for e-commerce website
    362
  19. Large-Scale Concurrency Control: Solving the DB Bottleneck from a Flood of “Likes”
    268
  20. Teraform from the scratch
    1,351
  21. Product Marketing & Recommendation Architecture in AWS for e-commerce website
    10,732
  22. Predictable Ad Spend: Architecting Pacing at Hyperscale
    419
  23. Measuring Ad Viewability: AWS Architectures for Programmatic Accountability
    473
  24. Ad Tech Multi-Tenancy: A Spectrum of Isolation Architectures on AWS
    327
  25. Enterprise Data Lake Architecture based on AWS
    10,402
  26. Media sharing service architecture in AWS
    10,363
  27. AWS System Design Example - Instagram Architecture
    1,915
  28. Amagi CLOUDPORT Architecture based on AWS for FAST for channel playout on AWS
    1,314
More in AWS
  1. Web front-end architecture in AWS for e-commerce website
    362
  3. Web Application Hosting provides the reliable, scalable, secure, and high-performance infrastructure
    10,613
  4. Time series processing architecture in AWS
    10,556
  5. Synchronous online gaming architecutre in AWS - Highly Available, Scalable & Elastic to support Millions of Players
    10,412
  6. Secure and Scalable LAMP Service Architecture in AWS
    10,657
  7. Live streaming / Static Content & Media Service based on AWS
    10,650
  8. Large scale messenger service architecture in AWS - Slack case study
    11,673
  9. Large scale computing based on huge data sets in AWS
    10,605
  10. Image Moderation Chatbot - a serverless chatbot on AWS that monitors chat channels and removes images suggestive or explicit image
    10,422
  11. Financial services grid computing in AWS
    10,354
  12. File synchronization service architecture in AWS
    10,374
  13. Fault Tolerance & High Availability based on multi Availability zones in AWS
    10,799
  15. Disaster recovery solution in AWS for local applications
    11,176
  16. Content Distribution for VOD streaming services in AWS
    10,353
  17. Checkout service architecture in AWS for e-commerce website
    11,004
  18. AWS Architecture for collecting Click-Through Events on Advertisement serving
    10,371
  19. Where to get AWS Access Key ?
    11,259
Recently updated
  1. How do I create animated background for Google Meet?
    18,998
  2. MaxMind GeoLite2 vs. DB-IP Lite vs. ip-api.com: Choosing the Right Free IP Geolocation Solution
    192
  3. Meet Valkey: Why Redis Changed Its License, Why Valkey is Better, and How to Use It as a Drop-in Replacement
    75
  4. How Rogue Web Scrapers Are Stealing My Content and Destroying My Server
    151
  5. Array of arguments passed to script in PHP
    83
  6. Invalid Auction Storms in Ad Tech
    91
  7. Building MRC-Compliant Instrumentation in Ad Tech: Implementation Steps and Key Challenges
    194
  8. The Two Clocks of Ad Server Architecture
    61
  10. Frequency Cap Architecture Overview
    505
  11. Collecting Proof-of-Play from 100,000 Screens
    139
  12. Vector Search in Relational Databases (RDBs): Trends & Usage Environments
    276
  13. What is RedisVL?
    102
  14. Designing a Skill Registry Architecture: Minimize Token Usage and Maximize AI Performance
    218
  15. TPU & GPU Cluster Management Tools
    229
  16. The Concept of Scaffolding in AI Agents
    217
  17. Beyond Static Pipelines: Architecting LLM Multi-Agent Systems for Autonomous Workflow Orchestration
    568
  18. AI-DLC vs. AI-SDLC: Model-Centric vs. System-Centric AI Development
    246
  19. Fine-Tuning Open-Weight LLMs: Training, Licensing, and Deploying Custom Models with Ollama
    321
  20. Inside the Next-Generation Streaming Architecture: Ultra-Low Latency, SGAI, and AI Codecs
    196
  21. Machine Learning: Training vs. Ranking
    200
  22. The Continuous Learning Machine: Designing a Loop That Actually Turns
    212
  23. The Next-Gen Low-Latency Standards: LL-HLS & MoQ
    188
  25. Designing a Data Curation System Around a Vector Database
    212
  26. Extracting Context from Live Video: The Architecture Got Simpler When We Stopped Building Features
    228
  27. The New Ad Stack Behind Live Sports Streaming: Inside SGAI, EABN, and the Standards Race to Fix Live Monetization
    646
  28. Designing MLOps Architectures to Handle Seasonal Data Drift in EV Fleet Models
    179
  29. From 15 to 40+ FPS: Accelerating PyTorch Object Detection on Automotive ECUs
    203
  30. Understanding Data Lineage
    251
  31. System Requirements and Architecture: Intelligent Priority-Based Scheduler
    159
  32. GPU & LLM Cluster Infrastructure & Task Scheduling Architecture
    229