Data Lake.zip

Enterprise Data Lake Architecture based on AWS

Enterprise Data Lake Architecture based on AWS

10,410

A data lake is a centralized repository that stores raw,unprocessed data from various sources,regardless of structure or format,using a schema-on-read approach.It allows organizations to collect and store large volumes of data at scale,enabling them to ana...

Architecting Ad Tech's Terabyte Log Pipeline - Log Aggregation

Architecting Ad Tech's Terabyte Log Pipeline - Log Aggregation

330

A single ad exchange processing 100k requests per second generates terabytes of logs per day: impression logs (one per auction),error logs (DSP failures,timeouts),financial logs (pacing,budget,spend),and audit logs (campaign changes,user access).This torre...

End-to-End AI/Machine Learning Architecture: AI/ML 6 Layers

End-to-End AI/Machine Learning Architecture: AI/ML 6 Layers

250

The architectural design for a modern Machine Learning and AI application adopts a cloud-native,event-driven microservices pattern structured across six core layers to satisfy functional requirements such as automated data transformation,model versioning,a...

End-to-End Autonomous Vehicle ML Training Pipeline

End-to-End Autonomous Vehicle ML Training Pipeline

335

The end-to-end architectural design for the machine learning training pipeline begins at the Vehicle,where telemetry and sensor data are collected and routed into the Data Ingestion system.This ingested data is stored centrally in a Data Lake and subsequen...

AWS System Design Example - Instagram Architecture

AWS System Design Example - Instagram Architecture

1,922

These are the core features and functionalities that the platform *must* offer to its users.I.User Account Management:Registration / Sign-up:Users must be able to create an account using email address,phone number,or social media (e.g.Facebook).The system ...

Media sharing service architecture in AWS

Media sharing service architecture in AWS

10,371

Media sharing is one of the hottest markets on the Internet. Customers have a staggering appetite for placing photos and videos on social networking sites,and for sharing their media in custom online photo albums.The growing popularity of media sharing mea...

Ad Tech Multi-Tenancy: A Spectrum of Isolation Architectures on AWS

Ad Tech Multi-Tenancy: A Spectrum of Isolation Architectures on AWS

335

Ad tech platforms serve hundreds to thousands of customers simultaneously: advertisers,agencies,publishers,networks.Each customer (tenant) expects isolated data,isolated compute resources,and isolated audit trails.Multi-tenancy is the architectural choice ...

Predictable Ad Spend: Architecting Pacing at Hyperscale

Predictable Ad Spend: Architecting Pacing at Hyperscale

426

Pacing control ensures that a campaign's daily budget is spent evenly across a 24-hour window rather than burning out in the first few hours when user traffic peaks.In real-time bidding,milliseconds matter — yet a campaign manager wants predictable,steady...

Teraform from the scratch

1,359

Terraform for AWS plays a key role in infrastructure as code (IaC),allowing developers to define and manage their AWS resources using a declarative configuration language.This eliminates the need for manual configuration and promotes consistency,scalabilit...

Web front-end architecture in AWS for e-commerce website

368

AWS Elastic Beanstalk offers a significant benefit for deploying high-availability frontend apps by automatically scaling your application based on traffic demand,ensuring your app stays responsive and accessible even during traffic surges.It handles load ...

Time series processing architecture in AWS

Time series processing architecture in AWS

10,561

When data arrives as a succession of regular measurements,it is known as time series information.Processing of time series information poses systems scaling challenges that the elasticity of AWS services is uniquely positioned to address. This elasticity i...

Secure and Scalable LAMP Service Architecture in AWS

Secure and Scalable LAMP Service Architecture in AWS

10,664

LAMP is an acronym for Linux,Apache,MySQL,and PHP,which are all open-source software components commonly used together to build dynamic websites and web applications.Linux provides the operating system foundation,Apache serves as the web server to handle i...

Live streaming / Static Content & Media Service based on AWS

Live streaming / Static Content & Media Service based on AWS

10,659

Amazon's most outstanding video streaming solutions include Amazon CloudFront,a global content delivery network (CDN) for fast and reliable delivery of video content,Amazon Elastic Transcoder,which converts videos into various formats and resolutions for o...

Large scale messenger service architecture in AWS - Slack case study

Large scale messenger service architecture in AWS - Slack case study

11,674

Designing a large-scale messenger service requires considering several key factors: scalability to handle millions of users and messages,reliability to ensure consistent service availability,security to protect user privacy and data,performance to provide ...

Large scale computing based on huge data sets in AWS

Large scale computing based on huge data sets in AWS

10,614

AWS Elastic Block Storage (EBS) offers significant technical benefits for large-scale computing with huge datasets by providing persistent,block-level storage volumes that can be attached to EC2 instances.This allows for high I / O performance,low latency,...

Financial services grid computing in AWS

Financial services grid computing in AWS

10,356

Grid computing is a distributed computing paradigm that connects geographically dispersed computers,often heterogeneous and owned by different organizations,into a single virtual supercomputer.This allows users to access and utilize the combined processing...

File synchronization service architecture in AWS

File synchronization service architecture in AWS

10,382

Given the straightforward,stateless client-server architecture in which web services are viewed as resources and can be identified by their URLs,development teams are free to create file sharing and syncing applications for their departments,for enterprise...

Disaster recovery solution in AWS for local applications

Disaster recovery solution in AWS for local applications

11,182

Building a Disaster Recovery Center (DRC) in the cloud allows organizations to maintain business continuity during disruptions by providing a secure and resilient environment for critical applications and data.This approach offers cost-effective scalabilit...

Content Distribution for VOD streaming services in AWS

Content Distribution for VOD streaming services in AWS

10,361

The most common content distribution method in the OTT industry is Content Delivery Networks (CDNs).CDNs are vast networks of servers strategically located around the globe,caching content closer to users to reduce latency and improve streaming quality.Thi...

Checkout service architecture in AWS for e-commerce website

Checkout service architecture in AWS for e-commerce website

11,015

Checkout in e-commerce refers to the final stage of an online purchase process where a customer provides their shipping and payment information,reviews their order,and confirms the transaction.It's the point where the customer formally commits to buying th...