Common Recommendation Service Architecture

Almost every large-scale recommendation service — TikTok, YouTube, Amazon, Netflix, Spotify — is built on the same architectural skeleton, regardless of which specific model it uses. Understanding this skeleton is more valuable than memorizing any single algorithm: once you know where each piece fits, the individual models (matrix factorization, two-tower, Wide & Deep, sequence models, and so on) are just interchangeable parts that slot into named stages.

The skeleton is a multi-stage funnel that turns a catalog of millions or billions of items into a short, ordered list for one user in tens of milliseconds. The defining pattern is two-stage retrieval then ranking: a cheap, high-recall stage narrows the catalog to a few hundred candidates, then an expensive, high-precision stage scores and orders that shortlist.

Around this funnel sit four supporting subsystems that make it work in production: a data pipeline, a feature platform, a training system, and a feedback loop.

Below is an improved AI pipeline similar to common recommendation service architecture.

I hope that you have some insight based on above pipeline.
 

Why Two Stages? (The Core Idea)

The reason for the split is computational. Suppose the catalog has N items and the precise ranking model costs c per item to score. Ranking the entire catalog per request costs O(N · c) — infeasible when N is in the hundreds of millions and the latency budget is ~100 ms.

The funnel breaks this into stages with very different cost/accuracy profiles:

• Candidate generation (retrieval): cheap per item, run over the whole catalog, optimized for recall. A model whose score is a simple vector similarity s(u, i) = u · v lets us precompute all item vectors and use an Approximate Nearest Neighbor (ANN) index to retrieve the top candidates in sub-linear time instead of scanning N items.
• Ranking: expensive per item, run over only the few hundred survivors, optimized for precision of the final ordering. Here we can afford a heavy model with rich features and full user–item feature interactions.

Formally, retrieval reduces the candidate set from N to k (where k ≈ 100–1000), so the expensive ranking model only pays O(k · c) — independent of catalog size. This separation of concerns is the single most important idea in production recommendation systems.

A typical third step, re-ranking, then adjusts the ranked list for objectives the pointwise ranker does not capture: diversity, freshness, de-duplication, and business or policy rules.

Requirements

To stand up a recommendation service you need:

• An event pipeline that captures user, item, and context events reliably and with low latency.
• A feature platform with both an online store (low-latency reads at serving time) and an offline store (large-scale reads for training), kept consistent to avoid train/serve skew.
• A retrieval system: an embedding model plus an ANN (Approximate Nearest Neighbor) index that is periodically rebuilt as items and models change.
• A ranking system: a feature-rich model served under a tight latency budget, with feature assembly fast enough to score hundreds of candidates per request.
• A training and orchestration system (batch, nearline, or online) to (re)train models on fresh data on a schedule.
• A feedback loop and monitoring: engagement logging, offline evaluation, online A/B testing, and bias correction for logged data.

Simplified Architecture

Reference Architecture

PRO
 

Reference Architecture (with Technology Stack)

The same blueprint mapped onto common production technologies — Kafka and Flink for ingestion, Redis plus Feast for features, TensorFlow/PyTorch for training, FAISS/ScaNN for the vector index, and Triton/TF Serving behind the API.

Stage-by-Stage Walkthrough

Data ingestion. Every interaction (impression, click, watch, purchase, skip) and every item/profile change is published as an event, typically to Apache Kafka. A stream processor (Apache Flink or Spark Streaming) computes near-real-time aggregates, while raw events land in a data lake (S3/HDFS) for batch processing and training.

Feature platform. Features are served from two stores kept in sync: an online store (Redis) for millisecond reads during serving, and an offline store (Hive/BigQuery) for generating training data. A feature framework such as Feast registers definitions so the same transformation runs in both places — preventing train/serve skew, one of the most common and damaging bugs in production ML.

Candidate generation (retrieval). One or more retrievers each propose candidates: a two-tower embedding model queried via ANN, plus complementary sources like co-visitation, trending/popularity, and recent-history rules. Their outputs are merged into a single candidate pool. The goal here is recall, not precise order.

Ranking. A heavy model (Wide & Deep, DeepFM, a deep ranking network, or an Multi-gate Mixture-of-Experts model for multiple objectives) scores each candidate using rich user, item, and context features. Modern systems are usually multi-objective, predicting several outcomes (click, watch time, satisfaction) and blending them.

Re-ranking. The ordered list is adjusted for whole-list properties the pointwise ranker ignores: diversity, freshness, de-duplication, fatigue/seen-item filtering, and business or policy constraints. Reinforcement-learning policies sometimes act here to optimize long-term engagement.

Feedback loop. Served impressions and the user’s responses are logged back to Kafka, closing the loop: they become training data for the next model refresh and the basis for evaluation and monitoring.

Signals & Features

The architecture is signal-agnostic, but the common families are: user features (IDs, demographics, long-term and session history embeddings), item features (IDs, content embeddings, category, creator, age/freshness), context features (device, time, location, query, position in feed), and interaction signals (clicks, watch fraction, dwell, likes, skips). The feature platform’s job is to make all of these available consistently offline (for training) and online (for serving) at the latency the funnel demands.

Training & Serving Considerations

Production systems blend three training cadences. Batch (offline) retraining on a daily/hourly schedule via Airflow-orchestrated Spark and TensorFlow/PyTorch jobs is the baseline. Nearline updates refresh embeddings or counters within minutes. Online (continuous) training updates parameters from the streaming feedback in near real time — the approach TikTok’s Monolith takes for maximum freshness.

On the serving side, the funnel’s latency budget is split across stages: retrieval must be sub-linear (hence ANN), and ranking is bounded by how many candidates k can be scored within the deadline — feature assembly from the online store is frequently the real bottleneck, not model inference. The ANN index must be rebuilt as the item tower retrains, and index staleness between rebuilds is a standing operational concern.

Evaluation

Each stage is judged differently. Retrieval is measured by recall@k — did the relevant items make it into the candidate pool? Ranking is measured by ordering metrics: NDCG (Normalized Discounted Cumulative Gain), MAP (Mean Average Precision), and, for click models, AUC (Area Under the ROC Curve) and LogLoss. End-to-end, the authoritative signals come from online A/B tests on engagement, watch time, and long-term retention. Because logged data is biased by what was previously shown, mature systems apply position- and selection-bias correction and use off-policy/counterfactual estimators before trusting offline numbers.

Strengths & Weaknesses

Strengths: the staged funnel makes precise ranking affordable at catalog scale; stages are modular, so each model family in this series can be swapped into the retrieval or ranking slot independently; the feature platform and feedback loop are reusable across products.

Weaknesses (common pitfalls): train/serve skew when offline and online features diverge; feedback loops and popularity bias that narrow recommendations over time; ANN index staleness; latency pressure that pushes feature assembly to the edge of the budget; and the substantial MLOps (Machine Learning Operations) burden of running multiple models, an index, and bias correction together.

How the Model Families Map onto This Architecture

This blueprint is where every other document in the series fits:

• Candidate generation / retrieval: Two-Tower (07), Matrix Factorization (03), Memory-Based CF (02), Content-Based Filtering (01), and Sequence Models (06) all serve as retrievers feeding the candidate pool.
• Ranking: Neural Collaborative Filtering (05) and Wide & Deep / DeepFM (08) are the canonical ranking-stage models.
• Re-ranking / policy: Reinforcement Learning (09) optimizes the long-term, whole-list objective.
• Hybrid (04) describes how multiple retrieval and ranking signals are combined — i.e., the funnel itself.
• Production systems TikTok Monolith (10) and YouTube (11) are full instantiations of this architecture, differing mainly in training freshness and multi-objective ranking.

Read those docs through this lens: each one is a deep dive into a single box in the diagrams above.