Memory-Based Collaborative Filtering for Recommendation Service

Memory-Based Collaborative Filtering (CF) makes recommendations from patterns of agreement across many users, without using any item content features. It is “memory-based” because it operates directly on the stored user–item interaction matrix at prediction time, computing similarities on demand rather than learning a compact parametric model.

There are two complementary formulations:

• User-based CF — find users with taste similar to the target user, then recommend what those neighbors liked.
• Item-based CF — recommend items that are frequently co-consumed with items the target user already likes. This is the classic “customers who bought X also bought Y” approach popularized by Amazon.

How It Works (The Math)

Let R be the user–item rating (or interaction) matrix, where r_{u,i} is user u’s rating of item i.

User-based prediction

First compute similarity between users. Pearson correlation over their co-rated items I_{uv}:

sim(u, v) = Σ_{i ∈ I_{uv}} (r_{u,i} − r̄_u)(r_{v,i} − r̄_v) / ( √Σ (r_{u,i} − r̄_u)² · √Σ (r_{v,i} − r̄_v)² )

Then predict the target rating as a similarity-weighted, mean-centered average over the neighborhood N(u):

r̂_{u,i} = r̄_u + ( Σ_{v ∈ N(u)} sim(u,v) · (r_{v,i} − r̄_v) ) / ( Σ_{v ∈ N(u)} |sim(u,v)| )

Item-based prediction

Compute item–item similarity, commonly adjusted cosine (centered on user means to remove rating bias):

sim(i, j) = Σ_{u ∈ U_{ij}} (r_{u,i} − r̄_u)(r_{u,j} − r̄_u) / ( √Σ (r_{u,i} − r̄_u)² · √Σ (r_{u,j} − r̄_u)² )

Predict from the items the user has already rated:

r̂_{u,i} = ( Σ_{j ∈ rated(u)} sim(i,j) · r_{u,j} ) / ( Σ_{j ∈ rated(u)} |sim(i,j)| )

Item-based CF is usually preferred in production because item–item similarities are more stable over time than user–user similarities and can be precomputed offline.

Requirements

• A user–item interaction matrix with sufficient density. Memory-based CF degrades sharply on extreme sparsity.
• A similarity computation and storage layer. For item-based CF this is a precomputed item–item similarity matrix (often top-N neighbors per item to bound size).
• Overlap in co-rated items/users, otherwise similarities are undefined or unreliable.
• No item features required — this is the defining advantage over content-based methods.
• Strategies for sparsity and scale: neighborhood size limits, significance weighting, and shrinkage to damp similarities estimated from few co-ratings.

Simplified Architecture

Reference Architecture

Reference Architecture (with Technology Stack)

The same flow on concrete tooling — Kafka for events, Spark for the nightly similarity job, and Redis (or Cassandra) for the precomputed item–item neighbor store.

Signals & Features

Memory-based CF consumes only interaction signals: explicit ratings, or implicit signals such as clicks, purchases, watches, add-to-cart, and dwell. Implicit feedback is treated as positive-only with confidence weighting. No item or user metadata is required, which is both the method’s strength (no feature engineering) and its limitation (it cannot reason about why items are similar).

Training & Serving Considerations

There is no parametric “training” in the gradient-descent sense. The expensive step is computing the similarity matrix, which is O(n²) in the naïve case over n items or users — the principal scalability bottleneck. Production systems mitigate this with sampling, locality-sensitive hashing (LSH), sparse co-occurrence counting, and top-N truncation. Item-based similarities are precomputed in nightly/periodic batch jobs; serving is then a fast sparse lookup and weighted sum.

Evaluation

Rating-prediction variants are scored with RMSE (Root Mean Square Error) / MAE (Mean Absolute Error); top-N recommendation is evaluated with precision@k, recall@k, MAP (Mean Average Precision), and NDCG (Normalized Discounted Cumulative Gain). Watch coverage and cold-item exposure closely, since memory-based CF systematically underexposes new and long-tail items (the popularity bias and cold-start problem).

Strengths & Weaknesses

Strengths: simple, intuitive, and explainable (“because you liked X”); requires no item features; item-based variant precomputes well and serves fast.

Weaknesses: scales poorly on huge user/item sets due to O(n²) similarity computation; sensitive to sparsity — few co-ratings make similarities unreliable; suffers cold-start for new users and new items (no interactions, no neighbors); prone to popularity bias that buries the long tail.

When to Use It

Memory-based CF is a strong baseline for small-to-medium catalogs with reasonable interaction density, for e-commerce “related items” modules (item-based CF), and as an interpretable benchmark to beat before investing in matrix factorization or deep models. On web-scale, sparse data it is typically superseded by model-based CF and deep retrieval, though item-based co-occurrence still appears as one signal inside larger hybrid systems.