Reinforcement Learning–Based Methods for CTR Improvement based on Recommendation System

Reinforcement Learning (RL) reframes recommendation as a sequential decision-making problem rather than a one-shot prediction. Instead of asking “what will the user click right now?”, it asks “what sequence of recommendations maximizes long-term reward — sustained engagement, satisfaction, retention?”

The recommender is an agent that observes the user’s state, takes an action (recommend item(s)), and receives a reward (the user’s response). It learns a policy that optimizes cumulative reward over a horizon, explicitly trading exploration (try new things to learn) against exploitation (serve known-good items). This directly addresses a blind spot of click-optimized models: maximizing immediate clicks can degrade long-term satisfaction (clickbait, filter bubbles).

How It Works (The Math)

The problem is modeled as a Markov Decision Process (MDP) (S, A, P, R, γ):

• State s — user representation: history, context, session features.
• Action a — the item (or slate of items) recommended.
• Reward r — engagement signal (watch time, return visit, satisfaction).
• Transition P(s' | s, a) — how the user state evolves after a recommendation.
• Discount γ ∈ [0,1) — how much future reward counts versus immediate.

The objective is to maximize expected discounted return:

J(π) = E_π [ Σ_{t=0}^{∞} γᵗ r_t ]

Value-based (Deep Q-Network / DQN style)

Learn an action-value function and act greedily/ε-greedily:

Q*(s, a) = E[ r + γ · max_{a'} Q*(s', a') ]

Policy-gradient (REINFORCE)

Directly optimize a parameterized policy π_θ(a | s):

∇_θ J = E_π [ ∇_θ log π_θ(a | s) · Aᵗ ]

where Aᵗ is the (advantage-corrected) return. YouTube’s top-K off-policy REINFORCE is a landmark production application, learning from logged data with importance-sampling correction because the data was collected under a different (old) policy than the one being learned.

The off-policy challenge

Recommenders cannot freely experiment on live users, so training mostly happens off-policy on logged data. Correcting the mismatch between the logging policy and the target policy (via importance weighting / propensity correction) is the central technical difficulty.

Requirements

• A reward definition that genuinely encodes long-term value — the hardest and most consequential design choice.
• A state representation of the user, usually produced by a sequence/embedding model.
• Logged interaction data with logging propensities for off-policy learning, or a simulator for safe pretraining.
• Off-policy correction machinery (importance sampling, propensity scores).
• Guardrails: exploration must be bounded so experimentation does not harm user experience; safety constraints prevent runaway optimization.
• Robust offline evaluation (off-policy/counterfactual estimators) since live trial-and-error is expensive and risky.

Simplified Architecture

Reference Architecture

Reference Architecture (with Technology Stack)

The same flow on concrete tooling — Flink for state features, Redis for the user-state store, Triton for policy serving, Ray RLlib/TF for off-policy training, and Spark for counterfactual evaluation.

Signals & Features

The state aggregates the same features used elsewhere (history embeddings, context, session signals). What distinguishes RL is the reward signal: it must capture long-horizon value — return visits, cumulative watch time, satisfaction surveys, retention — not just the immediate click. Poorly chosen rewards (e.g., raw watch time alone) invite reward gaming, so production systems blend multiple reward terms and add penalties for undesirable outcomes.

Training & Serving Considerations

Because online experimentation is costly and risky, training is predominantly off-policy on logged data with importance-sampling correction; simulators sometimes provide safe pretraining. Exploration is deliberately constrained so users are not subjected to too many bad recommendations. RL is frequently layered on top of a strong supervised system — supervised models propose candidates and an RL policy shapes long-term sequencing — rather than replacing it. This staging tames variance and keeps the system stable.

Evaluation

The defining difficulty is that offline accuracy metrics don’t capture long-term reward. Teams rely on off-policy / counterfactual estimators (importance-sampling, doubly-robust) to estimate a new policy’s value from logged data, then confirm with carefully scoped online A/B tests measuring retention and long-term engagement, not just session CTR (Click-Through Rate). Reward-gaming and distribution shift are monitored continuously.

Strengths & Weaknesses

Strengths: optimizes long-horizon objectives (retention, satisfaction) that myopic click models miss; adapts to evolving user state; principled handling of the exploration–exploitation trade-off.

Weaknesses: reward design is hard and mis-specification causes reward gaming / unintended optimization; off-policy training and offline evaluation are difficult; high variance and sample inefficiency; risky to experiment on live users; substantial engineering and theoretical complexity.

When to Use It

RL fits platforms where long-term engagement and retention are the real objective and where short-term-click optimization is known to backfire — large feeds and streaming services (YouTube’s REINFORCE work being the canonical example). It is best adopted after a solid supervised retrieval+ranking stack exists, added as a policy layer to optimize sequencing and long-term value, with strong off-policy evaluation and safety guardrails in place.