Bandit algorithms address the cold-start and feedback-loop problems in recommender systems by modeling uncertainty and deliberately exploring underexposed items. The post covers three core algorithms — ε-greedy, Upper Confidence Bound (UCB), and Thompson Sampling — with real-world implementations from Spotify, Yahoo, Netflix, Doordash, Twitter, Alibaba, and Amazon. Key findings include: UCB and Thompson Sampling outperform ε-greedy; Thompson Sampling is more robust to delayed feedback; pessimistic initialization beats naive initialization; and semi-personalized (cluster-based) bandits can outperform fully personalized ones. The post also covers warm-starting strategies using hierarchical data or importance-weighted logged events, and off-policy evaluation via the replay method, noting that supervised PR-AUC on biased logged data can diverge from real-world CTR performance.
