How Recommendation Algorithms Work

The Invisible Curators

Every time you open Netflix, Spotify, TikTok, or Amazon, a system decides within milliseconds what to show you. These systems collectively process more human attention than any editor, publisher, or programmer could consciously direct. They are not neutral pipes; they are active shapers of what you watch, read, buy, and believe.

Understanding recommendation algorithms doesn't require a computer science degree. It requires understanding three core ideas: what data goes in, what mathematical structure processes it, and — crucially — what the system is actually being told to optimise for.

Collaborative Filtering: You Are Your Behaviour

The oldest and still most widely used technique is collaborative filtering. The principle: find users whose past behaviour resembles yours, then recommend items those users liked that you haven't encountered yet.

Netflix's early recommendation system, Cinematch, was built largely on this idea. If you and 10,000 other users all loved the same obscure thriller from 2003, and those users collectively also loved a particular documentary series, the algorithm concludes that you might like the documentary too — without knowing anything about the content of either film.

This approach scales remarkably well. It requires no understanding of what items actually contain. It discovers patterns that would be impossible to encode manually. Its limitation is the "cold start" problem: new users have no history, so the system can't personalise for them. New items with no ratings can't be recommended at all.

Content-Based Filtering: Understanding the Item

Content-based filtering takes the opposite approach. Instead of watching what other users do, it analyses the properties of items you've already engaged with and recommends similar items.

Spotify's "Discover Weekly" was originally built using audio analysis: extracting tempo, key, instrumentation, and other acoustic features from tracks, then recommending songs with similar audio fingerprints to ones you'd played on repeat. Pandora's Music Genome Project takes this further — human analysts annotate songs across hundreds of attributes, and recommendations are driven by that structured metadata.

Content-based filtering avoids the cold start problem for items (a new song can be recommended immediately if its audio features match your taste profile). Its limitation is that it tends toward sameness — recommending more of what you already know you like, rather than expanding your horizons.

Reinforcement Learning: Optimising in Real Time

Modern recommendation systems at the largest scale — TikTok's For You Page, YouTube's homepage, Instagram's Reels tab — use variants of reinforcement learning. Rather than making recommendations based on static user profiles, they treat each session as an experiment.

The system shows you content, observes how you respond (did you watch the whole thing? did you pause and rewatch? did you share? did you comment? did you immediately scroll past?), updates its model of you in near-real-time, and adjusts subsequent recommendations accordingly. The reward signal is typically engagement: time spent on the platform, interactions generated, sessions started.

TikTok's algorithm is widely credited with being the most effective ever built for this purpose. Within a reported 35 minutes of a new user's first session, it can construct a highly accurate model of what will hold that specific person's attention. The speed comes from the richness of behavioural signals (watch time percentage is more informative than a like) and the speed of the feedback loop.

The Optimisation Problem Nobody Asked About

Here is the critical issue that is often lost in both techno-utopian and techno-panicking accounts of recommendation systems: these algorithms do exactly what they're told to do. The problem is that what they're told to optimise for is rarely what anyone actually wants.

Engagement is a proxy for satisfaction. But they diverge sharply in practice. Outrage is highly engaging. Content that triggers anxiety keeps users scrolling. Conspiracy theories, once encountered, generate strong engagement signals even if the user's underlying experience is negative. An algorithm optimising for watch time or clicks has no direct signal that the user is having a good experience — only that they're still there.

This isn't a bug; it's an alignment problem. The algorithm is aligned with a measured proxy (engagement) rather than the true objective (user satisfaction and wellbeing). Fixing it requires either better measurement — capturing signals like "did the user feel good about this session?" — or different objective functions that trade raw engagement for longer-term retention or user-reported satisfaction.

Diversity, Serendipity, and Filter Bubbles

Recommendation systems are inherently conservative in one sense: they model what you've done before and extrapolate. The "filter bubble" critique, popularised by Eli Pariser, argues that this creates personalised information environments where users are increasingly shown only content that confirms existing beliefs and tastes.

The empirical evidence on filter bubbles is actually more nuanced than the popular narrative suggests. A 2019 study found that algorithmic recommendation on Facebook showed users substantially more cross-cutting news than they would have encountered based purely on their own choices. But the psychological impact of the content shown matters as much as its ideological distribution.

Most platforms now build explicit diversity mechanisms into their recommendation systems — occasionally surfacing content outside your estimated taste profile specifically to keep users engaged in the long run and to avoid the recommendation death spiral of ever-narrowing content pools.