The December 2022 issue of IEEE Spectrum is here!

Close bar

The Little Hack That Could: The Story of Spotify’s “Discover Weekly” Recommendation Engine

How two software engineers pulled together machine learning tools to make one of Spotify's most popular offerings

3 min read
Screen shots of Spotify's Discover Weekly recommender system on a computer, a phone, and a tablet
Photo: Spotify

“Empower bottom-up innovation and amazing things will happen.” That’s what Spotify software engineer Edward Newett says. He was responsible for one of those amazing things: a way to help Spotify users discover new music called Discover Weekly. This tool launched about a year ago; it now has 40 million users and is helping to build the careers of new artists.

imgSpotify software engineer Edward Newett started building the Discover Weekly recommender system as a side project.Photo: Tekla Perry

Newett joined Spotify in 2013, initially working on a team developing a web page with personalized information, news about artists, album releases, and local concerts, along with a recommender system that offered suggestions of albums a user might find appealing. The recommendation feature, Newett recalled, seemed like a good idea, but wasn’t heavily used. “My hunch was that navigating to this page and looking at albums was too much work,” he said, speaking at the third annual @Scale conference held in San Jose last week. @Scale brings together engineers who build or maintain systems designed for vast numbers of users, including engineers from companies like Google, Airbnb, Dropbox, Netflix, and others.

Newett thought he could come up with a better way, and in 2014 convinced a colleague to help him hack together a prototype of a tool they called Discover Weekly. Discover Weekly would give users a personalized playlist of music they’d never listened to, designed to fit their musical tastes.

imgThe Discover Weekly engineers didn’t want to commit a lot of time to the project until they were convinced it was worth doing. So they changed Spotify’s traditional engineering approach to include more frequent user tests.Photo: Tekla Perry

To come up with the individualized lists, the two scrounged together a variety of machine learning tools used in other parts of the Spotify system. Their system looks at what the user is already listening to, and then find connections between those songs and artists and other songs and artists, crawling through user activity logs, playlists of other users, general news from around the web, and spectragrams of audio. It then filters the recommendations to eliminate music the user has already heard, and sends the individualized playlist to the user. To make it clear that this playlist was personalized, they decided to illustrate each with an image of the user, pulled from Facebook. And they determined that they would refresh the lists weekly, on Monday mornings.

To test Discover Weekly in early 2015, the two engineers quietly pushed their new tool out to the Spotify accounts of all company employees. “Everyone freaked out in a good way,” Newett said, saying things like “‘It’s as if my secret music twin put it together.’”

“We were feeling good, but we hadn’t tested it on [typical] users, so we then rolled it out to one percent of the user base,” he recalls, and again, he recalls, the reaction was awesome.

Then came the mid-2015 rollout to the the rest of Spotify’s customers, some 100 million active users around the world. “We had to refresh 100 million playlists every Sunday night, with about a terabyte of new data.”

Discover Weekly quickly became a habit for people, Newett reports, giving them something to look forward to on Monday mornings. Then, a few months after the mid-2015 launch, he says, the team had its “first production incident—it didn’t update.” Some users “went into blind rage or existential crisis.”

For Newett, that reaction validated the tool’s popularity.

Since then, the team moved the system from its own servers to Google’s Cloud Bigtable, and now runs its recommendation algorithms several days ahead of time, rather than trying to crunch through recommendations for its entire user base on Sunday for Monday release.

“This wasn’t a big company initiative,” Newett says, “just a team of passionate engineers who went about solving a problem we saw with the technology we had.”

The Conversation (0)

Why Functional Programming Should Be the Future of Software Development

It’s hard to learn, but your code will produce fewer nasty surprises

11 min read
A plate of spaghetti made from code
Shira Inbar

You’d expectthe longest and most costly phase in the lifecycle of a software product to be the initial development of the system, when all those great features are first imagined and then created. In fact, the hardest part comes later, during the maintenance phase. That’s when programmers pay the price for the shortcuts they took during development.

So why did they take shortcuts? Maybe they didn’t realize that they were cutting any corners. Only when their code was deployed and exercised by a lot of users did its hidden flaws come to light. And maybe the developers were rushed. Time-to-market pressures would almost guarantee that their software will contain more bugs than it would otherwise.

Keep Reading ↓Show less