Keetan Rutledge    About    Projects    Resume

Building a Recommendation System for a Leading Fitness Company

I built a recommender system for a leading fitness company that is currently being built into production on the company’s app. The app allows users to view health and wellness programs, workout videos, health articles, and other content. My recommender system recommends workout videos to users based on their viewing history. The company’s leadership and I decided to work to increase user engagement through video recommendations.

This project presented some key challenges. For one, the in-app video player didn’t record explicit feedback such as user ratings. Instead I had only video views and app action history, which are measures of implicit feedback. Also, video views were only granular at the month-level. In other words, I could see that a given user watched a video 12 times in January, but not which days or times it was watched. Finally, the user history was sparse, meaning each user only watched a handful of different videos. This means there is not much overlap between users, which will be relevant later for collaborative filtering.

In order to achieve the goal of increasing user engagement, our plan was:

  • For users who had not yet watched videos, recommend the videos that most drive engagement.
  • For users who had watched videos, provide personalized recommendations.

Which Videos Drive Engagement?

For each video, I looked at all the users who watched that video in one month, say January, and counted all of the actions those users did the next month, February. This is a measure of next-month engagement — it measures how engaged a user will tend to be given that they watched a certain video.

Here are each of the workout videos with their per-capita next-month engagement.

Looking at per-capita actions reveals the most engaging videos per-user. Yoga Basics 1 has the most next-month engagement.

Here are the view counts for the top 10 most-watched videos next their per-capita next-month engagement.

There is overlap between the most popular videos and the ones with the most next-month engagement, however there are some differences between them. For example, Athletic Stretch is the second most-popular video but only the seventh most-engaging. Whereas Yoga Basics 1 is the most engaging but only the fifth most-popular. This suggests that in order to drive engagement, it may be more effective to show users Yoga Basics 1.

I predicted user engagement in the second month from user video watching history in the first month. Not only did I want to predict engagement, but I also wanted insight into which videos are important predictors. I used a random forest model. Random forests build a large number of random decision trees and find which splits in the trees give the most predictive power. This indicates which predictors are the most important.

Here is a plot of the importance of each video, as measured by Gini Impurity:

It looks like videos that are Part 1 in a series, and standalone videos tend to be the most important predictors of engagement. In other words, if we are trying to predict user engagement, it is useful to know whether a user has watched Part 1 or standalone videos.

Generating Personalized Recommendations with Collaborative Filtering

Next, I generated personalized recommendations for users who had a video-watching history. In order to do so, I needed to measure the similarity between each pair of users based on their video-watching histories. I had no explicit feedback from the users — such as video ratings. Instead I had only implicit feedback — the number of times a user viewed a video. To predict user similarity, I used a metric called cosine similarity.

If we imagine a given user’s video watching history as a vector, then the cosine similarity between two users is the cosine of the angle between their vectors. For two users, A, and B, their cosine similarity is defined as:

After computing the similarity between all users, I imputed, for all users and all videos, the likelihood of a given user watching a given video. Then, for each user, I ordered the videos by the likelihood that they would watch them. The end result is a list of each user and the videos that they are most likely to watch. Here is an example of what a recommendation looks like for one user:

##   member_id                video_name score
## 1         1 Foundation Cardio Blast 1  0.93

Validation

The results of this recommender system are currently being built into the company’s app. In order to determine the effectiveness of the recommender system, we need to compare them against something. I outlined an A/B test to compare these recommendations to the status-quo. We are measuring overall user actions per week in the app, as well as specific actions of interest, such as video views and shares of app content. We are also comparing the effectiveness of different versions of the recommender. I’ll update this post with results of the A/B test.