Reddicommend Content recommendations for Reddit Motivation Reddit - PowerPoint PPT Presentation

Reddicommend Content recommendations for Reddit

Motivation ● Reddit is the fourth-most visited site in the US (8th globally) ● The ability of users to find content relevant to their interests is important to its utility as a content-aggregating site ● Reddicommend is an engine for personalized subreddit recommendations

Pipeline

Implementation Collaborative filtering using similarity measure based on the Pearson correlation: subreddit 3 subreddit 2 subreddit 1 Subreddit 1 2 3 4 5 r r r r r correlation e e e e e user 1 s s s s s u u u u u matrix subreddit 1 user 2 subreddit 2 user 3 subreddit 3 user 4 user 5

Scalable matrix multiplication in Spark ● Managing time complexity ○ Dense-matrix multiplication is O(n 3 ) (for square matrices) ■ > 1 million users per batch ■ > 100,000 subreddits

M N P Scalable matrix 3. 2. 0. 0. 1. multiplication in Spark 0. 5. x = 1. 0. 0. 0. 1. 0. 1. 0. 1. M: [MatrixEntry(0, 0, 3), MatrixEntry(0, 1, 2), MatrixEntry(1, 0, 1)] N: [MatrixEntry(0, 1, 1), MatrixEntry(1, 1, 1), MatrixEntry(2, 1, 1)]

M N P Scalable matrix 1. 2. 0. 0. 1. multiplication in Spark 0. 3. x = 0. 0. 0. 0. 1. 0. 0. 0. 1. [(0, (0, 1)), (1, (0, 2)] M: [MatrixEntry(0, 0, 1), MatrixEntry(0, 1, 2)] N: [MatrixEntry(0, 1, 1), MatrixEntry(1, 1, 1), MatrixEntry(2, 1, 1)] [(0, (1, 1)), (1, (1, 1)), (2, (1, 1))]

M N P Scalable matrix 1. 2. 0. 0. 1. multiplication in Spark 0. 3. x = 0. 0. 0. 0. 1. 0. 0. 0. 1. [(0, (0, 1)), (1, (0, 2)] [(0, ((0, 1), (1, 1))), (1, ((0, 2), (1, 1)))] [(0, (1, 1)), (1, (1, 1)), (2, (1, 1))]

M N P Scalable matrix 1. 2. 0. 0. 1. multiplication in Spark 0. 3. x = 0. 0. 0. 0. 1. 0. 0. 0. 1. [(0, ((0, 1), (1, 1))), (1, ((0, 2), (1, 1)))] [((0, 1), 1), ((0, 1), 2)]

M N P Scalable matrix 1. 2. 0. 0. 1. multiplication in Spark 0. 3. x = 0. 0. 0. 0. 1. 0. 0. 0. 1. [((0, 1), 3)] [(0, ((0, 1), (1, 1))), (1, ((0, 2), (1, 1)))] [((0, 1), 1), ((0, 1), 2)]

M N P Scalable matrix 1. 2. 0. 0. 1. multiplication in Spark 0. 3. x = 0. 0. 0. 0. 1. 0. 0. 0. 1. [((0, 1), 3)] [(0, ((0, 1), (1, 1))), (1, ((0, 2), (1, 1)))] [((0, 1), 1), ((0, 1), 2)] [MatrixEntry(0, 1, 3)]

Demo reddicommend.ddns.net

Oliver Hoidn ● B.S. in Physics, Harvey Mudd College ● PhD in Physics, University of Washington Past and current interests: ● Hiking ● Violin ● Giant pumpkin cultivation ● Lisp and functional programming