ML at Twitter: A Deep Dive into Twitters Timeline Cibele Montez - - PowerPoint PPT Presentation

▶
ml at twitter a deep dive into twitter s timeline
SMART_READER_LITE
LIVE PREVIEW

ML at Twitter: A Deep Dive into Twitters Timeline Cibele Montez - - PowerPoint PPT Presentation

Feb 27, 2024 147 likes •816 views

Twitter Cortex Twitter Timelines Quality ML at Twitter: A Deep Dive into Twitters Timeline Cibele Montez Halasz, Twitter Cortex Satanjeev Bano Banerjee, Twitter Timelines Quality OReilly AI Conference April 2019 Agenda 1 ML at

slide-1
SLIDE 1 Twitter Cortex Twitter Timelines Quality April 2019

ML at Twitter: A Deep Dive into Twitter’s Timeline

Cibele Montez Halasz, Twitter Cortex Satanjeev “Bano” Banerjee, Twitter Timelines Quality O’Reilly AI Conference

slide-2
SLIDE 2 2

1 2 3 4 5 6 ML at Twitter Timelines Product Modeling and Serving Pipeline Model Timelines on Tensorflow Lessons Learned and Conclusions

Agenda
slide-3
SLIDE 3 3

ML at Twitter

slide-4
SLIDE 4 4

Personalized Search

ML at Twitter
slide-5
SLIDE 5 5

Recommendations

ML at Twitter
slide-6
SLIDE 6 6

Health

ML at Twitter
slide-7
SLIDE 7 7

Ads

ML at Twitter
slide-8
SLIDE 8 ML at Twitter ML Feature Management ML Core Environment (TF based platform) ML Platform Tools ML Data Observability ML Modeling ML Extended

Cortex

8
slide-9
SLIDE 9 9

Timelines Product

slide-10
SLIDE 10 10

Tweets shown to users when they log in and ask for Tweets in the Home section of the app

Similar to Facebook News Feed, Instagram Home, etc.

Twitter “Timeline”

Timelines Product
slide-11
SLIDE 11 11

‘Reverse Chron’ Timeline

1m 5m 9m 13m 29m

1 minute old 9 minutes old 29 minutes old 5 minutes old 13 minutes old

Timelines Product
slide-12
SLIDE 12 12

Problems of Reverse Chron

  • The most recent Tweet may not be the most interesting Tweet
  • The most interesting Tweets for you may be
  • From hours ago
  • From people who tweet infrequently
  • From people you don’t even follow
  • Some Tweets may be bad: e.g. abusive, or near-duplicate content
Timelines Product
slide-13
SLIDE 13

Goal of Timelines Product

Show each user the Tweets they are most likely to be interested in

Timelines Product 13
slide-14
SLIDE 14 14

User interests —> User engagement

Timelines Product

Engagement == Likes / replies / retweets / ... Timelines Product’s Goal: Show users Tweets that they are more likely to engage with

slide-15
SLIDE 15 15

Relevance Ranked Timeline

1m 5m 9m 13m 29m

29 minutes old 13 minutes old 9 minutes old 5 minutes old 1 minutes old

Timelines Product
slide-16
SLIDE 16 16

ML problem: Given a Tweet and a user, predict the user’s probability of engaging with that Tweet

Timelines Product
slide-17
SLIDE 17 17

Modeling and Serving Pipeline

slide-18
SLIDE 18 Modeling and Serving Pipeline Optional caption copy here CLIENT 18
slide-19
SLIDE 19 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets 19
slide-20
SLIDE 20 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request 20
slide-21
SLIDE 21 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets 21
slide-22
SLIDE 22 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t 22
slide-23
SLIDE 23 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t Tweets with scores 23
slide-24
SLIDE 24 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t Tweets with scores sort, threshold, return Tweets to user 24
slide-25
SLIDE 25 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t TRAINING DATA Tweets with scores sort, threshold, return Tweets to user save as future training examples 25
slide-26
SLIDE 26 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t TRAINING DATA Tweets with scores sort, threshold, return Tweets to user save as future training examples 26
slide-27
SLIDE 27 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t TRAINING DATA Tweets with scores sort, threshold, return Tweets to user save as future training examples 27
slide-28
SLIDE 28 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t TRAINING DATA Tweets with scores sort, threshold, return Tweets to user save as future training examples save engagements as labels 28
slide-29
SLIDE 29 Modeling and Serving Pipeline TWEET SERVICE CLIENT request Tweets CANDIDATE TWEETS fetch Tweets request Tweets PREDICTION SERVICE s c
  • r
e T w e e t s r e q u e s t TRAINING DATA Tweets with scores sort, threshold, return Tweets to user save as future training examples save engagements as labels retrain engagement models 29
slide-30
SLIDE 30

126 M

Total Daily Monetizable Daily Active Users Source: Twitter 2018 Q4 Earnings Report Modeling and Serving Pipeline 30
slide-31
SLIDE 31 31

Modeling Engagements

Modeling and Serving Pipeline
slide-32
SLIDE 32 32

Modeling Engagements

Modeling and Serving Pipeline
slide-33
SLIDE 33 33

Modeling Engagements

Modeling and Serving Pipeline
slide-34
SLIDE 34 34

Modeling Engagements

Modeling and Serving Pipeline
slide-35
SLIDE 35 35

Modeling Engagements

Modeling and Serving Pipeline
slide-36
SLIDE 36 36

Modeling Engagements

Modeling and Serving Pipeline
slide-37
SLIDE 37 37

Modeling Engagements

Modeling and Serving Pipeline
slide-38
SLIDE 38

➔ Currently we combine the models with learned weights score = x × P(like) + y × P(retweet) + z × P(reply) + … ➔ Additional score adjustments ◆ Photo penalty ◆ Author diversity penalty ◆ Producer-side value boost ➔ Lots of experiments to tune these parameters

Combining Separate Engagement Scores per Tweet

38 Modeling and Serving Pipeline
slide-39
SLIDE 39 39

Predictive Features for Timelines Ranking

➔ Tweet features

#

➔ User’s network features ➔ User historical features ➔ Request features ➔ Text features

Modeling and Serving Pipeline
slide-40
SLIDE 40 40

Model

slide-41
SLIDE 41 41 The Model

Timelines Model

1

Percentile Discretizer

2 3

Sparse Layer Dense MLP

4

Isotonic Calibration
slide-42
SLIDE 42 The Model

Timelines Model: Percentile Discretization

Raw feature values Discretized feature values 42
slide-43
SLIDE 43 43 The Model

Timelines Model

Percentile Discretizer

1 2 3

Sparse Layer Dense MLP

4

Isotonic Calibration
slide-44
SLIDE 44 1 2 K ... V1 V2 Vn-2 Vn-1 V The Model

Timelines Model: Sparse Linear Layer

44
slide-45
SLIDE 45 45 The Model

Timelines Model

Percentile Discretizer

1 2 3

Sparse Layer Dense MLP

4

Isotonic Calibration
slide-46
SLIDE 46 46

Timelines Model

The Model
slide-47
SLIDE 47 47 The Model

Timelines Model

Percentile Discretizer

1 2

Sparse Layer Dense MLP

4

Probability Calibration

3

slide-48
SLIDE 48
  • Very few positive examples of likes, replies, retweets vs negative

examples (Tweets that were shown to the user, but the user did not like, reply, retweet, etc).

  • Solution:

○ Heavily downsample negative examples ○ Train without example weights ○ Then apply probability calibration in the end

The Model

Timelines Model: Skewed Dataset

48
slide-49
SLIDE 49 The Model

Sparse Linear Layer: Probability Calibration Sparse Linear Layer: Probability Calibration

Goal: Bring predicted CTR back in line with data CTR

49
slide-50
SLIDE 50 The Model

Sparse Linear Layer: Probability Calibration

0.8, 0.5, 0.6, 0.7, 0.8 1, 2 , 1, 2, 3 1, 0 , 1, 1, 1 predictions weights labels MLP (multilayer perceptron) 50
slide-51
SLIDE 51 The Model

Sparse Linear Layer: Probability Calibration

0.8, 0.5, 0.6, 0.7, 0.8 1, 2 , 1, 2, 3 1, 0 , 1, 1, 1 predictions weights labels MLP (multilayer perceptron) 0.5, 0.6, 0.7, 0.8, 0.8 2 , 1, 2, 3, 1 0 , 1, 1, 1, 1 sorted predictions sorted weights sorted labels 51
slide-52
SLIDE 52 The Model

Sparse Linear Layer: Probability Calibration

0.8, 0.5, 0.6, 0.7, 0.8 1, 2 , 1, 2, 3 1, 0 , 1, 1, 1 predictions weights labels MLP (multilayer perceptron) 0.5, 0.6, 0.7, 0.8, 0.8 2 , 1, 2, 3, 1 0 , 1, 1, 1, 1 sorted predictions sorted weights sorted labels 52
slide-53
SLIDE 53 The Model

Sparse Linear Layer: Probability Calibration

0.8, 0.5, 0.6, 0.7, 0.8 1, 2 , 1, 2, 3 1, 0 , 1, 1, 1 predictions weights labels MLP (multilayer perceptron) 0.5, 0.6, 0.7, 0.8, 0.8 2 , 1, 2, 3, 1 0 , 1, 1, 1, 1 binning sorted predictions sorted weights sorted labels 53
slide-54
SLIDE 54 The Model

Sparse Linear Layer: Probability Calibration

0.8, 0.5, 0.6, 0.7, 0.8 1, 2 , 1, 2, 3 1, 0 , 1, 1, 1 predictions weights labels MLP (multilayer perceptron) 0.5, 0.6, 0.7, 0.8, 0.8 2 , 1, 2, 3, 1 0 , 1, 1, 1, 1 sorted predictions sorted weights sorted labels binning Compute average prediction per bin 54
slide-55
SLIDE 55

Sparse Linear Layer: Probability Calibration: Closer Look

Source: Scikit-learn 55 The Model
slide-56
SLIDE 56 56

Timelines on Tensorflow

slide-57
SLIDE 57 Timelines on Tensorflow

Timelines Model and Tensorflow Hub

MLP (multilayer perceptron) Probability Calibration 57
slide-58
SLIDE 58 Timelines on Tensorflow

Tensorflow: Profiler

58
slide-59
SLIDE 59 Source: Tensorflow Medium Timelines on Tensorflow

Tensorflow: Tensorflow Model Analysis

59
slide-60
SLIDE 60 Timelines on Tensorflow

Tensorflow: Tensorboard

60
slide-61
SLIDE 61 61

Lessons Learned, Conclusions, Next Steps

slide-62
SLIDE 62 62

➔ Unifying teams under a single unified platform ➔ ML Automation is crucial for productionizing models ➔ Visualization tools are really helpful for ML practitioners

Lessons Learned: Platform

Lessons Learned / Conclusion / Next Steps
slide-63
SLIDE 63 63

➔ Choice of what to optimize for: ◆ Engagement? Time on app? Active days? ➔ Feedback loop – training on previously served data ➔ Automated and frequent model refreshes ➔ Speed of development: Feature engineering, new modeling techniques

Lessons Learned: Product

Lessons Learned / Conclusion / Next Steps
slide-64
SLIDE 64

➔ Goal of Timelines Product: Show users with most relevant Tweets ➔ Huge scale ◆ Infrastructure constraints ◆ Modeling Decisions ◆ Tensorflow offerings used by Timelines

Lessons Learned / Conclusion / Next Steps

Conclusion

64
slide-65
SLIDE 65

➔ Continuous Learning ➔ Model integration: automated model evaluation ➔ More model exploration for Timelines Quality in Tensorflow

Next Steps

65 Lessons Learned / Conclusion / Next Steps
slide-66
SLIDE 66

Thank you! Questions?

Twitter Cortex Twitter Timelines Quality

If you are interested in learning more about Twitter Cortex and Twitter Timelines Quality, please DM us on Twitter: @cibelemh @satanjeev

66