Embeddings @ Twitter Making ML easy with Embeddings !!! Sept 2018 - - PowerPoint PPT Presentation

▶
embeddings twitter
SMART_READER_LITE
LIVE PREVIEW

Embeddings @ Twitter Making ML easy with Embeddings !!! Sept 2018 - - PowerPoint PPT Presentation

Dec 17, 2022 321 likes •735 views

Embeddings @ Twitter Making ML easy with Embeddings !!! Sept 2018 Agenda 1 Team 2 Whats an Embedding ? 3 Why Embeddings ? 4 Embeddings Pipeline 5 Whats Next Agenda 1 Team 2 Whats an Embedding ? 3 Why Embeddings ? 4

slide-1
SLIDE 1

Sept 2018

Embeddings @ Twitter

Making ML easy with Embeddings !!!

slide-2
SLIDE 2

1 2 3 4 5 Team Whats an Embedding ? Why Embeddings ? Embeddings Pipeline What’s Next

Agenda

slide-3
SLIDE 3

Team Whats an Embedding ? Why Embeddings ? Embeddings Pipeline What’s Next 1 2 3 4 5

Agenda

slide-4
SLIDE 4

Section

Cortex

Team

slide-5
SLIDE 5

To unify and advance recommendation systems.

Team

slide-6
SLIDE 6

Recommendation Systems

Team

slide-7
SLIDE 7

Home Explore

Team

slide-8
SLIDE 8

Email

Team

slide-9
SLIDE 9

Notifications

Team

slide-10
SLIDE 10

Twitter

Team

slide-11
SLIDE 11

Team and Product Whats an Embedding ? Why Embeddings ? Embeddings Pipeline What’s Next 1 2 3 4 5

Agenda

slide-12
SLIDE 12

Discrete Continuous Space! word

twitter: [ 0.07, -0.001, -0.208 ] @jack: [ 0.427, 0.225, -0.082 ] SF: [ 0.541, 0.496, -0.362 ] #TwitterNBA: [ 0.414, 0.068, -0.196 ]

What is an Embedding ?

Model

slide-13
SLIDE 13
slide-14
SLIDE 14

Team and Product Whats an Embedding ? Why Embeddings ? Embeddings Pipeline What’s Next 1 2 3 4 5

Agenda

slide-15
SLIDE 15

Model Features Feature Compression

Reduced infrastructure cost and improved efficiency

Nearest Neighbor Search

Similarity search on the embedding space

Transfer Learning

Knowledge exchange between related domains while reducing training time and boosting performance

Why Embeddings ?

Lead to improved model performance when used as input features

slide-16
SLIDE 16

Model Features

  • ML practitioners typically use one-hot encoding

to represent categorical inputs ○ Incapable of encoding relationships ○ Sparsity issues make it less useful for large dimensions

  • Embeddings are outputs of ML models

○ Conserve relationships amongst entities ○ Compress the sparse input space into dense vectors

Why Embeddings ?

slide-17
SLIDE 17
slide-18
SLIDE 18

Model Features

Why Embeddings ?

slide-19
SLIDE 19

Feature Compression

Why Embeddings ?

slide-20
SLIDE 20

Feature Compression

Why Embeddings ?

slide-21
SLIDE 21

Feature Compression

Why Embeddings ?

  • Generate embeddings from a sub-network offline
  • Update at the same frequency as the raw features
slide-22
SLIDE 22

Feature Compression

Why Embeddings ?

slide-23
SLIDE 23

Nearest Neighbor Search

Why Embeddings ?

slide-24
SLIDE 24

Nearest Neighbor Search

Why Embeddings ?

slide-25
SLIDE 25

Nearest Neighbor Search

Why Embeddings ?

  • Essential component for Candidate Generation

pipelines ○ Co-embed users and items ○ Given a user, lookup neighbors ○ Use approximate methods to scale

  • Finds application in many other areas
slide-26
SLIDE 26

Transfer Learning

Why Embeddings ?

  • Model trained for one task is used in another

○ Typically by initializing network weights and fine tuning

  • Very attractive from a business point of view

○ Reduced development time ○ Cross domain information sharing

slide-27
SLIDE 27

Team and Product Whats an Embedding ? Why Embeddings ? Embeddings Pipeline What’s Next 1 2 3 4 5

Agenda

slide-28
SLIDE 28

Quality and Relevance Creation & consumption with ease Sharing & discoverability

Enable adapting to evolving data distributions

  • ver time

If applicable the learnt embeddings should be of value across product ML models Enable teams to learn embeddings at scale using the appropriate algorithm Enable teams to consume embeddings at scale

Goals

Embedding pipeline

Enable cross team collaboration Improvements/learning in

  • ne domain can drive

improvements elsewhere

slide-29
SLIDE 29
slide-30
SLIDE 30
slide-31
SLIDE 31

Item Selection & Data Preprocessing

Embedding pipeline

  • Identify the set of entities to learn embeddings for
  • Assemble dataset that represents the relationships

between these entities ○ Data representation defined by the learning algorithm

slide-32
SLIDE 32

Model Fitting

Embedding pipeline

  • Fit a model on the collected data

○ Use pre-built algorithms ○ Option to plug in a custom algorithm

slide-33
SLIDE 33

Benchmarking

Embedding pipeline

  • Developed a variety of standard benchmarking

tasks for each type of embedding ○ User Topic Prediction: Predictive performance of a logistic regression model learnt on the users embedding.

slide-34
SLIDE 34

Benchmarking

Embedding pipeline

  • Developed a variety of standard benchmarking

tasks for each type of embedding ○ User metadata prediction : Predictive performance of a logistic regression model learnt on the users embedding.

slide-35
SLIDE 35

Benchmarking

Embedding pipeline

  • Developed a variety of standard benchmarking

tasks for each type of embedding ○ User Follow Jaccard: Jaccard index of the users’ embedding similarity and their follow sets'

slide-36
SLIDE 36

Feature Store

Embedding pipeline

  • Publish embeddings to the "feature store",

Twitter's shared feature repository

  • Enables ML teams throughout Twitter to easily

discover, access, and utilize freshly trained embeddings. ○ Easy offline & online access ○ Discovery through UX

slide-37
SLIDE 37

Team and Product Whats an Embedding ? Why Embeddings ? Embeddings Pipeline What’s Next 1 2 3 4 5

Agenda

slide-38
SLIDE 38

Whats Next ?

  • New embedding learning algorithms
  • Increasing number of datasets available as embeddings
  • Large scale approximate nearest neighbor (ANN) solution
  • Further exploration with embeddings as means for feature compression
slide-39
SLIDE 39

@tayal_abhishek

Thank you

September, 2018

slide-40
SLIDE 40

Section Abhishek Tayal @tayal_abhishek

We are Hiring !!! #TwitterCortex #MLX

00 09 Sep 2018 5k 10k