[PPT] - In-Database Factorized Learning Dan Olteanu Joint work with M. PowerPoint Presentation

SLIDE 1

1 / 32

In-Database Factorized Learning

Dan Olteanu Joint work with

M. Schleich, J. Zavodny & FDB Team
M. Abo-Khamis, H. Ngo, X. Nguyen

http://www.cs.ox.ac.uk/projects/FDB/ Recent Trends in Knowledge Compilation Dagstuhl, Sept 2017

SLIDE 2

We Work on In-Database Analytics

In-database analytics = solve optimization problems inside the database engine. Why in-database analytics?

1. Bring analytics close to data

⇒ Save non-trivial export/import time

2. Large chunks of analytics code can be rewritten into database queries

⇒ Use scalable systems and low complexity for query processing

3. Used by LogicBlox retail-planning and forecasting applications

Unified in-database analytics solution for a host of optimization problems.

2 / 32

SLIDE 3

Problem Formulation

3 / 32

SLIDE 4

Problem Formulation

A typical machine learning task is to solve θ∗ := arg minθ J(θ), where J(θ) :=

(x,y)∈D

L (g(θ), h(x) , y) + Ω(θ). θ = (θ1, . . . , θp) ∈ Rp are the parameters of the learned model

4 / 32

SLIDE 5

Problem Formulation

A typical machine learning task is to solve θ∗ := arg minθ J(θ), where J(θ) :=

(x,y)∈D

L (g(θ), h(x) , y) + Ω(θ). θ = (θ1, . . . , θp) ∈ Rp are the parameters of the learned model D is the training dataset with features x and response y

◮ Typically, D is the result of a feature extraction query over a database. 4 / 32

SLIDE 6

Problem Formulation

A typical machine learning task is to solve θ∗ := arg minθ J(θ), where J(θ) :=

(x,y)∈D

L (g(θ), h(x) , y) + Ω(θ). θ = (θ1, . . . , θp) ∈ Rp are the parameters of the learned model D is the training dataset with features x and response y

◮ Typically, D is the result of a feature extraction query over a database.

L is a loss function, Ω is the regularizer

4 / 32

SLIDE 7

Problem Formulation

A typical machine learning task is to solve θ∗ := arg minθ J(θ), where J(θ) :=

(x,y)∈D

L (g(θ), h(x) , y) + Ω(θ). θ = (θ1, . . . , θp) ∈ Rp are the parameters of the learned model D is the training dataset with features x and response y

◮ Typically, D is the result of a feature extraction query over a database.

L is a loss function, Ω is the regularizer functions g : Rp → Rm and h : Rn → Rm for n numeric features (m > 0)

◮ g = (gj)j∈[m] is a vector of multivariate polynomials ◮ h = (hj)j∈[m] is a vector of multivariate monomials 4 / 32

SLIDE 8

Problem Formulation

A typical machine learning task is to solve θ∗ := arg minθ J(θ), where J(θ) :=

(x,y)∈D

L (g(θ), h(x) , y) + Ω(θ). θ = (θ1, . . . , θp) ∈ Rp are the parameters of the learned model D is the training dataset with features x and response y

◮ Typically, D is the result of a feature extraction query over a database.

L is a loss function, Ω is the regularizer functions g : Rp → Rm and h : Rn → Rm for n numeric features (m > 0)

◮ g = (gj)j∈[m] is a vector of multivariate polynomials ◮ h = (hj)j∈[m] is a vector of multivariate monomials

Example problems: ridge linear regression, degree-d polynomial regression, degree-d factorization machines; logistic regression, SVM; PCA.

4 / 32

SLIDE 9

Ridge Linear Regression

General problem formulation: J(θ) :=

(x,y)∈D

L (g(θ), h(x) , y) + Ω(θ). Under square loss L , ℓ2-regularization, data points x = (x0, x1, . . . , xn), p = n + 1 parameters θ = (θ0, . . . , θn),

◮ x0 = 1 corresponds to the bias parameter θ0

g and h identity functions g(θ) = θ and h(x) = x

◮ g(θ), h(x) = θ, x = n

k=0 θkxk

we obtain the following formulation for ridge linear regression: J(θ) := 1 2|D|

(x,y)∈D
n
k=0

θkxk − y 2 + λ 2 θ2

2 .

5 / 32

SLIDE 10

Rewriting the Objective Function J

We decouple the parameters θ from the data-dependent features x in J. We can rewrite the loss function J(θ) := 1 2|D|

(x,y)∈D
n
k=0

θkxk − y 2 + λ 2 θ2

2 .

as follows: J(θ) = 1 2θ⊤Σθ − θ, c + sY 2 + λ 2 θ2

2 , where

Σ = (σi,j)i,j∈[n], σi,j = 1 |D|

(x,y)∈D

xi · xj c = (ci)i∈[n], ci = 1 |D|

(x,y)∈D

y · xi sY = 1 |D|

(x,y)∈D

y 2.

6 / 32

SLIDE 11

Batch Gradient Descent for Parameter Computation

Repeatedly update θ in the direction of the gradient until convergence: θ := θ − α · ∇J(θ). Since J(θ) = 1 2θ⊤Σθ − θ, c + sY 2 + λ 2 θ2

2 ,

the gradient vector ∇J(θ) becomes: ∇J(θ) = Σθ − c + λθ.

7 / 32

SLIDE 12

Key Insights

The computation of the training dataset entails a high degree of redundancy, which can be avoided by factorized joins. Compressed lossless representations of query result that are: deterministic Decomposable Ordered Multi-Valued Diagrams Aggregates can be computed directly over factorized joins.

8 / 32

SLIDE 13

Factorization Example

9 / 32

SLIDE 14

Factorization Example

Orders (O for short) customer day dish Elise Monday burger Elise Friday burger Steve Friday hotdog Joe Friday hotdog Dish (D for short) dish item burger patty burger

nion

burger bun hotdog bun hotdog

nion

hotdog sausage Items (I for short) item price patty 6

nion

2 bun 2 sausage 4

Consider the join of the above relations:

O(customer, day, dish), D(dish, item), I(item, price) customer day dish item price Elise Monday burger patty 6 Elise Monday burger

nion

2 Elise Monday burger bun 2 Elise Friday burger patty 6 Elise Friday burger

nion

2 Elise Friday burger bun 2 . . . . . . . . . . . . . . .

10 / 32

SLIDE 15

Factorization Example

O(customer, day, dish), D(dish, item), I(item, price) customer day dish item price Elise Monday burger patty 6 Elise Monday burger

nion

2 Elise Monday burger bun 2 Elise Friday burger patty 6 Elise Friday burger

nion

2 Elise Friday burger bun 2 . . . . . . . . . . . . . . .

A relational algebra expression encoding the above query result is:

Elise × Monday × burger × patty × 6 ∪ Elise × Monday × burger ×

nion

× 2 ∪ Elise × Monday × burger × bun × 2 ∪ Elise × Friday × burger × patty × 6 ∪ Elise × Friday × burger ×

nion

× 2 ∪ Elise × Friday × burger × bun × 2 ∪ . . .