Its Not What Machines Can Learn, Its What We Cannot Teach ICML 2020 - - PowerPoint PPT Presentation

It’s Not What Machines Can Learn, It’s What We Cannot Teach

ICML 2020

Gal Yehuda, Moshe Gabel, Assaf Schuster

Applications of machine learning

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

2

Example : TSP

Given a graph, we feed it to a model which outputs whether a route with cost < C exists

3

GNN

YES NO

Prates, Avelar, Lemos, Lamb, Vardi, Learning to Solve NP-Complete Problems - A Graph Neural Network for Decision TSP ,AAAI 2019

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

The machine learning process

4

Generate Data Propose: architecture, features, embedding Train model Evaluate SUCCESS

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Current Data Generation

SotA ML methods are data hungry

• Need many labeled examples

Labeling training data is slow

• Need to solve TSP, check 3-SAT, etc.

Instead, data augmentation:

• Start with small labeled training set
• Apply label-preserving transformation

5

YES ? NO ?

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Our Main Result

When starting with NP-hard problem, any efficient data generation or augmentation provably results in easier subproblem. This creates a catch-22:

• Slow data generation à dataset too small
• Fast data generation à easier subproblem

6

slow (non-poly-time) data generation fast (poly-time) data generation or augmentation NP ∩ coNP NP-hard

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Case Study: Conjunctive Query Containment

Experiment on a case study, CQC. Used common data sampling + augmentation approach Model appears to learn well! Results on “real” space much lower.

• Up to 30% drop

7

10 20 30 40 50 60 70 80 90 100 Augmented Sampled Accuracy

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Takeaways

Efficient data generation results in easier subproblem when training. Can cause overestimation of accuracy when testing. Results in catch-22:

• small amounts of training data from right problem?
• or large amounts of training data from easier subproblem?

8 ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

9

Let’s dive deeper

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

What exactly did we show?

Let L be an NP-hard language The binary classification problem: is x ∈ L or not? Sampler for L : probabilistic algorithm that generates labeled instances Efficient Sampler for L : a sampler that runs in poly-time

10

Sampler

, YES , NO

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Result 1: All polynomial time samplers are incomplete

• There are infinitely many instances it cannot generate !

11

The original problem space The problem space, seen by efficient sampler poly-time sampler

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Result 2: Poly-time sampler yields easier subproblem

If 𝑇! is a polynomial time sampler for a language 𝑀, then the classification task over the instances 𝑇! generates is in NP ∩ coNP.

12

L The original problem was NP-hard poly-time sampler ( , YES/NO) Is in L? Resulting problem is NP ∩ coNP

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Meaning: efficient sampling does not preserve hardness

13

P NP-hard coNP NP NP-complete easier harder

Even if we started with an NP-hard problem, what’s left after an efficient sampling is an easier sub-problem

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Proof

14

NP = easy to verify that x ∈ L For all x, ∃𝑣 such that M(x,u) = 1 ⟺ x ∈ L coNP = easy to verify that x ∉ L For all x, ∃𝑣 such that M(x,u) = 1 ⟺ x∉ L

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Proof

15

If x was generated by an efficient sampler 𝑇!, we can use the randomness used by the sampler both as a membership certificate and a non-membership cetificate To show that x ∈ L , check if 𝑇!(u) outpus (x, YES) è L ∈ NP To show that x ∉ L, check if 𝑇!(u) outputs (x, NO) è L ∈ co-NP

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Result 3: It can get really bad…

We show an L such that:

• 1. Original L is NP-hard.
• 2. Output of any polynomial time sampler for L is trivial to classify:

the first bit of X is the label with high probability.

16 ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

poly-time sampler

L YES NO 1 sample x

It can get really bad…

17

Meaning: any learning algorithm trained

• n efficiently generated data ”thinks” it

has 100% accuracy, where in fact it learns nothing about the original problem.

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

P NP-hard coNP NP NP-complete easier harder

constant time

Case study: Conjunctive Query Containment

• A conjunctive query q over a dataset is a first order predicate of the

form:

• The task: given two queries q and p, are the results of q contained in

the results of p regardless of database they run on?

• This is an NP-complete problem.
• Implications on query optimization, cache management, and more.

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

18

Case study: CQC

19 ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

sample from phase transition Vampire theorem prover label using solver

N Y Y Y N N N

data augmentation

N Y Y Y N N N N Y Y Y N N N Y Y Y N N N

label preserving transformations

Case study: CQC

Proposed an architecture and trained it to high validation accuracy

20

0.0 2.5 5.0 7.5 10.0 12.5 15.0

Million samples

70 75 80 85 90 95

Accuracy (%)

0.2 0.3 0.4 0.5

Loss

Accuracy Loss

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

Case study: CQC

Evaluate

21

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

Accuracy

aug all-cqc µ(10, 8)

Test set 0.942 0.804 0.647

ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

30% accuracy drop

In Summary

• Can we use Machine Learning to

approximately solve NP-hard problems?

• Not enough to worry about the representation

power of the network. Also worry about the procedure used to generate the data.

• All poly-time data generators result in easier

sub-problems.

• And it may be very easy.
• We must be careful when we evaluate our

models.

22 ICML 2020

• G. Yehuda, M. Gabel, A. Schuster. It's Not What Machines Can Learn, It's What We Cannot Teach.

THANK YOU!

We will he bappy to discuss the work and answer questions. ygal@cs.technion.ac.il mgabel@cs.toronto.edu