cheap tricks and the perils of machine learning
play

Cheap Tricks and the Perils of Machine Learning Percy Liang - PowerPoint PPT Presentation

Oct 05, 2023 •268 likes •1.16k views

Cheap Tricks and the Perils of Machine Learning Percy Liang Stanford / (Semantic Machines / Microsoft) NAACL Workshop on New Forms of Generalization June 5, 2018 [with Pranav Rajpurkar et al; 2016] Reading Comprehension (SQuAD) 1 2 [with

  1. Cheap Tricks and the Perils of Machine Learning Percy Liang Stanford / (Semantic Machines / Microsoft) NAACL Workshop on New Forms of Generalization — June 5, 2018

  2. [with Pranav Rajpurkar et al; 2016] Reading Comprehension (SQuAD) 1

  3. 2

  4. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] very influential 3

  5. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] 3

  6. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] troubling 3

  7. Humans versus machines 4

  8. Harder tests? We want multiple-choice questions that people can answer easily. But we also want to avoid as much as possible questions that can be answered using cheap tricks (aka heuristics). — Hector Levesque, 2013 5

  9. Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 6

  10. Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. The dog chased the cat , which ran up a tree. It waited at the bottom. 6

  11. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  12. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  13. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  14. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the top. 7

  15. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  16. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  17. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  18. [hat tip: Fernando Pereira] Winograd schema The dog chased the cat , which ran up a tree. It waited at the bottom. 8

  19. But what about... Commonsense knowledge Logical reasoning Linguistic phenomena Intuitive physics ... 9

  20. But what about... Commonsense knowledge Logical reasoning Linguistic phenomena Intuitive physics ... MACHINE LEARNING DOESN’T CARE 9

  21. Interpolation is insufficient ≈ Train Test 10

  22. Interpolation is insufficient ≈ Train Test Any expressive model with enough data will do the job 10

  23. Extrapolation is harder �≈ Train Test 11

  24. Extrapolation is harder �≈ Train Test 11

  25. Extrapolation is harder �≈ Train Test To extrapolate (and be robust), must get a more ”correct” model 11

  26. Outline Harder data Stronger models 12

  27. Adversarial evaluation of reading comprehension (EMNLP 2017) Robin Jia 13

  28. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] very influential 14

  29. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] 14

  30. [with Robin Jia; EMNLP 2017] Reading comprehension The way a teacher promotes the course they are teaching, the more the student will get out of the subject matter. The three most important aspects of teacher enthusiasm are enthusiasm about teaching, enthusiasm about the students, and enthusiasm about the subject matter. A teacher must enjoy teaching. If they do not enjoy what they are doing, the students will be able to tell. They also must enjoy being around their students. A teacher who cares for their students is going to help that individual succeed in their life in the future. The teacher also needs to be enthusiastic about the subject matter they are teaching. For example, a teacher talking about chemistry needs to enjoy the art of chemistry and show that to their students. A spark in the teacher may create a spark of excitement in the student as well. An enthusiastic teacher has the ability to be very influential in the young student ’s life. An unenthusiastic teacher can be troubling to a young student. What can an enthusiastic teacher be to a young student? r-net+ [MSR-A] troubling 14

  31. worksheets.codalab.org submit model to evaluate on hidden test set 15

  32. Results on SQuAD models Model Original F1 Adversarial F1 SLQA+ 88.6 64.2 r-net+ 88.5 63.4 ReasoNet-E 81.1 49.8 SEDT-E 80.1 46.5 BiDAF-E 80.0 46.9 Mnemonic-E 79.1 55.3 Ruminating 78.8 47.7 jNet 78.6 47.0 Mnemonic-S 78.5 56.0 ReasoNet-S 78.2 50.3 MPCM-S 77.0 50.0 16

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Beating the Perils of Non-Convexity: Machine Learning using Tensor Methods Anima Anandkumar ..

Beating the Perils of Non-Convexity: Machine Learning using Tensor Methods Anima Anandkumar ..

Beating the Perils of Non-Convexity: Machine Learning using Tensor Methods Anima Anandkumar .. Joint work with Majid Janzamin and Hanie Sedghi. U.C. Irvine Learning with Big Data Learning is finding needle in a haystack Learning with Big

726 views • 60 slides
Machine Learning Tricks Philipp Koehn 13 October 2020 Philipp Koehn Machine Translation:

Machine Learning Tricks Philipp Koehn 13 October 2020 Philipp Koehn Machine Translation:

Machine Learning Tricks Philipp Koehn 13 October 2020 Philipp Koehn Machine Translation: Machine Learning Tricks 13 October 2020 Machine Learning 1 Myth of machine learning given: real world examples automatically build model

928 views • 53 slides
Adversarial Machine Learning Daniel Lowd University of Oregon

Adversarial Machine Learning Daniel Lowd University of Oregon

Adversarial Machine Learning Daniel Lowd University of Oregon Example: Spam Filtering From: spammer@example.com 1. Cheap mortgage now!!! Feature Weights cheap = 1.0

316 views • 14 slides
An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning

An Exercise in An Exercise in Machine Learning Machine Learning http://www.cs.iastate.edu/~cs573x/bbsilab.html Machine Learning Software Preparing Data Building Classifiers Interpreting Results Machine Learning Software

496 views • 26 slides
Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by

Machine Learning By Alex Scarlatos What is Machine Learning? Machine Learning is the process by which computers can be trained through observation, rather than being explicitly programmed. Basically, a program is given input and adjusts its

556 views • 17 slides
Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning: Study of algorithms that improve their performance P at some task T

Machine Learning 10-601 Tom M. Mitchell Machine Learning Department Carnegie Mellon University January 12, 2015 Today: Readings: What is machine learning? The Discipline of ML Decision tree learning Mitchell, Chapter 3

872 views • 29 slides
Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning

GCT634/AI613: Musical Applications of Machine Learning (Fall 2020) Traditional Machine Learning: Unsupervised Learning Juhan Nam Traditional Machine Learning Pipeline in Classification Tasks A set of hand-designed audio features are selected

398 views • 26 slides
CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine

1/22/19 CS 335 Machine Learning What is Machine Learning? Dan Sheldon Spring 2019 What is Machine Learning? A Simple Task: Recognize Obama How do you program a computer to Recognize faces? Recommend movies? Decide which web

581 views • 5 slides
Businesses and Tax The Perils of Perception October 2015 Business and Tax Perils of

Businesses and Tax The Perils of Perception October 2015 Business and Tax Perils of

Businesses and Tax The Perils of Perception October 2015 Business and Tax Perils of Perception | October 2015 | FINAL | Public 1 Business contributes the largest proportion of tax of any group of tax payers (29%), yet people massively

270 views • 14 slides
Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their

Machine Learning Machine Learning: algorithms that use experience to improve their performance We use machine learning in situations where it is very challenging (or impossible) to define the rules by hand: e.g. face detection

1.04k views • 28 slides
Social Learning in edX ABSTRACT edX was born following two revolutions in technology: cheap,

Social Learning in edX ABSTRACT edX was born following two revolutions in technology: cheap,

Social Learning in edX ABSTRACT edX was born following two revolutions in technology: cheap, robust video streaming and the growth of on-line social, in a belief that this progress would allow us to create effective, at-scale digital learning

778 views • 45 slides
Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine

Introduction to Machine Learning COMPSCI 371D Machine Learning COMPSCI 371D Machine Learning Introduction to Machine Learning 1 / 12 Outline 1 Nearest Neighbor Prediction 2 Complexity Considerations 3 The Voronoi Diagram 4 Overfitting

564 views • 12 slides
1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

1 Why Study Machine Learning? Why Study Machine Learning? Cognitive Science The Time is Ripe

What is Learning? Herbert Simon: Learning is any process by which a system improves performance from CS 391L: Machine Learning experience. Introduction What is the task? Classification Problem solving / planning /

603 views • 6 slides
MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING

JACOPO DI IORIO, MATTEO FONTANA, DANIELE OCCHIUTO, CLAUDIA VOLPETTI MACHINE LEARNING, STATISTICAL LEARNING AND PARALLEL COMPUTING INTRODUCTION VS MACHINE LEARNING STATISTICAL LEARNING Separate evolutions, but with shared methodologies MERGE

565 views • 55 slides
Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine

Apache PredictionIO End-to-End Machine Learning Server with Apache Spark What is Machine Learning? What is Machine Learning? Examples of machine learning applications: Facial recognition software E-mail spam detectors Automated

302 views • 17 slides
Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning

Machine Learning 11 AI Slides (6e) c Lin Zuoquan@PKU 1998-2020 11 1 11 Machine Learning 11.1 Learning agents 11.2 Inductive learning 11.3 Deep learning 11.4 Statistical learning 11.5 Reinforcement learning 11.6 Transfer learning

1.7k views • 159 slides
Whats wrong with these sentences? I am anxious to meet you. Fred bit off more than he

Whats wrong with these sentences? I am anxious to meet you. Fred bit off more than he

Paragraph and Section Development Whats wrong with these sentences? I am anxious to meet you. Fred bit off more than he could chew. Paragraph and Section Development Avoid cliches. Hold your tongue Beat a dead horse March

211 views • 10 slides
Introduction Alessandro Moschitti Department of Computer Science and Information Engineering

Introduction Alessandro Moschitti Department of Computer Science and Information Engineering

MACHINE LEARNING Introduction Alessandro Moschitti Department of Computer Science and Information Engineering University of Trento Email: moschitti@disi.unitn.it Course Schedule - Revised 27 apr 9:30-12:30 Garda (Introduction to Machine

801 views • 68 slides
Selective Search for Object Recognition J.R.R. Uijlings 1,2 , K.E.A. van de Sande 2 , T.

Selective Search for Object Recognition J.R.R. Uijlings 1,2 , K.E.A. van de Sande 2 , T.

Selective Search for Object Recognition J.R.R. Uijlings 1,2 , K.E.A. van de Sande 2 , T. Gevers 2 , and A.W.M. Smeulders 2 1 University of Trento, Italy 2 University of Amsterdam, the Netherlands Technical Report 2012, submitted to IJCV

673 views • 31 slides
Welcome! Assistance Animals in Public Accommodations & Housing will begin at 2:00 p.m.

Welcome! Assistance Animals in Public Accommodations & Housing will begin at 2:00 p.m.

8/11/2014 Welcome! Assistance Animals in Public Accommodations & Housing will begin at 2:00 p.m. Eastern Time 1 Listening to the Webinar Online: Please make sure your computer speakers are turned on or your headphones are plugged in

686 views • 19 slides
Scaling Data Products Under Startup Constraints A Case Study of ML Bias Testing Scaling Data

Scaling Data Products Under Startup Constraints A Case Study of ML Bias Testing Scaling Data

Scaling Data Products Under Startup Constraints A Case Study of ML Bias Testing Scaling Data Products Under Startup Constraints A Case Study of ML Bias Testing Edwin Ong @edwin Co-Founder, TinyData Founded CastTV (acquired by Tribune)

1.01k views • 54 slides
Enthymemes as Rhetorical Resources Ellen Breitholtz and Robin Cooper Department of Philosophy,

Enthymemes as Rhetorical Resources Ellen Breitholtz and Robin Cooper Department of Philosophy,

Enthymemes as Rhetorical Resources Ellen Breitholtz and Robin Cooper Department of Philosophy, Linguistics and Theory of Science University of Gothenburg June 17th 2011 Consider the interpretation of rise in (1): (1) Cherrilyn: Yeah I mean

999 views • 89 slides
Primitive Types and Strings Variables, Values, and Expressions The Class String

Primitive Types and Strings Variables, Values, and Expressions The Class String

Primitive Types and Strings Variables, Values, and Expressions The Class String Reading: => Section 1.2 1 Variables and Values Variables are memory locations that store data such as numbers and letters. The data stored by

844 views • 62 slides
Distilling Collective Intelligence from Twitter Crowdsourcing and Human Computation Lecture 17

Distilling Collective Intelligence from Twitter Crowdsourcing and Human Computation Lecture 17

Distilling Collective Intelligence from Twitter Crowdsourcing and Human Computation Lecture 17 Instructor: Chris Callison-Burch TA: Ellie Pavlick Website: crowdsourcing-class.org Todays slides come courtesy of Miles Osborne and Benjamin

1.05k views • 67 slides

More recommend