Introduction to the Course Prof. Sameer Singh CS 295: STATISTICAL - - PowerPoint PPT Presentation

▶
introduction to the course
SMART_READER_LITE
LIVE PREVIEW

Introduction to the Course Prof. Sameer Singh CS 295: STATISTICAL - - PowerPoint PPT Presentation

Jun 23, 2023 335 likes •891 views

Introduction to the Course Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 10, 2017 Based on slides from Nathan Schneider, Mohit Bansal, Sebastian Riedel, Yejin Choi, and everyone else they copied from. About Me Academic

slide-1
SLIDE 1

Introduction to the Course

  • Prof. Sameer Singh

CS 295: STATISTICAL NLP WINTER 2017

January 10, 2017

Based on slides from Nathan Schneider, Mohit Bansal, Sebastian Riedel, Yejin Choi, and everyone else they copied from.

slide-2
SLIDE 2

About Me

CS 295: STATISTICAL NLP (WINTER 2017) 2

  • New Assistant Professor at UC Irvine! (2016 -)
  • Postdoc at University of Washington (2013 -)
  • PhD from University of Massachusetts, Amherst (2014)

Academic Positions

  • Natural Language Processing: information extraction, relation

extraction, entity linking and disambiguation, joint modeling

  • Machine Learning: interpretable ML, semi-supervised learning,

matrix/tensor factorization, probabilistic graphical models Research Interests http://sameersingh.org sameer@uci.edu

slide-3
SLIDE 3

Natural Language Processing

CS 295: STATISTICAL NLP (WINTER 2017) 3

Introduction to NLP Course Information Upcoming deadlines

slide-4
SLIDE 4

Natural Language Processing

CS 295: STATISTICAL NLP (WINTER 2017) 4

Introduction to NLP Course Information Upcoming deadlines

slide-5
SLIDE 5

Knowledge Representation

NLP Unstructured Ambiguous Lots and lots of it! Humans can read them, but … very slowly … can’t remember all … can’t answer questions “Knowledge” Structured Precise, Actionable Specific to the task Computers can use … quickly answer questions … memory is not a problem … don’t get tired

CS 295: STATISTICAL NLP (WINTER 2017) 5

slide-6
SLIDE 6

“Deep” understanding

CS 295: STATISTICAL NLP (WINTER 2017) 6

slide-7
SLIDE 7

Lots of Existing Applications

CS 295: STATISTICAL NLP (WINTER 2017) 7

slide-8
SLIDE 8

But a long long way to go…

CS 295: STATISTICAL NLP (WINTER 2017) 8

slide-9
SLIDE 9

Future Applications

CS 295: STATISTICAL NLP (WINTER 2017) 9

slide-10
SLIDE 10

Future Applications

Question Answering (instead of search) Science, by reading papers for you News Summarization Law, by reading past cases for you Healthcare, by

  • rganizing records

Assistive Technologies (dialog systems) Computational Social Sciences Digital Humanities (historical texts)

CS 295: STATISTICAL NLP (WINTER 2017) 10

slide-11
SLIDE 11

Turing’s test for Artificial Intelligence

Human or Computer?

CS 295: STATISTICAL NLP (WINTER 2017) 11

slide-12
SLIDE 12

Challenges in NLP

WHY ISN’T NLP SOLVED YET?

CS 295: STATISTICAL NLP (WINTER 2017) 12

slide-13
SLIDE 13

Three main challenges

CS 295: STATISTICAL NLP (WINTER 2017) 13

Ambiguity Sparsity Variation

slide-14
SLIDE 14

Three main challenges

CS 295: STATISTICAL NLP (WINTER 2017) 14

Ambiguity Sparsity Variation

slide-15
SLIDE 15

Language is Ambiguous

One tries to be as informative as one possibly can, and gives as much information as is needed, and no more.

  • Grice’s Maxim of Quantity

Corollary: The more you know, the less you need. Computers “know” very little.

CS 295: STATISTICAL NLP (WINTER 2017) 15

slide-16
SLIDE 16

Words have many meanings

Hershey’s Bars Protest

CS 295: STATISTICAL NLP (WINTER 2017) 16

slide-17
SLIDE 17

Words have many meanings

He knows you like your mother.

CS 295: STATISTICAL NLP (WINTER 2017) 17

slide-18
SLIDE 18

Attachment Ambiguities

Stolen painting found by tree.

CS 295: STATISTICAL NLP (WINTER 2017) 18

slide-19
SLIDE 19

How he got into my pajamas I'll never know.

  • Groucho Marx

Attachment Ambiguities

One morning I shot an elephant in my pajamas.

CS 295: STATISTICAL NLP (WINTER 2017) 19

slide-20
SLIDE 20

Attachment Ambiguities

She saw the man with the telescope.

CS 295: STATISTICAL NLP (WINTER 2017) 20

slide-21
SLIDE 21

And so on…

  • Enraged Cow Injures Farmer with Ax
  • Ban on Nude Dancing on Governor’s Desk
  • Teacher Strikes Idle Kids
  • Hospitals Are Sued by 7 Foot Doctors
  • Iraqi Head Seeks Arms
  • Kids Make Nutritious Snacks
  • Local HS Dropouts Cut in Half

CS 295: STATISTICAL NLP (WINTER 2017) 21

slide-22
SLIDE 22

My girlfriend and I met my lawyer for a drink,

Coreference Ambiguities

but she became ill and had to leave.

CS 295: STATISTICAL NLP (WINTER 2017) 22

slide-23
SLIDE 23

Coreference Ambiguities

The city councilmen refused the demonstrators a permit because they feared violence. The city councilmen refused the demonstrators a permit because they advocated violence. “Context” is important

CS 295: STATISTICAL NLP (WINTER 2017) 23

Winograd Schema: An Open Challenge for AI

slide-24
SLIDE 24

Coreference Ambiguities

CS 295: STATISTICAL NLP (WINTER 2017) 24

slide-25
SLIDE 25

Entity Types and Identities

Types

  • Washington, Georgia,

Clinton, Adams

  • John Deere, Williams,

Dow Jones, Thomas Cook

  • Princeton, Amazon,

Kingston Identities

  • Same Name:

Kevin Smith, Jamaica, Springfield

  • Multiple “Names”:

President, Obama, Chief, Bambam,…

“Context” is important

CS 295: STATISTICAL NLP (WINTER 2017) 25

slide-26
SLIDE 26

CS 295: STATISTICAL NLP (WINTER 2017) 26

Entity Types and Identities

Not easy even for humans

slide-27
SLIDE 27

Three main challenges

CS 295: STATISTICAL NLP (WINTER 2017) 27

Ambiguity Sparsity Variation

slide-28
SLIDE 28

Sparsity of Words

the

  • f

to and cornflakes mathematician s fuzziness jumbling pseudo-rapporteur lobby-ridden perfunctorily Lycketoft UNCITRAL H-0695 policyfor Commissioneris >1/3

  • ccur only once

CS 295: STATISTICAL NLP (WINTER 2017) 28

slide-29
SLIDE 29

Sparsity of Words

CS 295: STATISTICAL NLP (WINTER 2017) 29

slide-30
SLIDE 30

Rescaling the Axes

CS 295: STATISTICAL NLP (WINTER 2017) 30

Zipf’s Law Regardless of the size of the data, there will be many rare words.

slide-31
SLIDE 31

Not unique to English

CS 295: STATISTICAL NLP (WINTER 2017) 31

In a document in which each character has been chosen randomly from a uniform distribution of all letters (plus a space character), the "words" follow the general trend of Zipf's. (Try it at home!)

slide-32
SLIDE 32

Three main challenges

CS 295: STATISTICAL NLP (WINTER 2017) 32

Ambiguity Sparsity Variation

slide-33
SLIDE 33

Many ways to say something

CS 295: STATISTICAL NLP (WINTER 2017) 33

She gave the book to Tom vs. She gave Tom the book Some kids popped by vs. A few children visited Is that window still open? vs Please close the window

slide-34
SLIDE 34

Variations in Domains

ikr smh he asked fir yo last name so he can add u on fb lolololtw Its vanished trees, the trees that had made way for Gatsby’s house, had once pandered in whispers to the last and greatest of all human dreams; for a transitory enchanted moment man must have held his breath in the presence of this continent, compelled into an aesthetic contemplation he neither understood nor desired, face to face for the last time in history with something commensurate to his capacity for wonder.

CS 295: STATISTICAL NLP (WINTER 2017) 34

slide-35
SLIDE 35

Tools & Methods

HOW CAN WE GET COMPUTERS TO SOLVE THIS PROBLEM?

CS 295: STATISTICAL NLP (WINTER 2017) 35

slide-36
SLIDE 36

CS 295: STATISTICAL NLP (WINTER 2017) 36

Sanders was born in Brooklyn, to Dorothy and Eli Sanders.

NNP VBD VBD IN NNP TO NNP CC NNP NNP

Sanders was born in Brooklyn, to Dorothy and Eli Sanders.

Person Location Person Person Bernie.. Bernie Sanders...

  • Mrs. Sanders..

.. his mother .. his father Eli he the city Sentence Dependency Parsing, Part of speech tagging, Named entity recognition… Document Discourse analysis, Coreference, Sentiment analysis...

Bernie Sanders Eli Sanders Dorothy Sanders Brooklyn

birthplace childOf childOf spouse

Corpus Entity resolution, Entity linking, Relation extraction…

slide-37
SLIDE 37

Two Different Approaches

DIRECTLY USE LINGUISTICS Expensive, time-consuming... … but also, incomplete! MACHINE LEARNING! Automatically learn from data! … if the right data exists

“Every time I fire a linguist, my accuracy goes up.”

  • Frederick Jelinek

CS 295: STATISTICAL NLP (WINTER 2017) 37

slide-38
SLIDE 38

Example: Machine Translation

From https://medium.com/@ageitgey/machine-learning-is-fun-part-5-language-translation-with-deep-learning-and-the-magic-of-sequences-2ace0acca0aa

CS 295: STATISTICAL NLP (WINTER 2017) 38

slide-39
SLIDE 39

Example: Machine Translation

Step 1: Break into Chunks

CS 295: STATISTICAL NLP (WINTER 2017) 39

slide-40
SLIDE 40

Example: Machine Translation

Step 2: Translations for each chunk

CS 295: STATISTICAL NLP (WINTER 2017) 40

slide-41
SLIDE 41

Example: Machine Translation

Step 3: Generate all possible sequences In same order In different order Step 4: Find the most human sounding one 😖 😋

I want to go to the prettiest beach.

CS 295: STATISTICAL NLP (WINTER 2017) 41

slide-42
SLIDE 42

In summary…

Language to Knowledge

  • Lots of applications…
  • Made a lot of progress, but not done

It’s quite difficult

  • Varied, sparse, and lots of ambiguities
  • Context really matters

Machine Learning!

  • With enough data and math, we can do it
  • The future looks really exciting for NLP

CS 295: STATISTICAL NLP (WINTER 2017) 42

slide-43
SLIDE 43

Natural Language Processing

CS 295: STATISTICAL NLP (WINTER 2017) 43

Introduction to NLP Course Information Upcoming deadlines

slide-44
SLIDE 44

Course Logistics

CS 295: STATISTICAL NLP (WINTER 2017) 44

  • Room: ICS 180
  • Tues/Thursday 9:30-10:50
  • No holidays this quarter (Yay!)

Meetings

  • Zhengli Zhao, PhD student
  • Email: zhengliz@uci.edu
  • But, contact us only on Piazza

Reader

  • Room: DBH 4204
  • Tuesdays 1pm - 5pm (by appt only)
  • https://calendly.com/sameersingh/office-hours

Office Hours Course webpage: http://sameersingh.org/courses/statnlp/wi17/

slide-45
SLIDE 45

Learning Goals

CS 295: STATISTICAL NLP (WINTER 2017) 45

  • Familiarize you with NLP terms
  • Tasks: Sequence Tagging, …
  • Methods: Neural approaches, …
  • Applications: Question Answering, …
  • Solve any NLP problem intelligently!

Basics of NLP

  • Be able to read recent papers
  • Appreciate their motivation
  • Understand their approach
  • Evaluate their results
  • Can discuss ideas with NLP researchers!

Critical Analysis

  • Be able to define a novel problem
  • Study literature to identify overlap
  • Implement existing and new methods
  • Work in a team with researchers of different background
  • With little guidance, have an NLP research agenda!

Research Projects

slide-46
SLIDE 46

Topics (subject to change)

CS 295: STATISTICAL NLP (WINTER 2017) 46

  • Text Classification: discriminative, generative, semi-supervised
  • Word Vectors: vector semantics, dense embeddings, neural approaches

Words and Representations

  • Language Models: generative, discriminative, neural model
  • Sequence Modeling: Part of speech and named entities, HMMs, CRFs

Language and Sequence Modeling

  • Context free grammars, Probabilistic CFGs, constituent/dependency parsing
  • Recursive neural models, sequence to sequence mapping, neural parsing

Sentence Structure Modeling

  • Information Extraction: relations, coreference, entity linking, question answering
  • Text generation, machine translation, entailment, reading comprehension, dialogs

Applications and other topics

slide-47
SLIDE 47

Speech Recognition

CS 295: STATISTICAL NLP (WINTER 2017) 47

slide-48
SLIDE 48

Cognitive Sciences/ Psycho-linguistics

CS 295: STATISTICAL NLP (WINTER 2017) 48

slide-49
SLIDE 49

Grading

CS 295: STATISTICAL NLP (WINTER 2017) 49 Programming Homework 40% Course Project 30% Paper Summaries 15% Participation 15%

  • You get four grace days
  • Mention in the write-up
  • Across all assignments
  • Use everyone’s for projects
  • Full credit when used (no q asked)
  • 0 if you run out (no partial credit)

Late Submissions

  • All submissions through Canvas
  • All deadlines are available now
  • Will not be changing..
  • So start planning now

Assignments

slide-50
SLIDE 50

Programming Assignments

CS 295: STATISTICAL NLP (WINTER 2017) 50

  • Throughout the quarter

4 Programming Assignments

  • Should be pretty straightforward
  • Some skeleton Python code provided..
  • ..which you can ignore
  • Piazza for potential bugs, weird results, etc.

Source Code (Python)

  • Open-ended analysis of your approach
  • Plots, figures, tables, examples…
  • Think of it as a short research paper

Writing Up (PDF)

slide-51
SLIDE 51

Paper Summaries

CS 295: STATISTICAL NLP (WINTER 2017) 51

  • Due closer to the end of the quarter

3 Paper Summaries

  • Content Summary: what they proposed
  • Critical Analysis: what you liked/hated
  • Instructions on the webpage already

Summaries

  • Cover all kinds of topics
  • Randomly assigned to students
  • You may not understand them!
  • But still have to summarize…

Recent Conference Papers

slide-52
SLIDE 52

Group Projects

CS 295: STATISTICAL NLP (WINTER 2017) 52

  • Ideal team size is 3, and diverse!
  • Absolute maximum of 4
  • <3 if I approve (ongoing work)

Groups for the Project

  • More on projects in the next lecture..
  • Bigger the team, more ambitious the goal
  • Has to be novel in some way
  • At least “workshop-level”
  • Pitch and discuss ideas on Piazza

Scope of Work

  • First two reports are very short (~1 page)
  • Final report matters the most

Submit Four Reports

slide-53
SLIDE 53

Participation

CS 295: STATISTICAL NLP (WINTER 2017) 53

  • Propose project ideas
  • Ask/answer questions and issues
  • Provide feedback to Instructor and TA
  • Discuss readings and papers

Piazza participation

  • Attend all the classes!
  • Lectures should be discussions
  • Ask questions! Answer them!

Class participation

slide-54
SLIDE 54

Natural Language Processing

CS 295: STATISTICAL NLP (WINTER 2017) 54

Introduction to NLP Course Information Upcoming deadlines

slide-55
SLIDE 55

Upcoming…

CS 295: STATISTICAL NLP (WINTER 2017) 55

  • Check out course webpage
  • Check out Canvas, especially for deadlines
  • Sign up for Piazza

Misc.

  • Homework 1 is up!
  • Next two lectures will cover the topic
  • Sign up for the Kaggle account (@uci.edu email)
  • Due: January 26, 2017

Homework

  • Project pitch is due January 23, 2017!
  • Start assembling teams now! (use Piazza)
  • Start looking at papers, data, etc. for ideas

Project