cs325 artificial intelligence natural language processing
play

CS325 Artificial Intelligence Natural Language Processing I (Ch. 22) - PowerPoint PPT Presentation

Jul 02, 2023 •165 likes •1.3k views

CS325 Artificial Intelligence Natural Language Processing I (Ch. 22) Dr. Cengiz Gnay, Emory Univ. Spring 2013 Gnay Natural Language Processing I (Ch. 22) Spring 2013 1 / 30 AI in Natural Language Processing (NLP) Whats NLP? Gnay

  1. Remember Bag of Words? P ( Hello ) = 2 5 P ( I ) = 1 5 = P ( Will ) = P ( Say ) Günay Natural Language Processing I (Ch. 22) Spring 2013 7 / 30

  2. Remember Bag of Words? P ( Hello ) = 2 5 P ( I ) = 1 5 = P ( Will ) = P ( Say ) Words are independent? Günay Natural Language Processing I (Ch. 22) Spring 2013 7 / 30

  3. Remember Bag of Words? P ( Hello ) = 2 5 P ( I ) = 1 5 = P ( Will ) = P ( Say ) Words are independent? Called unigram or 1 -gram : Günay Natural Language Processing I (Ch. 22) Spring 2013 7 / 30

  4. Remember Bag of Words? P ( Hello ) = 2 5 P ( I ) = 1 5 = P ( Will ) = P ( Say ) Words are independent? Called unigram or 1 -gram : � P ( w 1 , w 2 , . . . , w n ) = P ( w i ) i Günay Natural Language Processing I (Ch. 22) Spring 2013 7 / 30

  5. Can we get more from Bayes? Distinguish between: “I will say hello” “I hello say will” Günay Natural Language Processing I (Ch. 22) Spring 2013 8 / 30

  6. Can we get more from Bayes? Distinguish between: “I will say hello” “I hello say will” P (” hello ” | ” I will say ”) > < P (” will ” | ” I hello say ”) Günay Natural Language Processing I (Ch. 22) Spring 2013 8 / 30

  7. Can we get more from Bayes? Distinguish between: “I will say hello” “I hello say will” P (” hello ” | ” I will say ”) > P (” will ” | ” I hello say ”) Günay Natural Language Processing I (Ch. 22) Spring 2013 8 / 30

  8. Can we get more from Bayes? Distinguish between: “I will say hello” “I hello say will” P (” hello ” | ” I will say ”) > P (” will ” | ” I hello say ”) Words dependent on previous words: called N -gram Günay Natural Language Processing I (Ch. 22) Spring 2013 8 / 30

  9. Can we get more from Bayes? Distinguish between: “I will say hello” “I hello say will” P (” hello ” | ” I will say ”) > P (” will ” | ” I hello say ”) Words dependent on previous words: called N -gram P ( w 1 , w 2 , . . . , w n ) = P ( w 1 : n ) � = P ( w i | w 1 :( i − 1 ) ) i Günay Natural Language Processing I (Ch. 22) Spring 2013 8 / 30

  10. Must Remember All Words That Came Before? P (” 1752 ” | ” Thomas Bayes . . . ”) =? Günay Natural Language Processing I (Ch. 22) Spring 2013 9 / 30

  11. Must Remember All Words That Came Before? P (” 1752 ” | ” Thomas Bayes . . . ”) =? Markov assumption: Only remember last N words: N -gram. Günay Natural Language Processing I (Ch. 22) Spring 2013 9 / 30

  12. Must Remember All Words That Came Before? P (” 1752 ” | ” Thomas Bayes . . . ”) =? Markov assumption: Only remember last N words: N -gram. k � P ( w 1 : k ) = P ( w i | w ( i − N ):( i − 1 ) ) i Günay Natural Language Processing I (Ch. 22) Spring 2013 9 / 30

  13. Let’s Read Shakespeare. . . In Unigram Unigram=1-gram Günay Natural Language Processing I (Ch. 22) Spring 2013 10 / 30

  14. Shakespeare In Bigram N = 2: bigram Günay Natural Language Processing I (Ch. 22) Spring 2013 11 / 30

  15. Shakespeare In Trigram N = 3: trigram Günay Natural Language Processing I (Ch. 22) Spring 2013 12 / 30

  16. Shakespeare In 4-gram Günay Natural Language Processing I (Ch. 22) Spring 2013 13 / 30

  17. Shakespeare N -gram Quiz Find: 1 real quote 3x unigram picks 3x bigram picks 3x trigram picks Günay Natural Language Processing I (Ch. 22) Spring 2013 14 / 30

  18. Shakespeare N -gram Quiz Find: 1 real quote 3x unigram picks 3x bigram picks 3x trigram picks Günay Natural Language Processing I (Ch. 22) Spring 2013 14 / 30

  19. Bigram Probability Question P ( ˆ woe is me | ˆ ) =? Given that: ˆ: symbol showing start of sentence P ( woe i | ˆ i − 1 ) = . 0002 P ( is i | woe i − 1 ) = . 07 P ( me i | is i − 1 ) = . 0005 Günay Natural Language Processing I (Ch. 22) Spring 2013 15 / 30

  20. Bigram Probability Question P ( ˆ woe is me | ˆ ) = . 0002 × . 07 × . 0005 = 7 × 10 − 9 Given that: ˆ: symbol showing start of sentence P ( woe i | ˆ i − 1 ) = . 0002 P ( is i | woe i − 1 ) = . 07 P ( me i | is i − 1 ) = . 0005 Günay Natural Language Processing I (Ch. 22) Spring 2013 15 / 30

  21. Other Tricks Stationarity assumption: Context doesn’t change over time. Günay Natural Language Processing I (Ch. 22) Spring 2013 16 / 30

  22. Other Tricks Stationarity assumption: Context doesn’t change over time. Smoothing: Remember Laplace smooting? Günay Natural Language Processing I (Ch. 22) Spring 2013 16 / 30

  23. Other Tricks Stationarity assumption: Context doesn’t change over time. Smoothing: Remember Laplace smooting? Hidden variables: E.g., identify what a “noun” is. Günay Natural Language Processing I (Ch. 22) Spring 2013 16 / 30

  24. Other Tricks Stationarity assumption: Context doesn’t change over time. Smoothing: Remember Laplace smooting? Hidden variables: E.g., identify what a “noun” is. Use abstractions: Group “New York City”, or just look at letters. Günay Natural Language Processing I (Ch. 22) Spring 2013 16 / 30

  25. Smaller Than Words? What if we cannot distinguish words? Günay Natural Language Processing I (Ch. 22) Spring 2013 17 / 30

  26. Smaller Than Words? What if we cannot distinguish words? Günay Natural Language Processing I (Ch. 22) Spring 2013 17 / 30

  27. Smaller Than Words? What if we cannot distinguish words? English: “choosespain.com” “Choose Spain” OR “Chooses Pain”? Günay Natural Language Processing I (Ch. 22) Spring 2013 17 / 30

  28. Smaller Than Words? What if we cannot distinguish words? English: “choosespain.com” “Choose Spain” OR “Chooses Pain”? Segmentation: Dividing into words. Günay Natural Language Processing I (Ch. 22) Spring 2013 17 / 30

  29. Smaller Than Words? What if we cannot distinguish words? English: “choosespain.com” “Choose Spain” OR “Chooses Pain”? Segmentation: Dividing into words. Use Bayes again: s ∗ = max P ( w 1 : n ) = max � P ( w i | w 1 : i ) i Günay Natural Language Processing I (Ch. 22) Spring 2013 17 / 30

  30. Smaller Than Words? What if we cannot distinguish words? English: “choosespain.com” “Choose Spain” OR “Chooses Pain”? Segmentation: Dividing into words. Use Bayes again: s ∗ = max P ( w 1 : n ) = max � P ( w i | w 1 : i ) i Or Markov assumption (e.g., unigram): s ∗ = max � P ( w i ) i Günay Natural Language Processing I (Ch. 22) Spring 2013 17 / 30

  31. Segmentation Complexity s ∗ = max � P ( w i ) i What’s the complexity of segmenting: ”nowisthetime” ? Günay Natural Language Processing I (Ch. 22) Spring 2013 18 / 30

  32. Segmentation Complexity s ∗ = max � P ( w i ) i What’s the complexity of segmenting: ”nowisthetime” ? 1 n − 1 2 ( n − 1 ) 2 3 ( n − 1 )! 4 2 n − 1 5 ( n − 1 ) n Günay Natural Language Processing I (Ch. 22) Spring 2013 18 / 30

  33. Segmentation Complexity s ∗ = max � P ( w i ) i What’s the complexity of segmenting: ”nowisthetime” ? 1 n − 1 2 ( n − 1 ) 2 3 ( n − 1 )! 4 2 n − 1 5 ( n − 1 ) n Günay Natural Language Processing I (Ch. 22) Spring 2013 18 / 30

  34. Segmentation Complexity s ∗ = max � P ( w i ) i What’s the complexity of segmenting: ”nowisthetime” ? 1 n − 1 2 ( n − 1 ) 2 3 ( n − 1 )! 4 2 n − 1 5 ( n − 1 ) n Solution: Separate each character with n − 1 divisions, form words by whether division exists or not. Günay Natural Language Processing I (Ch. 22) Spring 2013 18 / 30

  35. Reducing Segmentation Complexity Exploit independence: ”nowisthetime” ? Günay Natural Language Processing I (Ch. 22) Spring 2013 19 / 30

  36. Reducing Segmentation Complexity Exploit independence: ”nowisthetime” ? Divide into first, f , and recurse for rest, r : s ∗ = max s = f + r P ( f ) · s ∗ ( r ) Günay Natural Language Processing I (Ch. 22) Spring 2013 19 / 30

  37. Reducing Segmentation Complexity Exploit independence: ”nowisthetime” ? Divide into first, f , and recurse for rest, r : s ∗ = max s = f + r P ( f ) · s ∗ ( r ) Gives 99% accuracy and easy implementation ! Günay Natural Language Processing I (Ch. 22) Spring 2013 19 / 30

  38. Segmentation Problems Günay Natural Language Processing I (Ch. 22) Spring 2013 20 / 30

  39. Segmentation Problems How can we improve? 1 More Data 2 Markov 3 Smoothing Günay Natural Language Processing I (Ch. 22) Spring 2013 20 / 30

  40. Segmentation Problems How can we improve? 1 More Data 2 Markov 3 Smoothing Günay Natural Language Processing I (Ch. 22) Spring 2013 20 / 30

  41. Segmentation Problems How can we improve? 1 More Data 2 Markov 3 Smoothing Need to get the context . Günay Natural Language Processing I (Ch. 22) Spring 2013 20 / 30

  42. Segmentation Problems (2) Günay Natural Language Processing I (Ch. 22) Spring 2013 21 / 30

  43. Segmentation Problems (2) How can we improve? 1 More Data 2 Markov 3 Smoothing Günay Natural Language Processing I (Ch. 22) Spring 2013 21 / 30

  44. Segmentation Problems (2) How can we improve? 1 More Data 2 Markov 3 Smoothing Günay Natural Language Processing I (Ch. 22) Spring 2013 21 / 30

  45. Segmentation Problems (2) How can we improve? 1 More Data 2 Markov 3 Smoothing Günay Natural Language Processing I (Ch. 22) Spring 2013 21 / 30

  46. Segmentation Problems (2) How can we improve? 1 More Data 2 Markov 3 Smoothing Need to know more words . Günay Natural Language Processing I (Ch. 22) Spring 2013 21 / 30

  47. What Else Can We Do with Letters? Language identification? Günay Natural Language Processing I (Ch. 22) Spring 2013 22 / 30

  48. What Else Can We Do with Letters? Language identification? Günay Natural Language Processing I (Ch. 22) Spring 2013 22 / 30

  49. Bigram Recognition with Letters Günay Natural Language Processing I (Ch. 22) Spring 2013 23 / 30

  50. Bigram Recognition with Letters Günay Natural Language Processing I (Ch. 22) Spring 2013 23 / 30

  51. Trigram Recognition with Letters Günay Natural Language Processing I (Ch. 22) Spring 2013 24 / 30

  52. Trigram Recognition with Letters Günay Natural Language Processing I (Ch. 22) Spring 2013 24 / 30

  53. Trigram Recognition with Letters Günay Natural Language Processing I (Ch. 22) Spring 2013 24 / 30

  54. Trigram Recognition with Letters 99% accuracy from trigrams! Günay Natural Language Processing I (Ch. 22) Spring 2013 24 / 30

  55. Can We Identify Categories Too? Günay Natural Language Processing I (Ch. 22) Spring 2013 25 / 30

  56. Can We Identify Categories Too? Text classification Günay Natural Language Processing I (Ch. 22) Spring 2013 25 / 30

  57. Text Classification What algorithms can we use? Naive Bayes: Günay Natural Language Processing I (Ch. 22) Spring 2013 26 / 30

  58. Text Classification What algorithms can we use? Naive Bayes: Spam vs. Ham Günay Natural Language Processing I (Ch. 22) Spring 2013 26 / 30

  59. Text Classification What algorithms can we use? Naive Bayes: Spam vs. Ham k -Nearest Neighbor: Günay Natural Language Processing I (Ch. 22) Spring 2013 26 / 30

  60. Text Classification What algorithms can we use? Naive Bayes: Spam vs. Ham k -Nearest Neighbor: Similar words Günay Natural Language Processing I (Ch. 22) Spring 2013 26 / 30

  61. Text Classification What algorithms can we use? Naive Bayes: Spam vs. Ham k -Nearest Neighbor: Similar words Support Vector Machines: Günay Natural Language Processing I (Ch. 22) Spring 2013 26 / 30

  62. Text Classification What algorithms can we use? Naive Bayes: Spam vs. Ham k -Nearest Neighbor: Similar words Support Vector Machines: Supervised learning Günay Natural Language Processing I (Ch. 22) Spring 2013 26 / 30

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CS325 Artificial Intelligence Natural Language Processing II (Ch. 23) Dr. Cengiz Gnay, Emory

CS325 Artificial Intelligence Natural Language Processing II (Ch. 23) Dr. Cengiz Gnay, Emory

CS325 Artificial Intelligence Natural Language Processing II (Ch. 23) Dr. Cengiz Gnay, Emory Univ. Gnay () Natural Language Processing II (Ch. 23) Spring 2013 1 / 18 So Probabilities Enough for Understanding Language? He came from out of

1.04k views • 71 slides
Natural Language Processing Computational Linguistics Text processing Artificial Intelligence

Natural Language Processing Computational Linguistics Text processing Artificial Intelligence

Natural Language Processing Computational Linguistics Text processing Artificial Intelligence Lecture 6 Karim Bouzoubaa Content Acknowledgments Examples Defintions History Objective Levels - Problems Applications

1.03k views • 48 slides
Foundations of Artificial Intelligence 15. Natural Language Processing Understand, interpret,

Foundations of Artificial Intelligence 15. Natural Language Processing Understand, interpret,

Foundations of Artificial Intelligence 15. Natural Language Processing Understand, interpret, manipulate, generate human language (text and audio) Joschka Boedecker and Wolfram Burgard and Frank Hutter and Bernhard Nebel and Michael Tangermann

366 views • 36 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Introduction and

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Introduction and

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Introduction and Overview Murhaf Fares &amp; Stephan Oepen Language Technology Group (LTG) August 23, 2017 Topics for Today AI, NLP, ML What are they?

731 views • 34 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Introduction and

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Introduction and

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Introduction and Overview Murhaf Fares &amp; Stephan Oepen Language Technology Group (LTG) August 23, 2017 Topics for Today AI, NLP, ML What are they?

1.71k views • 74 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Common Lisp

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Common Lisp

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Common Lisp Fundamentals Stephan Oepen &amp; Murhaf Fares Language Technology Group (LTG) September 1, 2016 Last Week: What is AI? Since the 1950s: Chatbots,

931 views • 26 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Common Lisp

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Common Lisp

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Common Lisp Fundamentals Stephan Oepen &amp; Murhaf Fares Language Technology Group (LTG) August 30, 2017 Last Week: What is AI? Since the 1950s: Chatbots,

1.2k views • 104 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov Models Murhaf Fares &amp; Stephan Oepen Language Technology Group (LTG) October 27, 2016 Recap: Probabilistic Language Models Basic probability

1.48k views • 130 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov Models Murhaf Fares &amp; Stephan Oepen Language Technology Group (LTG) October 27, 2016 Recap: Probabilistic Language Models Basic probability

471 views • 31 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Wrap-Up and Exam

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Wrap-Up and Exam

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Wrap-Up and Exam Preparation Stephan Oepen &amp; Milen Kouylekov Language Technology Group (LTG) November 26, 2014 INF4820: Algorithms for Artificial

663 views • 31 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Data Structures

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Data Structures

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Data Structures &amp; Lisp Quiz Stephan Oepen &amp; Murhaf Fares Language Technology Group (LTG) September 15, 2016 Agenda Previously More Common Lisp

453 views • 23 slides
Artificial Intelligence Games need opponents that are challenging, or allies that are helpful

Artificial Intelligence Games need opponents that are challenging, or allies that are helpful

10/4/2012 Introduction to Artificial Intelligence (AI) Many applications for AI Computer vision, natural language processing, speech recognition, search But games are some of the more interesting apps Artificial Intelligence

519 views • 8 slides
Artificial Intelligence Opponents that are challenging, or allies that are helpful Unit

Artificial Intelligence Opponents that are challenging, or allies that are helpful Unit

2/17/2012 Introduction to Artificial Intelligence (AI) Many applications for AI Computer vision, natural language processing, speech recognition, search But games are some of the more interesting Artificial Intelligence Opponents

302 views • 8 slides
Embodied Machines Artificial vs. Embodied Intelligence Artificial Intelligence (AI)

Embodied Machines Artificial vs. Embodied Intelligence Artificial Intelligence (AI)

Embodied Machines Artificial vs. Embodied Intelligence Artificial Intelligence (AI) Natural Language Processing (NLP) Goal: write programs that understand and identify grammatical patterns Assign conventional meanings to

273 views • 26 slides
CSE 473: Artificial Intelligence Advanced Applic's: Natural Language Processing Steve Tanimoto

CSE 473: Artificial Intelligence Advanced Applic's: Natural Language Processing Steve Tanimoto

CSE 473: Artificial Intelligence Advanced Applic's: Natural Language Processing Steve Tanimoto --- University of Washington [Some of these slides were created by Dan Klein and Pieter Abbeel for CS188 Intro to AI at UC Berkeley. What is NLP?

695 views • 43 slides
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing More Common Lisp

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing More Common Lisp

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing More Common Lisp Stephan Oepen &amp; Murhaf Fares Language Technology Group (LTG) September 6, 2017 Agenda Previously Common Lisp essentials

1.39k views • 93 slides
CS 188: Artificial Intelligence Nave Bayes Instructors: Brijen Thananjeyan and Aditya Baradwaj

CS 188: Artificial Intelligence Nave Bayes Instructors: Brijen Thananjeyan and Aditya Baradwaj

CS 188: Artificial Intelligence Nave Bayes Instructors: Brijen Thananjeyan and Aditya Baradwaj --- University of California, Berkeley [These slides were created by Dan Klein, Pieter Abbeel, Sergey Levine, with some materials from A. Farhadi.

744 views • 39 slides
Alvord Lake - Stanyan across from Haight St. William Alvord (1833 - 1904) - merchant, banker, 14th

Alvord Lake - Stanyan across from Haight St. William Alvord (1833 - 1904) - merchant, banker, 14th

- Trained at Royal Botanic Garden, John Hays McLaren Edinburgh 1846 - 1943 - Hired by wealthy American to be Head Gardener at Rancho San Mateo (10 sq. mi. estate) -1872 immigrates to U.S. - Recommended to Park Commission by Ham Hall to

297 views • 14 slides
Aggregation and Ordering in Factorized Databases B akibayev, K o y, O lteanu, and Z cisk

Aggregation and Ordering in Factorized Databases B akibayev, K o y, O lteanu, and Z cisk

Aggregation and Ordering in Factorized Databases B akibayev, K o y, O lteanu, and Z cisk avodn y University of Oxford VLDB Sept 2, 2014 http://www.cs.ox.ac.uk/projects/FDB/ 1 / 19 Outline What are Factorized Databases?

302 views • 19 slides
RAAGs in Ham Misha Kapovich UC Davis June 30, 2011 Motivation M is a compact surface, is

RAAGs in Ham Misha Kapovich UC Davis June 30, 2011 Motivation M is a compact surface, is

RAAGs in Ham Misha Kapovich UC Davis June 30, 2011 Motivation M is a compact surface, is area form on M . Diff ( M , ) is the group of area-preserving diffeomorphisms of M . Motivation M is a compact surface, is area form on M

1.16k views • 86 slides
David Ham dolfin-adjoint: automating the adjoints of models written in the Python interface to

David Ham dolfin-adjoint: automating the adjoints of models written in the Python interface to

David Ham dolfin-adjoint: automating the adjoints of models written in the Python interface to DOLFIN David A. Ham 1 , 2 Patrick E. Farrell 1 Simon W. Funke 1 , 2 Marie E. Rognes 3 1 Department of Earth Science and Engineering, Imperial College

755 views • 18 slides
Classifica)on Example: Spam Filter Input: an email Dear

Classifica)on Example: Spam Filter Input: an email Dear

Machine Learning CS 232: Ar)ficial Intelligence Nave Bayes Oct 26, 2015 Part 1 of course: how use a model to make op)mal decisions

394 views • 10 slides
REPEATERS, WWARA &amp; THE FUTURE KU7M What is a Repeater? Who has heard of repeater

REPEATERS, WWARA &amp; THE FUTURE KU7M What is a Repeater? Who has heard of repeater

REPEATERS, WWARA &amp; THE FUTURE KU7M What is a Repeater? Who has heard of repeater coordination? Who authorizes repeater coordinators? WHAT IS REPEATER COORDINATION? 90% Bookkeeping 9% Arbitration The other 1% Repeater Timeline

379 views • 12 slides
Syntax and Semantics Philipp Koehn 3 November 2020 Philipp Koehn Machine Translation: Syntax

Syntax and Semantics Philipp Koehn 3 November 2020 Philipp Koehn Machine Translation: Syntax

Syntax and Semantics Philipp Koehn 3 November 2020 Philipp Koehn Machine Translation: Syntax and Semantics 3 November 2020 1 syntax Philipp Koehn Machine Translation: Syntax and Semantics 3 November 2020 Tree-Based Models 2

1.17k views • 57 slides

More recommend