Machine Learning for Computational Linguistics ar ltekin University - - PowerPoint PPT Presentation

▶

Sep 12, 2023 523 likes •786 views

Machine Learning for Computational Linguistics ar ltekin University of Tbingen Seminar fr Sprachwissenschaft April 12, 2016 Practical matters What and why The course plan When/where ccoltekin@sfs.uni-tuebingen.de).

SLIDE 1

Machine Learning for Computational Linguistics

Çağrı Çöltekin

University of Tübingen Seminar für Sprachwissenschaft

April 12, 2016

SLIDE 2

Practical matters What and why The course plan

When/where

▶ Lectures: Tuesday/Thursday 08:30 at Hörsaal 0.02 ▶ Offjce hours: Tuesday 10:00-12:00, or by appointment (email

ccoltekin@sfs.uni-tuebingen.de).

▶ Course web page:

http://coltekin.net/cagri/courses/ml.

▶ Reading material: no (single) textbook. Course web page will

include pointers to reading material for each lecture.

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 1 / 16

SLIDE 3

Practical matters What and why The course plan

Literature

▶ James et al. (2013) [online!] ▶ Hastie, Tibshirani, and J. Friedman (2009) [online!] ▶ Barber (2012) ▶ Murphy (2012) ▶ Bishop (2006) ▶ Mitchell (1997) ▶ Goodfellow, Bengio, and Courville (2016) [online copy] ▶ Alpaydın (2004) ▶ Witten and Frank (2005) ▶ Richert (2015) ▶ Lantz (2015) ▶ Cho (2015) ▶ Goldberg (2015)

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 2 / 16

SLIDE 4

Practical matters What and why The course plan

Evaluation

▶ Homeworks (30%)

6 (maybe 7) homeworks, to be done individually.

▶ Term project / term paper (70%)

▶ Team work (up to 3 team members) is encouraged ▶ The project has to include a machine learning ‘experiment’ ▶ The results should be presented in a term paper (details will be

announced later)

▶ You should already start thinking about project topics, and

forming teams

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 3 / 16

SLIDE 5

Practical matters What and why The course plan

Machine learning is …

The fjeld of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience. —Mitchell (1997) Machine Learning is the study of data-driven methods capable of mimicking, understanding and aiding human and biological information processing tasks. —Barber (2012) Statistical learning refers to a vast set of tools for understanding data. —James et al. (2013)

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 4 / 16

SLIDE 6

Practical matters What and why The course plan

Machine learning is …

The fjeld of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience. —Mitchell (1997) Machine Learning is the study of data-driven methods capable of mimicking, understanding and aiding human and biological information processing tasks. —Barber (2012) Statistical learning refers to a vast set of tools for understanding data. —James et al. (2013)

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 4 / 16

SLIDE 7

Practical matters What and why The course plan

Machine learning is …

The fjeld of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience. —Mitchell (1997) Machine Learning is the study of data-driven methods capable of mimicking, understanding and aiding human and biological information processing tasks. —Barber (2012) Statistical learning refers to a vast set of tools for understanding data. —James et al. (2013)

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 4 / 16

SLIDE 8

Practical matters What and why The course plan

Machine learning and computational linguistics

▶ Majority of the computational linguistic tasks and applications

are based on machine learning

▶ Tokenization ▶ Part of speech tagging ▶ Parsing ▶ … ▶ Speech recognition ▶ Named Entity recognition ▶ Document classifjcation ▶ Question answering ▶ Machine translation ▶ … Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 5 / 16

SLIDE 9

Practical matters What and why The course plan

Refresher: linear algebra

Thursday!

▶ Vectors, vector operations, their geometric interpretations ▶ Vector norms, distances between vectors ▶ Matrices, matrix operations ▶ Some useful matrix properties ▶ Linear transformations

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 6 / 16

SLIDE 10

Practical matters What and why The course plan

Refresher: probability and statistics

Next week

▶ Probabilities: where do they come from? ▶ Random variables, probability distributions ▶ Joint, conditional, marginal probabilities, chain rule ▶ Bayes’ formula ▶ Some concepts from information theory

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 7 / 16

SLIDE 11

Practical matters What and why The course plan

Regression

20 40 60 80 20 40 60 80

x y

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 8 / 16

SLIDE 12

Practical matters What and why The course plan

Classifjcation

x2 x1 + + + + – – – –

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 9 / 16

SLIDE 13

Practical matters What and why The course plan

Classifjcation

x2 x1 ? + + + + – – – –

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 9 / 16

SLIDE 14

Practical matters What and why The course plan

Classifjcation

x2 x1 ? + + + + – – – –

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 9 / 16

SLIDE 15

Practical matters What and why The course plan

Machine learning basics

▶ How to measure success in an ML experiment? ▶ Variance and bias ▶ Overfjtting and underfjtting ▶ Cross validation ▶ Training/test/development set split

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 10 / 16

SLIDE 16

Practical matters What and why The course plan

Unsupervised learning

▶ Clustering ▶ Density estimation ▶ Dimensionality reduction

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 11 / 16

SLIDE 17

Practical matters What and why The course plan

Neural networks

x1 x2 x3 x4 Output Output Output Input layer Hidden layer Output layer

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 12 / 16

SLIDE 18

Practical matters What and why The course plan

Distributed representations

▶ Sparse feature representations ▶ Dense representations ▶ Word/character embeddings ▶ How to obtain meaningful combinations?

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 13 / 16

SLIDE 19

Practical matters What and why The course plan

Deep learning

▶ Convolutional networks ▶ Recurrent networks ▶ Auto-encoder/decoders ▶ …

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 14 / 16

SLIDE 20

Practical matters What and why The course plan

Bayesian learning (if time allows)

▶ Bayesian inference ▶ Graphical models

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 15 / 16

SLIDE 21

Practical matters What and why The course plan

Summary

▶ Besides what what is covered during the course, we will note

some that was not covered

▶ Decision trees, random forests ▶ Rule learning ▶ Memory based learning ▶ Support vector machines ▶ Local regression / generalized additive models ▶ Learning sequences (e.g., HMMs) ▶ Active learning ▶ Reinforcement learning ▶ Ensemble methods ▶ … Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 16 / 16

SLIDE 22

Machine learning books/resources

The following is an unsorted list of machine lerning related books and resources.

Abu-Mostafa, Yaser S., Malik Magdon-Ismail, and Hsuan-Tien Lin (2012). Learning from Data: A Short Course. AMLBook.com. isbn: 9781600490064. Alpaydın, Ethem (2004). Introduction to machine learning. Adaptive computation and machine learning. MIT

Press. isbn: 0262012111,9780262012119.

Barber, David (2012). Bayesian Reasoning and Machine Learning. Cambridge University Press. isbn: 9780521518147. Bishop, Christopher M. (2006). Pattern Recognition and Machine Learning. Springer. isbn: 978-0387-31073-2. Bowles, Michael (2015). Machine Learning in Python: Essential Techniques for Predictive Analysis. Wiley. isbn: 9781118961766. Cho, Kyunghyun (2015). Natural Language Understanding with Distributed Representation. arXiv preprint arXiv:1511.07916. Flach, Peter (2012). Machine Learning: The Art and Science of Algorithms that Make Sense of Data. Cambridge University Press. isbn: 9781107096394. Goldberg, Yoav (2015). A Primer on Neural Network Models for Natural Language Processing. url: http://www.cs.biu.ac.il/~yogo/nnlp.pdf. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville (2016). “Deep Learning”. Book in preparation for MIT Press. url: http://www.deeplearningbook.org. Grus, Joel (2015). Data Science from Scratch: First Principles with Python. O’Reilly Media. isbn: 9781491904404. Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 A.1

SLIDE 23

Machine learning books/resources (cont.)

Harrington, Peter (2012). Machine Learning in Action. Manning Publications Company. isbn: 9781617290183. Hastie, Trevor, Robert Tibshirani, and Jerome Friedman (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Second. Springer series in statistics. Springer-Verlag New York. isbn:

9780387848587. url: http://web.stanford.edu/~hastie/ElemStatLearn/.

Haykin, Simon O. (2011). Neural Networks and Learning Machines. Pearson Education. isbn: 9780133002553. James, G., D. Witten, T. Hastie, and R. Tibshirani (2013). An Introduction to Statistical Learning: with Applications in R. Springer Texts in Statistics. Springer New York. isbn: 9781461471387. url: http://www-bcf.usc.edu/~gareth/ISL/. Jaynes, Edwin T (2007). Probability Theory: The Logic of Science. Ed. by G. Larry Bretthorst. Cambridge University Press. isbn: 978-05-2159-271-0. Koller, Daphne and Nir Friedman (2009). Probabilistic Graphical Models: Principles and Techniques. The MIT

Press. isbn: 978-02-6201-319-2.

Kubat, Miroslav (2015). An Introduction to Machine Learning. Springer International Publishing. isbn: 9783319200101. Kulkarni, Sanjeev and Gilbert Harman (2011). An Elementary Introduction to Statistical Learning Theory. Wiley Series in Probability and Statistics. Wiley. isbn: 9781118023433. Lantz, Brett (2015). Machine Learning with R. second. Packt Publishing. isbn: 9781782162148. MacKay, David J. C. (2003). Information Theory, Inference and Learning Algorithms. Cambridge University Press. isbn: 978-05-2164-298-9. url: http://www.inference.phy.cam.ac.uk/itprnn/book.html. Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 A.2

SLIDE 24

Machine learning books/resources (cont.)

Marsland, Stephen (2015). Machine Learning: An Algorithmic Perspective, Second Edition. Chapman et Hall/CRC machine learning et pattern recognition series. CRC Press. isbn: 9781466583337. Mitchell, Thomas (1997). Machine Learning. 1st. McGraw Hill Higher Education. isbn: 0071154671,0070428077,9780071154673,9780070428072. Munakata, Toshinori (2008). Fundamentals of the New Artifjcial Intelligence: Neural, Evolutionary, Fuzzy and

More. Springer. isbn: 978-1-84628-839-5.

Murphy, Kevin P. (2012). Machine Learning: A Probabilistic Perspective. Adaptive computation and machine learning series. MIT Press. isbn: 9780262018029. Raschka, Sebastian (2015). Python Machine Learning. Packt Publishing. isbn: 9781783555147. Richert, Willi (2015). Building Machine Learning Systems with Python. second. Packt Publishing. isbn: 9781784392772. Shalev-Shwartz, S. and S. Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press. isbn: 9781107057135. Smola, Alex and S. V. N. Vishwanathan (2008). Introduction to machine learning. Cambridge University Press. isbn: 0521825830. Suykens, J.A.K., M. Signoretto, and A. Argyriou (2014). Regularization, Optimization, Kernels, and Support Vector

Machines. Chapman & Hall/CRC machine learning & pattern recognition series. CRC Press. isbn:

9781482241402. Witten, I.H. and E. Frank (2005). Data Mining: Practical Machine Learning Tools and Techniques, Second Edition.

2nd. The Morgan Kaufmann Series in Data Management Systems. Elsevier Science. isbn: 0120884070.

Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 A.3

SLIDE 25

Machine learning books/resources (cont.)

Zumel, Nina and John Mount (2014). Practical Data Science with R. Manning. isbn: 9781617291562. Ç. Çöltekin, SfS / University of Tübingen April 12, 2016 A.4