Deep Neural Networks Machine Learning Organizational info All - PowerPoint PPT Presentation

10-701: Introduction to Deep Neural Networks Machine Learning

Organizational info • All up-to-date info is on piazza. • Instructors - Ziv Bar-Joseph - Eric Xing • TAs: See info on piazza for recitations, office hours etc. • See also piazza for contact info, office hours, etc. • Piazza would be used for questions / comments and for class quizzes. Make sure you are subscribed. • We will also use piazza for determining class participation

Eric Xing (epxing@cs.cmu.edu) Models and Algorithms • Graphical • Nonparametric • Regularized • Spectral/Matrix • Sparse Structured Neural Nets • Sparse Coding • Large-Margin Models Bayesian Models Bayesian Methods Methods I/O Regression Trustworthy Personalized Meta ML Auto ML ML ML ML Compositionality Adaptive Distributed Big Data Scheduler ML Systems Tools System Compositionality Hardware and infrastructure • Network switches • Network attached storage • Server machines • GPUs • Cloud compute • Virtual Machines • Infiniband • Flash storage • Desktops/Laptops (e.g. Amazon EC2) 2020 IC, SCS@CMU • NUMA machines • Research Interests – Machine Learning: Theory & System – Healthcare and other Applications – Way to Learn: Auto, Trustworthy, Office: GHC Personalizable, and Transferable ML 8101 Office hours: TBD Course Instructor

Daniel Bird (dpbird@andrew.cmu.edu) Education Associate for 10-701 Please email me if you have any issues in the course!

Roger Iyengar (raiyenga@andrew.cmu.edu) PhD in Computer Science Department Interests: Edge Computing, Wearable Cognitive Assistance, Distributed Systems Research Interests in ML: Computer Vision, Natural Language Processing

Abhi Adduri (aadduri@andrew.cmu.edu) PhD in Computational Biology

Clay Yoo (hyungony@andrew.cmu.edu) Masters in Language Technology Institute Area of interest: Natural Language Processing, Data Visualization, Model Interpretability

John Grace (jmgrace@andrew.cmu.edu) Masters in Computer Science Department Area of interest: Parallel Computing and Automated Program Synthesis.

Chandreyee Bhaumik (cbhaumik@andrew.cmu.edu) Masters in the Robotics Institute Area of Interest: Reinforcement Learning, Computer Vision

Bhuvan Agrawal (bhuvana@andrew.cmu.edu) Masters in Computer Science Department

Jie Jiao (jiejiao@andrew.cmu.edu) Undergraduate in Computer Science Department Area of interest: Reinforcement Learning, Natural Language Processing

8/31 Intro, Three Axes of ML: Data, Algorithms, Tasks, Intro to probability Foundations and Non- 9/2 Bayesian Estimation, MAP, MLE Parametric Methods 9/7 – no class, labor day 9/9 – Decision Theory, Risk Minimization, K nearest neighbors 9/14 – Naive Bayes, Generative vs Discriminative 9/16 – Decision Trees 9/21 - Linear regression 9/23 - Logistic Regression Prediction, 9/28 – No class, Yom Kippur Parametric 9/30 – Support Vector Machines 1 Methods 10/05 – SVM2 10/07 – Neural Networks and Deep Learning 10/12 – Neural Networks and Deep Learning II 10/14 – Boosting, Surrogate Losses, Ensemble Methods Unsupervised Learning 10/19 - Clustering, Kmeans 10/21 - Clustering: Mixture of Gaussians, Expectation Maximization 10/26 – Representation Learning: Feature Transformation, Random Features, PCA Graphical and sequence 10/28 – Representation Learning: PCA Contd, ICA/ project proposals due 11/02 - Graphical Models (Bayesian Networks) models 11/04 - Graphical Models (BN2) 11/09 - Sequence Models: HMMs 11/11 - Sequence Models: State Space Models, other time series models Theoretical 11/16 – Learning Theory: Statistical Guarantees for Empirical Risk Minimization considerations 11/18 – Generalization, Model Selection 11/23 (Wednesday): Exam 11/23 - Exam 11/25 – No class, Thanksgiving break 11/30 – Industry lecture 12/02 – Reinforcement Learning 12/07 – Reinforcement Learning 2 12/09 (Wednesday) Poster presentations Actions 12/09 - Project presentations

Grading • 5 Problem sets - 40% • Exam - 30% • Project - 30%

Class assignments • 5 Problem sets - Both theoretical and programming assignments • Project - Select from a small list of suggested topics - We expect that multiple groups would work on a similar project - Groups of 3 - Poster session (recorded) and a short writeup • Exams - A single exams covering all topics taught in class up to the date - During class dates but likely in the afternoon (5-7pmt) • Recitations - Every Friday - Expand on material learned in class, go over problems from previous classes etc. • Office hours based on your section

What is Machine Learning? Easy part: Machine Hard part: Learning - Short answer: Methods that can help generalize information from the observed data so that it can be used to make better decisions in the future

What is Machine Learning? LEARNING DATA KNOWLEDGE ALGORITHMS

Machine Learning LEARNING DATA KNOWLEDGE ALGORITHMS • Algorithms that improve their knowledge towards some task with data • How is it different from Statistics? - Same, but with better PR? - Statistics + Computation? • What is its relationship with AI, Data Science, Data Mining?

Machine Learning • It is useful to differentiate these different fields by their goals • The goal of machine learning is the underlying mechanisms and algorithms that allow improving our knowledge with more data - Data construed broadly, e.g. “experiences” - Knowledge construed broadly e.g. possible actions 18

While there is overlap, there are also differences • Statistics: the goal is the understanding of the data at hand • Artificial Intelligence: the goal is to build an intelligent agent • Data Mining: the goal is to extract patterns from large-scale data • Data Science: the science encompassing collection, analysis, and interpretation of data

From Data to Understanding … Machine Learning in Action 20

Machine Learning in Action Supervised learning • Decoding thoughts from brain scans Rob a bank … 21

Machine Learning in Action Supervised and unsupervised learning • Stock Market Prediction Y = ? X = Feb01 22

Machine Learning in Action Supervised and unsupervised learning • Document classification Sports Science News 23

Machine Learning in Action Supervised learning • Spam filtering Spam/ Not spam 24

semi supervised learning

Machine Learning in Action Supervised and reinforcement learning • Cars navigating on their own Boss, the self-driving SUV 1st place in the DARPA Urban Challenge. Photo courtesy of Tartan Racing. 26

Supervised learning (though can also be trained in an unsupervised way) Google translate

Distributed gradient descent based on bacterial movement Reasoning under uncertainty