lecture 4 optimization
play

Lecture 4: Optimization Justin Johnson Lecture 4 - 1 September - PowerPoint PPT Presentation

Jun 07, 2023 •260 likes •1.12k views

Lecture 4: Optimization Justin Johnson Lecture 4 - 1 September 16, 2019 Waitlist Update We will open the course for enrollment later today / tomorrow Justin Johnson Lecture 4 - 2 September 16, 2019 Reminder: Assignment 1 Was due

  1. Lecture 4: Optimization Justin Johnson Lecture 4 - 1 September 16, 2019

  2. Waitlist Update We will open the course for enrollment later today / tomorrow Justin Johnson Lecture 4 - 2 September 16, 2019

  3. Reminder: Assignment 1 Was due yesterday! (But you do have late days…) Justin Johnson Lecture 4 - 3 September 16, 2019

  4. Assignment 2 • Will be released today • Use SGD to train linear classifiers and fully-connected networks • After today, can do linear classifiers section • After Wednesday, can do fully-connected networks • If you have a hard time computing derivatives, wait for next Monday’s lecture on backprop • Due Monday September 30, 11:59pm (two weeks from today) Justin Johnson Lecture 4 - 4 September 16, 2019

  5. Course Update - A1: 10% - A2: 10% - A3: 10% - A4: 10% - A5: 10% - A6: 10% - Midterm: 20% - Final: 20% Justin Johnson Lecture 4 - 5 September 16, 2019

  6. Course Update: No Final Exam - A1: 10% - A1: 10% - A2: 10% - A2: 13% - A3: 10% - A3: 13% - A4: 10% - A4: 13% Expect A5 and A6 - A5: 10% - A5: 13% to be longer than other homework - A6: 10% - A6: 13% - Midterm: 20% - Midterm: 25% - Final: 20% - Final Justin Johnson Lecture 4 - 6 September 16, 2019

  7. Last Time: Linear Classifiers Visual Viewpoint Algebraic Viewpoint Geometric Viewpoint One template Hyperplanes f(x,W) = Wx per class cutting up space Justin Johnson Lecture 4 - 7 September 16, 2019

  8. Last Time: Loss Functions quantify preferences We have some dataset of (x, y) - We have a score function: - We have a loss function : - Linear classifier Softmax SVM Full loss Justin Johnson Lecture 4 - 8 September 16, 2019

  9. Last Time: Loss Functions quantify preferences Q : How do we find the best W? We have some dataset of (x, y) - We have a score function: - We have a loss function : - Linear classifier Softmax SVM Full loss Justin Johnson Lecture 4 - 9 September 16, 2019

  10. Optimization Justin Johnson Lecture 4 - 10 September 16, 2019

  11. This image is CC0 1.0 public domain Walking man image is CC0 1.0 public domain Justin Johnson Lecture 4 - 11 September 16, 2019

  12. This image is CC0 1.0 public domain Walking man image is CC0 1.0 public domain Justin Johnson Lecture 4 - 12 September 16, 2019

  13. Idea #1: Random Search ch (bad idea!) Justin Johnson Lecture 4 - 13 September 16, 2019

  14. Idea #1: Random Search ch (bad idea!) 15.5% accuracy! not bad! Justin Johnson Lecture 4 - 14 September 16, 2019

  15. Idea #1: Random Search ch (bad idea!) 15.5% accuracy! not bad! (SOTA is ~95%) Justin Johnson Lecture 4 - 15 September 16, 2019

  16. Idea #2: Fo Follow the slope Justin Johnson Lecture 4 - 16 September 16, 2019

  17. Idea #2 : : Follow th the e slope In 1-dimension, the derivative of a function gives the slope: Justin Johnson Lecture 4 - 17 September 16, 2019

  18. Idea #2 : : Follow th the e slope In 1-dimension, the derivative of a function gives the slope: In multiple dimensions, the gradient is the vector of (partial derivatives) along each dimension The slope in any direction is the dot product of the direction with the gradient The direction of steepest descent is the negative gradient Justin Johnson Lecture 4 - 18 September 16, 2019

  19. current W: gradient dL/dW: [0.34, [?, -1.11, ?, 0.78, ?, 0.12, ?, 0.55, ?, 2.81, ?, -3.1, ?, -1.5, ?, 0.33,…] ?,…] loss 1.25347 Justin Johnson Lecture 4 - 19 September 16, 2019

  20. current W: W + h (first dim) : gradient dL/dW: [0.34, [0.34 + 0.0001 , [?, -1.11, -1.11, ?, 0.78, 0.78, ?, 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25322 Justin Johnson Lecture 4 - 20 September 16, 2019

  21. current W: W + h (first dim) : gradient dL/dW: [0.34, [0.34 + 0.0001 , [ -2.5 , -1.11, -1.11, ?, 0.78, 0.78, ?, 0.12, 0.12, ?, (1.25322 - 1.25347)/0.0001 0.55, 0.55, ?, = -2.5 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25322 Justin Johnson Lecture 4 - 21 September 16, 2019

  22. current W: W + h (second dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11 + 0.0001 , ?, 0.78, 0.78, ?, 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25353 Justin Johnson Lecture 4 - 22 September 16, 2019

  23. current W: W + h (second dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11 + 0.0001 , 0.6 , 0.78, 0.78, ?, 0.12, 0.12, ?, 0.55, 0.55, ?, (1.25353 - 1.25347)/0.0001 2.81, 2.81, ?, = 0.6 -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25353 Justin Johnson Lecture 4 - 23 September 16, 2019

  24. current W: W + h (third dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11, 0.6, 0.78, 0.78 + 0.0001 , ?, 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, ?, -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25347 Justin Johnson Lecture 4 - 24 September 16, 2019

  25. current W: W + h (third dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11, 0.6, 0.78, 0.78 + 0.0001 , 0.0 , 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, (1.25347 - 1.25347)/0.0001 -3.1, -3.1, ?, = 0.0 -1.5, -1.5, ?, 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25347 Justin Johnson Lecture 4 - 25 September 16, 2019

  26. current W: W + h (third dim) : gradient dL/dW: [0.34, [0.34, [-2.5, -1.11, -1.11, 0.6, 0.78, 0.78 + 0.0001 , 0.0 , 0.12, 0.12, ?, 0.55, 0.55, ?, 2.81, 2.81, ?, -3.1, -3.1, Numeric Gradient : ?, - Slow: O(#dimensions) -1.5, -1.5, ?, - Approximate 0.33,…] 0.33,…] ?,…] loss 1.25347 loss 1.25347 Justin Johnson Lecture 4 - 26 September 16, 2019

  27. Loss is a function of W want Justin Johnson Lecture 4 - 27 September 16, 2019

  28. Loss is a function of W: Analytic Gradient want Use calculus to compute an analytic gradient This image is in the public domain This image is in the public domain Justin Johnson Lecture 4 - 28 September 16, 2019

  29. current W: gradient dL/dW: [0.34, [-2.5, dL/dW = ... -1.11, 0.6, (some function 0.78, 0, data and W) 0.12, 0.2, 0.55, 0.7, 2.81, -0.5, -3.1, 1.1, -1.5, 1.3, 0.33,…] -2.1,…] loss 1.25347 Justin Johnson Lecture 4 - 29 September 16, 2019

  30. current W: gradient dL/dW: [0.34, [-2.5, dL/dW = ... -1.11, 0.6, (some function 0.78, 0, data and W) 0.12, 0.2, 0.55, 0.7, 2.81, -0.5, -3.1, (In practice we will compute 1.1, dL/dW using backpropagation; -1.5, 1.3, see Lecture 6) 0.33,…] -2.1,…] loss 1.25347 Justin Johnson Lecture 4 - 30 September 16, 2019

  31. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 31 September 16, 2019

  32. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 32 September 16, 2019

  33. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 33 September 16, 2019

  34. Computing Gradients Numeric gradient : approximate, slow, easy to write - Analytic gradient : exact, fast, error-prone - In practice: Always use analytic gradient, but check implementation with numerical gradient. This is called a gradient check. Justin Johnson Lecture 4 - 34 September 16, 2019

  35. Gradient Descent Iteratively step in the direction of the negative gradient (direction of local steepest descent) Hyperparameters : - Weight initialization method - Number of steps - Learning rate Justin Johnson Lecture 4 - 35 September 16, 2019

  36. negative Gradient Descent gradient original W direction Iteratively step in the direction of W_2 the negative gradient (direction of local steepest descent) Hyperparameters : - Weight initialization method - Number of steps - Learning rate W_1 Justin Johnson Lecture 4 - 36 September 16, 2019

  37. Gradient Descent Iteratively step in the direction of the negative gradient (direction of local steepest descent) Hyperparameters : - Weight initialization method - Number of steps - Learning rate Justin Johnson Lecture 4 - 37 September 16, 2019

  38. Batch ch Gradient Desce cent Full sum expensive when N is large! Justin Johnson Lecture 4 - 38 September 16, 2019

  39. cent (SGD) Stoch chastic c Gradient Desce Full sum expensive when N is large! Approximate sum using a minibatch of examples 32 / 64 / 128 common Hyperparameters : - Weight initialization - Number of steps - Learning rate - Batch size - Data sampling Justin Johnson Lecture 4 - 39 September 16, 2019

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic:

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic:

AM 205: lecture 20 Today: PDE optimization, constrained optimization example New topic: eigenvalue problems PDE Constrained Optimization We will now consider optimization based on a function that depends on the solution of a PDE Let us

604 views • 26 slides
CMU-Q 15-381 Lecture 8: Optimization I: Optimization for CSP Local Search Teacher: Gianni A.

CMU-Q 15-381 Lecture 8: Optimization I: Optimization for CSP Local Search Teacher: Gianni A.

CMU-Q 15-381 Lecture 8: Optimization I: Optimization for CSP Local Search Teacher: Gianni A. Di Caro L OCAL S EARCH FOR CSP Real-life CSPs can be very large and hard to solve Methods so far: construct a solution by assigning one

767 views • 19 slides
CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26

CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26

CSC321 Lecture 8: Optimization Roger Grosse Roger Grosse CSC321 Lecture 8: Optimization 1 / 26 Overview Weve talked a lot about how to compute gradients. What do we actually do with them? Todays lecture: various things that can go

675 views • 29 slides
CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25

CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25

CSC321 Lecture 7: Optimization Roger Grosse Roger Grosse CSC321 Lecture 7: Optimization 1 / 25 Overview Weve talked a lot about how to compute gradients. What do we actually do with them? Todays lecture: various things that can go

676 views • 28 slides
Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is

Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is

Introduction to Numerical Optimization Biostatistics 615/815 Lecture 14 Lecture 14 Course is More Than Half Done! If you have comments they are very welcome y y Lectures Lecture notes Weekly Homework Midterm

795 views • 42 slides
Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric

Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric

Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric Optimization Minimize/maximize something subject to some constraints Have seen: Linear Programming Minimum Enclosing Ball

337 views • 17 slides
Optimization X Heuristic Optimization R Lecture 1 f : X R Algorithm Engineering Group

Optimization X Heuristic Optimization R Lecture 1 f : X R Algorithm Engineering Group

Optimization X Heuristic Optimization R Lecture 1 f : X R Algorithm Engineering Group Hasso Plattner Institute, University of Potsdam Goal: 14 April 2015 Find z X such that f ( z ) f ( x ) for all x X (minimization) Find z

612 views • 5 slides
AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector

AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector

AM 205: lecture 17 Last time: introduction to optimization Today: scalar and vector optimization Note: last years midterm is now available on the website for practice Motivation: Optimization If the objective function or any of the

1.1k views • 48 slides
15-780 Graduate Artificial Intelligence: Optimization J. Zico Kolter (this lecture) and Ariel

15-780 Graduate Artificial Intelligence: Optimization J. Zico Kolter (this lecture) and Ariel

15-780 Graduate Artificial Intelligence: Optimization J. Zico Kolter (this lecture) and Ariel Procaccia Carnegie Mellon University Spring 2017 1 Outline Introduction to optimization Types of optimization problems, convexity Solving

831 views • 30 slides
Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013

Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013

Convex Optimization ( EE227A: UC Berkeley ) Lecture 6 (Conic optimization) 07 Feb, 2013 Suvrit Sra Organizational Info Quiz coming up on 19th Feb. Project teams by 19th Feb Good if you can mix your research with class projects

1.19k views • 98 slides
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 13: Optimality Conditions for

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 13: Optimality Conditions for

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 13: Optimality Conditions for Convex Optimization Instructor: Shaddin Dughmi Announcements Today: short lecture wrapping up convex optimization Thursday: We begin combinatorial

396 views • 20 slides
CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems

CS599: Convex and Combinatorial Optimization Fall 2013 Lecture 9: Convex Optimization Problems Instructor: Shaddin Dughmi Announcements Homework: Due beginning of next class Must submit a hard copy, unless you have a good excuse If using late

934 views • 62 slides
Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos

Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos

Lecture 19 Additional Material: Optimization CS109A Introduction to Data Science Pavlos Protopapas and Kevin Rader Outline Optimization Challenges in Optimization Momentum Adaptive Learning Rate Parameter Initialization Batch

457 views • 45 slides
10 Analytics & Optimization From Code to Product gidgreen.com/course Lecture 10

10 Analytics & Optimization From Code to Product gidgreen.com/course Lecture 10

10 Analytics & Optimization From Code to Product gidgreen.com/course Lecture 10 Introduction Data collection Website metrics Optimization Competitive intelligence Surveys Tools and books From Code to Product

948 views • 68 slides
Lecture 6: Optimization CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline

Lecture 6: Optimization CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline

Lecture 6: Optimization CS109B Data Science 2 Pavlos Protopapas and Mark Glickman Outline Optimization Challenges in Optimization Momentum Adaptive Learning Rate Parameter Initialization Batch Normalization CS109B, P

495 views • 47 slides
Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic

Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic

An Example Outline Heuristics for Continuos Optimization DM812 METAHEURISTICS Lecture 13 Heuristic Methods for Continuous 1. An Application Example in Econometrics Optimization 2. Heuristics for Continuos Optimization Marco Chiarandini

410 views • 6 slides
Robotic Motion Planning: Potential Functions Robotics Institute 16-735

Robotic Motion Planning: Potential Functions Robotics Institute 16-735

Robotic Motion Planning: Potential Functions Robotics Institute 16-735 http://www.cs.cmu.edu/~motion Howie Choset http://www.cs.cmu.edu/~choset The Basic Idea A really simple idea: Suppose the goal is a point g 2 Suppose

294 views • 28 slides
Quantum tunneling in nonintegrable systems: beyond the leading order semiclassical description

Quantum tunneling in nonintegrable systems: beyond the leading order semiclassical description

Quantum tunneling in nonintegrable systems: beyond the leading order semiclassical description Akira Shudo Department of Physics, Tokyo Metropolitan University collaboration with Y Hanada and K S Ikeda Introduction Dynamical tunneling - No

482 views • 33 slides
Dynamics of a growth model with negative and positive externalities A. Antoci, S. Borghesi, M.

Dynamics of a growth model with negative and positive externalities A. Antoci, S. Borghesi, M.

Dynamics of a growth model with negative and positive externalities A. Antoci, S. Borghesi, M. Galeotti, P. Russu Negative and positive externalities We consider a model with optimizing agents, in which economic dynamics is conditioned by:

582 views • 12 slides
Nonlinear Control Lecture # 2 Two-Dimensional Systems Nonlinear Control Lecture # 2

Nonlinear Control Lecture # 2 Two-Dimensional Systems Nonlinear Control Lecture # 2

Nonlinear Control Lecture # 2 Two-Dimensional Systems Nonlinear Control Lecture # 2 Two-Dimensional Systems x 1 = f 1 ( x 1 , x 2 ) = f 1 ( x ) x 2 = f 2 ( x 1 , x 2 ) = f 2 ( x ) Let x ( t ) = ( x 1 ( t ) , x 2 ( t )) be a solution that

426 views • 21 slides
Extensive-Form Correlated Equilibrium Gabriele Farina 1 Chun Kai Ling 1 Fei Fang 2 Tuomas Sandholm

Extensive-Form Correlated Equilibrium Gabriele Farina 1 Chun Kai Ling 1 Fei Fang 2 Tuomas Sandholm

Efficient Regret Minimization Algorithm for Extensive-Form Correlated Equilibrium Gabriele Farina 1 Chun Kai Ling 1 Fei Fang 2 Tuomas Sandholm 1,3,4,5 1 Computer Science Department, Carnegie Mellon University 2 Institute for Software Research,

449 views • 11 slides
LOCAL HOUSING ALLOWANCE What is Local Housing Allowance

LOCAL HOUSING ALLOWANCE What is Local Housing Allowance

LOCAL HOUSING ALLOWANCE What is Local Housing Allowance

342 views • 10 slides
Welcome to the LIFE Webinar Series. We will be starting soon. The Low-Income Forum on Energy

Welcome to the LIFE Webinar Series. We will be starting soon. The Low-Income Forum on Energy

Welcome to the LIFE Webinar Series. We will be starting soon. The Low-Income Forum on Energy Presents: Increasing Access to Solar for Low- to Moderate-Income Communities Jeff Cook and Jenny Heeter National Renewable Energy Laberatory December

672 views • 55 slides
First Quarter Fiscal 2017 Quarter Ended December 31, 2016 Safe Harbor Statement Certain

First Quarter Fiscal 2017 Quarter Ended December 31, 2016 Safe Harbor Statement Certain

First Quarter Fiscal 2017 Quarter Ended December 31, 2016 Safe Harbor Statement Certain statements made during this presentation, including the Company's outlook for the second quarter fiscal year 2017 and expectations for fiscal 2017 being

547 views • 21 slides

More recommend