on the noisy gradient descent that generalizes as sgd
play

On the Noisy Gradient Descent that Generalizes as SGD Jingfeng Wu , - PowerPoint PPT Presentation

Feb 10, 2023 •535 likes •672 views

On the Noisy Gradient Descent that Generalizes as SGD Jingfeng Wu , Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, Zhanxing Zhu Johns Hopkins University, Missouri University of Science and Technology, Baidu Research, Styling AI, Peking

  1. On the Noisy Gradient Descent that Generalizes as SGD Jingfeng Wu , Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, Zhanxing Zhu Johns Hopkins University, Missouri University of Science and Technology, Baidu Research, Styling AI, Peking University

  2. <latexit sha1_base64="bS1Z0XerBUCN69KWD1FLEosGR7k=">ACynicbVHRStxAFJ3EWnVr21UfRlcWhTpklRBfRCkfHB4WuWjZLmEzu7g5OJmHmRghb/6eH+AP9Ds6iRF3Vy+EnDn3nHuHM1EmhUHPe3LcpQ/LH1dW1zqf1j9/+drd2Lw2a45DHgqU30bMQNSKBigQAm3mQaWRBJuorvfdf/mHrQRqfqDRQajhE2UGAvO0FJh9zHAKSALS9z3K/r9lLZnpD9oYAENUmuPNONQBihkDOWk2n0R7VU0CDq08eUqfhUuzJhoFoctXdGLWf+LZNZe/+4A31tY6+emzQ6rwm7P63tN0bfAb0GPtHUZdv8FcrzBRyYwZ+l6Go5JpFxC1QlyA5m9DJvA0ELFEjCjsom9ot8sE9Nxqu2nkDbsrKNkiTFElwnBqFns1+V5vmOP4eFQKleUIij8vGueSYkrN6Sx0MBRFhYwroW9K+VTZkND+9JzW5KiWdKxwfiLMbwF1z/7/mH/5Oqwd/arjWiVbJMdskt8ckTOyDm5JAPCnQPnrxM53L1wtVu45bPUdVrPFpkr9+E/w2XezQ=</latexit> <latexit sha1_base64="p6Gzhfmu2a6M/o0dA0E02QuYdk=">ACNXicbVDBThsxFPRSaGkKbdoe7GIKqWXaBchNRVCQnAoBw4gNSRSNl15nbeJFdu7st+iRsv+Ch/QH+gPcIU7B26IK79QZ5NDAx3J0mjePM3zxJkUFn3/1lt5sbr28tX69qbjc237+rvP5zZNDcOjyVqenFzIUGjoUEIvM8BULKEbTw5n8+45GCtS/QOnGQwUG2mRCM7QSVG9fdwMcQzIvtA9GiaG8SIoC13S0OYqKsReUP7UNAQpafNXJOguXdijesNv+RXocxIsSGP/O/0dRhejk6j+EA5TnivQyCWzth/4GQ4KZlBwCWUtzC1kjE/YCPqOaqbADorqhyX97JQhTVLjnkZaqf9uFExZO1WxcyqGY/t0NhP/N+vnmLQHhdBZjqD5PCjJcWUzuqiQ2GAo5w6wrgR7lbKx8zVhK7UpRQ1rUJqrpjgaQ3Pydl2K9hpfTt1DR2QOdbJ7JFmiQgX8k+OSInpEM4uSRX5JrceH+8O+/e5hbV7zFzkeyBO/xL6azrY=</latexit> <latexit sha1_base64="NO81viQx6ZAH+ykfVPko4iKqA=">ACDnicbVDLSsNAFJ3UV62vVJduBotQNyWRgrorunFZwT6gKWEynbZDJw9mbioh9B/8Abf6B+7Erb/gD/gdTtMsbOuBC4dz7uVcjhcJrsCyvo3CxubW9k5xt7S3f3B4ZJaP2yqMJWUtGopQdj2imOABawEHwbqRZMT3BOt4k7u535kyqXgYPEISsb5PRgEfckpAS65ZnrqOGg1w1YExA+LChWtWrJqVAa8TOycVlKPpmj/OIKSxzwKgijVs60I+imRwKlgs5ITKxYROiEj1tM0ID5T/TR7fYbPtTLAw1DqCQBn6t+LlPhKJb6nN30CY7XqzcX/vF4Mw+t+yoMoBhbQRdAwFhCPO8BD7hkFESiCaGS618xHRNJKOi2lL8JAsp6WLs1RrWSfuyZtdrNw/1SuM2r6iITtEZqiIbXaEGukdN1EIUPaEX9IrejGfj3fgwPherBSO/OUFLML5+ARZrnAU=</latexit> <latexit sha1_base64="WuEeC+IspeWG7WEBXAwi0QNv8w=">ACO3icbVDPSxtBFJ7VWmPUGvXoZWgQ0kvYVUFkWAv9pZCE4VsWGYnb5PBmdl5q0Ylvw3/hn2H+jVXnoueJBePHjv5Mehmj4Y+N73zeN74szKSz6/i9vYfHd0vl0kp5dW39w0Zlc6t09xwaPFUpuYqZhak0NBCgRKuMgNMxRIu4+vPY/3yBowVqf6Gwy6ivW1SARn6KiochYmhvEiGBXxiIY2V1EhQqHpeYSu7xvWi4oQB4DMtSBl7TYSJ3TKRPgpqlT9uj8pOg+CGag2vuyf3l8rzajymPYS3muQCOXzNpO4GfYLZhBwSWMymFuIWP8mvWh46BmCmy3mNw5oruO6dEkNe5pBP234mCKWuHyl2yqxgO7FtTP5P6+SYHULobMcQfOpUZJLikdh0Z7wgBHOXSAcSPcrpQPmAsOXbSvXNRwYlJ2wQRvY5gH7b16cFA/uoSOifTKpEd8pHUSEAOSYNckCZpEU7uyA/yQH56373f3pP3Z/p1wZvNbJNX5b38Bcgrsdc=</latexit> Stochastic gradient descent (SGD) n L ( ✓ ) = 1 Loss function X ` ( x i ; ✓ ) n i =1 1 X r θ ` ( x i ; ✓ t ) b i ∈ B t SGD z }| { θ t +1 = θ t � η g ( θ t ) ˜ = θ t � η r θ L ( θ t ) } � η (˜ g ( θ t ) � r θ L ( θ t )) | {z | {z } GD v sgd ( θ t ) (unbiased) gradient noise

  3. Noise matters! SGD >> GD • How? <= Still open… • Which? <= This work! CIFAR-10, ResNet-18, w/o weight decay, w/o data augmentation

  4. <latexit sha1_base64="kj+yFB6VfOGxOf9MuzV4MT7235s=">ACOXicbVDLSgNBEJz1bXxFPXoZDIeDLsSUEFB9OLBg4LRQDYs7OdzZDZBzO9gbDkZ/wNf8Cr3jx6EfHqDzh5KGosGCiquqme8lMpNr2szUxOTU9Mzs3X1hYXFpeKa6u3egkUxyqPJGJqvlMgxQxVFGghFqgEW+hFu/fdb3bzugtEjia+ym0IhYGIum4AyN5BWPOp6rw4Bu9gCZDv0mLoZA0/JZ2qRsqFnj5UOhdfDlesWSX7QHoOHFGpERGuPSKr26Q8CyCGLlkWtcdO8VGzhQKLqFXcDMNKeNtFkLd0JhFoBv54Jc9umWUgDYTZV6MdKD+3MhZpHU38s1kxLCl/3p98T+vnmHzoJGLOM0QYj4MamaSYkL7ldFAKOAou4YwroS5lfIWU4yjKfZXStQdhBRMc7fGsbJzV7ZqZQPryqlk9NRXNkg2ySbeKQfXJCzsklqRJO7sgDeSRP1r31Yr1Z78PRCWu0s05+wfr4BNijrFQ=</latexit> Which noise matters? v sgd ( θ ) = ˜ g ( θ ) � r θ L ( θ ) 1. Magnitude <= YES! (e.g., Jastrzkebski et al. 2017) 2. Covariance structure <= YES! (e.g., Zhu et al. 2018) 3. Distribution class <= ? No!!! (this work) Bernoulli? Gaussian? Levy?...

  5. <latexit sha1_base64="lb+H4UTjzf5Kv/JL3KQaDibfw=">ACNXicbVDLSgNBEJz1GeMr6tHLYBUNOxKwIiXoBePCkMZJcwO+mYIbMPZnolYcmv+Bv+gFe9e/AmufoLTmIEYywYKq6qZ7yYyk02vabNTe/sLi0nFnJrq6tb2zmtrZrOkoUhyqPZKTqPtMgRQhVFCihHitgS/hzu9ejfy7B1BaRGEF+zF4AbsPRVtwhkZq5kou9GLg2Ogdu9gBZM2Kl7og5UHv4kc4pCd0Wjo6HDRzebtgj0FniTMheTLBTM3dFsRTwIkUumdcOxY/RSplBwCYOsm2iIGe+ye2gYGrIAtJeOfzig+0Zp0XakzAuRjtXfGykLtO4HvpkMGHb0X28k/uc1EmyXvFSEcYIQ8u+gdiIpRnRUF20JZdqRfUMYV8LcSnmHKcbRlDqVEvTHIVlTjPO3hlSOy04xcL5bTFfvpxUlCG7ZI8cEIeckTK5JjekSjh5JM/khbxaT9a79WENv0fnrMnODpmC9fkFJx+riw=</latexit> <latexit sha1_base64="6fN/KzCDnCBe5BNfzP9VFrwepSQ=">ACVnicbZDfShtBFMYna6Mxrbray94MBsFSDLsiqBdCaG960QsFkyjZsMzOniSDs7PLzFkhLPtsvoY+QHvZvoE4SVb8kx4Y+M53vsMZflEmhUHPe6g5Kx/q2uN9ebHTxubW+72Ts+kuebQ5alM9VXEDEihoIsCJVxlGlgSehHNz9m8/4taCNSdYnTDIYJGysxEpyhtUL3OsAJIAsL/OaX9IxWLdIDGlhBg1zFoCPNOBTBWLM4LBaRkv7afw5/LZ/jty9e6La8tjcvuiz8SrRIVeh+yeIU54noJBLZszA9zIcFkyj4BLKZpAbyBi/YWMYWKlYAmZYzBGUdM86MR2l2j6FdO6+3ihYsw0iWwyYTgx72cz83+zQY6jk2EhVJYjKL4NMolxZTOeNJYaOAop1YwroX9K+UTZnmhpf7mSjKdH2laMP57DMuid9j2j9qnF0etzvcKUYN8Ibtkn/jkmHTIT3JOuoSTO/Kb/CX/ave1R6furC2iTq3a+UzelOM+AbQitZQ=</latexit> Intuition For quadratic loss, the generalization error x, θ T [ ` ( x ; ✓ T ) − ` ( x ; ✓ ∗ )] only depends on the first two moments of 𝜄 ! , which only depend on the first two moments of 𝑤(𝜄) . θ t +1 = θ t � η r θ L ( θ t ) } � η v ( θ t ) | {z Linear Noise matters! But noise class does not!!!

  6. <latexit sha1_base64="TAwr4vz/1WeH84dKCYFz3HPYwgM=">ACKHicbVDLSsNAFJ3UV62vqEs3g6UgqCWRg8Qim5cVrAPaEKZTCft0MkzEyEPIR/oY/4Fb/wJ1068LvcNJ2YVsPDBzOmcu593gRo1JZ1tgorKyurW8UN0tb2zu7e+b+QUuGscCkiUMWio6HJGUk6aipFOJAgKPEba3ug+9vPREga8ieVRMQN0IBTn2KktNQzT502RqznyEH/Bjpl6PgC4dTOUi+Dt9A7c3LR0pSfez2zbFWtCeAysWekDGZo9Mwfpx/iOCBcYak7NpWpNwUCUxI1nJiSWJEB6hAelqylFApJtOjspgRSt96IdCP67gRP07kaJAyiTQe1YCpIZy0cvF/7xurPwrN6U8ihXheBrkxwyqEOYNwT4VBCuWaIKwoHpXiIdI16J0j3MpQTIJKeli7MUalknromrXqtePtXL9blZRERyBY3ACbHAJ6uABNEATYPAC3sA7+DBejU/jyxhPvxaM2cwhmIPx/Qv7o6QE</latexit> <latexit sha1_base64="6nU86pnc4vVkxCH7hZrZS+zvU=">ACa3icbVHLSgMxFM2M7/qulMXQREqSpkRQUWEohsXLhSsCp0yZNLbNjSTGZI74jD0L/w0N/6AO3/AleljYdULISfnvgJEqlMOh5747NT0zOze/UFpcWl5ZLa+tP5gk0xzqPJGJfoqYASkU1FGghKdUA4sjCY9R72qQf3wGbUSi7jFPoRmzjhJtwRlaKiyroKNZKywC7AKyPg1uOJOV0WufXtAg0oz3AItJXQBSVl5C/3ysPKRBK0Fjr/9k6nAs64flXa/qDYP+Bf4Y7NZ2goPX91p+G5Y/bGOexaCQS2ZMw/dSbBZMo+AS+qUgM5DaDVkHGhYqFoNpFkNf+nTPMi3aTrQ9CumQ/VlRsNiYPI6sMmbYNb9zA/K/XCPD9mzECrNEBQfDWpnkmJCBybTltDAUeYWMK6F3ZXyLrNOov2KiSlxPhxSsb4v234Cx6Oqv5x9ezOnRJRjFPtsgOqRCfnJAauSa3pE4eSNfzpQz7Xy6G+6muz2Sus64ZoNMhLv3DRKv14=</latexit> <latexit sha1_base64="0bKvXi7R+xe/ISAKMPyZNSTCdio=">ACnicbZHPatAEMZXStuk7j8nOfay1BTSQ4wUQtMeAqGF0kNKU6gdg2XEaDWl6xWYncUMEIPk8fqC/Q5spbdNko6sPAxv5md4ZukVNJSEPzy/K1Hj59s7ztPXv+4uWr/u7e2BaVETgShSrMJAGLSmockSFk9Ig5InCy+Tq84pfXqOxstA/aVniLIdMy7kUQC4V92+iSqdoEgMC6+s4slnKDyJaIMG7hp9yfpdHJFWKPtXcNjlmYE0rte0Of9b1jvlXcSjcwHqD+eRSAvq/jR2vN2mifuDYBi0wR+KcCMGbBMXcf93lBaiylGTUGDtNAxKmtVgSAqFTS+qLJYgriDqZMacrSzurWy4W9dJuXzwrinibfZux015NYu8RV5kALe5+tkv9j04rmH2a1GVFqMV60LxSnAq+ugtPpUFBaukECPdrlwswJlB7nqdKfmyHdJzxoT3bXgoxkfD8Hj48cfx4OzTxqId9pq9YQcsZCfsjH1lF2zEhLftHXrvROf+1/8b/73danvbXr2WSf8yS3AuM4x</latexit> <latexit sha1_base64="tpgXxRfVOTUgPLu6xgEHKXJFoQ=">ACMXicbVDLSiQxFE35Gm1fPbqcTbARdNUSYPOTsbNLGahMK1CV9PcSt3qDqaSIrk10BT1I/Mb/oBb5w/cDS7c+BOmHwtfBwIn59zLSU5SKOkoDB+ChcWl5ZUvq2uN9Y3Nre3m151LZ0orsCuMvY6AYdKauySJIXhUXIE4VXyc3ZxL/6g9ZJo3/TuMB+DkMtMymAvDRoduKhXRQxTRCgprHvwSog9ntkMciNcTjzIKorSdWw0Jsmg2Qrb4RT8I4nmpMXmOB80n+LUiDJHTUKBc70oLKhfgSUpFNaNuHRYgLiBIfY81ZCj61fT39V83yspz4z1RxOfq83KsidG+eJn8yBRu69NxE/83olZSf9SuqiJNRiFpSVipPhk6p4Ki0KUmNPQFjp38rFCHwX5At9k5KPpyENX0z0voaP5PKoHXa3y86rdMf84pW2Te2xw5YxI7ZKfvJzlmXCfaX3bF79i+4DR6C/8HjbHQhmO/sjcInl8AhGeq6w=</latexit> <latexit sha1_base64="i9wUYTmMfkengD4WERgFodbQRU=">ACInicbVDNSsNAGNzUv1r/qh69LJaCF0siBfUgFL14rGB/oAlhs9m0SzebsLsRQsgT+Bq+gFd9A2/iSfDsc7hNi9jWgYXZme9jdseLGZXKND+N0srq2vpGebOytb2zu1fdP+jKBGYdHDEItH3kCSMctJRVDHSjwVBocdIzxvfTPzeAxGSRvxepTFxQjTkNKAYKS251brdxYi5thz68Aravd/LKbQDgXBm5RnP3WrNbJgF4DKxZqQGZmi71W/bj3ASEq4wQ1IOLDNWToaEopiRvGInksQIj9GQDTlKCTSyYrv5LCuFR8GkdCHK1iofzcyFEqZhp6eDJEayUVvIv7nDRIVXDgZ5XGiCMfToCBhUEVw0g30qSBYsVQThAXVb4V4hHQLSjc4lxKmRUhF2Mt1rBMumcNq9m4vGvWtezisrgCByDE2CBc9ACt6ANOgCDR/AMXsCr8WS8Ge/Gx3S0ZMx2DsEcjK8f4pKj0A=</latexit> <latexit sha1_base64="4RaNxqViWuH516fg4e9gd2nVKJQ=">ACLHicbVDLSgMxFM3UV62vqks3wSLUTZ2RgrorunHhoJ9QKeUO5m0Dc08SO4IZehn+Bv+gFv9Azcibvsdpo+FbT0QODnXk5yvFgKjb9ZWXW1jc2t7LbuZ3dvf2D/OFRXUeJYrzGIhmpgeaSxHyGgqUvBkrDoEnecMb3E38xjNXWkThEw5j3g6gF4quYIBG6uQv3J4Cv5O62OcI+o+MJDF2e2cusyPkLoNo5kR3fNHnXzBLtlT0FXizEmBzFHt5MeuH7Ek4CEyCVq3HDvGdgoKBZN8lHMTzWNgA+jxlqEhBFy30+nHRvTMKD7tRsqcEOlU/buRQqD1MPDMZADY18veRPzPayXYvW6nIowT5CGbBXUTSTGik5aoLxRnKIeGAFPCvJWyPihgaLpcSAmG05CcKcZrmGV1C9LTrl081guVG7nFWXJCTklReKQK1Ih96RKaoSRF/JG3smH9Wp9Wt/Wz2w0Y813jskCrPEvGhSopA=</latexit> A closer look at the noise of SGD v sgd ( θ ) | {z } = ˜ |{z} � r θ L ( θ ) g ( θ ) } = r θ L ( θ ) · V sgd | {z |{z} r θ L ( θ ) · 1 Gradient Sampling r θ L ( θ ) · W sgd noise noise n • Gradient matrix r θ L ( ✓ ) = ( r θ ` ( x 1 ; ✓ ) , . . . , r θ ` ( x n , ✓ )) W sgd : #1 • Sampling vector b = b, #0 = n − b V sgd = W sgd − 1 • Sampling noise n

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap

CS6501: Deep Learning for Visual Recognition Stochastic Gradient Descent (SGD) Todays Class Stochastic Gradient Descent (SGD) SGD Recap Regression vs Classification Generalization / Overfitting / Underfitting Regularization

633 views • 30 slides
Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to

Fitting Neural Networks Gradient Descent and Stochastic Gradient Descent CS109A Introduction to Data Science Pavlos Protopapas, Kevin Rader and Chris Tanner New requirement for the final project: For the first time ever, researchers who submit

619 views • 34 slides
1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

1 Gradient descent with fixed step In this section, we discuss a gradient descent method with

MAP562 Optimal design of structures by Gr egoire Allaire, Thomas Wick Ecole Polytechnique Academic year 2016-2017 Gradient descent and Newtons method for minimization in a PDE / function space context We recapitulate the basic steps to

327 views • 5 slides
Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade

Machine Learning (CSE 446): Gradient Descent and Stochastic Gradient Descent Sham M Kakade 2018 c University of Washington cse446-staff@cs.washington.edu 1 / 12 Announcements Midterm: Weds, Feb 7th. Policies: You may use a single

333 views • 18 slides
CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University

CS 6316 Machine Learning Gradient Descent Yangfeng Ji Department of Computer Science University of Virginia Overview 1. Gradient Descent 2. Stochastic Gradient Descent 3. SGD with Momentum 4. Adaptive Learning Rates 1 Gradient Descent

767 views • 66 slides
LOGISTIC REGRESSION, GRADIENT LOGISTIC REGRESSION, GRADIENT DESCENT, NEWTON DESCENT, NEWTON

LOGISTIC REGRESSION, GRADIENT LOGISTIC REGRESSION, GRADIENT DESCENT, NEWTON DESCENT, NEWTON

LOGISTIC REGRESSION, GRADIENT LOGISTIC REGRESSION, GRADIENT DESCENT, NEWTON DESCENT, NEWTON Matthieu R Bloch Thursday, January 30, 2020 1 LOGISTICS LOGISTICS TAs and Office hours Monday: Mehrdad (TSRB 523a) - 2pm-3:15pm Tuesday: TJ (VL

299 views • 14 slides
Gradient Descent Michail Michailidis &amp; Patrick Maiden Outline

Gradient Descent Michail Michailidis &amp; Patrick Maiden Outline

Gradient Descent Michail Michailidis &amp; Patrick Maiden Outline Mo4va4on Gradient Descent Algorithm Issues &amp; Alterna4ves Stochas4c Gradient Descent

839 views • 29 slides
Learning to learn by gradient descent by gradient descent Liyan Jiang July 18, 2019 1

Learning to learn by gradient descent by gradient descent Liyan Jiang July 18, 2019 1

Learning to learn by gradient descent by gradient descent Liyan Jiang July 18, 2019 1 Introduction The general aim of machine learning is always learning the data by itself, with as less human efforts as possible. Then it comes to the focus

396 views • 10 slides
Applied Machine Learning Gradient Descent Methods Siamak Ravanbakhsh COMP 551 (Fall 2020)

Applied Machine Learning Gradient Descent Methods Siamak Ravanbakhsh COMP 551 (Fall 2020)

Applied Machine Learning Gradient Descent Methods Siamak Ravanbakhsh COMP 551 (Fall 2020) Learning objectives Basic idea of gradient descent stochastic gradient descent method of momentum using an adaptive learning rate sub-gradient

568 views • 34 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. MLSS 2020 Aaron Mishkin, amishkin@cs.ubc.ca 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of

634 views • 21 slides
Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS

Painless Stochastic Gradient Descent : Interpolation, Line-Search, and Convergence Rates. NeurIPS 2019 Aaron Mishkin 1 21 Stochastic Gradient Descent: Workhorse of ML? Stochastic gradient descent (SGD) is today one of the main

1.22k views • 21 slides
Conjugate Gradient (CG) Majid Lesani Alireza Masoum Overview Backpropagation Gradient

Conjugate Gradient (CG) Majid Lesani Alireza Masoum Overview Backpropagation Gradient

Conjugate Gradient (CG) Majid Lesani Alireza Masoum Overview Backpropagation Gradient Descent Quadratic Forms Gradient Descent in Quadratic Forms Eigen vectors and values Gradient Descent Convergence Conjugate

1.1k views • 50 slides
Conjugate gradient training algorithm Steepest descent algorithm Definitions: So far: j

Conjugate gradient training algorithm Steepest descent algorithm Definitions: So far: j

Conjugate gradient training algorithm Steepest descent algorithm Definitions: So far: j Heuristic improvements to gradient descent (momentum) w j = weight vector at step . Steepest descent training algorithm [ ] E w j j

376 views • 20 slides
Gradient descent revisited Geoff Gordon &amp; Ryan Tibshirani Optimization 10-725 / 36-725 1

Gradient descent revisited Geoff Gordon &amp; Ryan Tibshirani Optimization 10-725 / 36-725 1

Gradient descent revisited Geoff Gordon &amp; Ryan Tibshirani Optimization 10-725 / 36-725 1 Gradient descent Recall that we have f : R n R , convex and differentiable, want to solve x R n f ( x ) , min i.e., find x such that f ( x

329 views • 31 slides
Feedforward Networks Gradient Descent Learning and Backpropagation Christian Jacob CPSC 565

Feedforward Networks Gradient Descent Learning and Backpropagation Christian Jacob CPSC 565

Feedforward Networks Gradient Descent Learning and Backpropagation Christian Jacob CPSC 565 Winter 2003 Department of Computer Science University of Calgary Canada Learning by Gradient Descent Definition of the Learning Problem Let us

538 views • 25 slides
Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression

Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression

Learning From Data Lecture 9 Logistic Regression and Gradient Descent Logistic Regression Gradient Descent M. Magdon-Ismail CSCI 4100/6100 recap: Linear Classification and Regression The linear signal: s = w t x Good Features are Important

272 views • 23 slides
Realistic noise simulation in LArSoft Andrea Scarpelli (CNRS/APC) Outline 3x1x1 noise patterns

Realistic noise simulation in LArSoft Andrea Scarpelli (CNRS/APC) Outline 3x1x1 noise patterns

Realistic noise simulation in LArSoft Andrea Scarpelli (CNRS/APC) Outline 3x1x1 noise patterns Simulation strategy Results Summary of the model FFT MODEL CREATION FFT based on 3x1x1 729-0 noise run (the pedestal reference for this

382 views • 6 slides
Energy Infrastructure Package Regulation on guidelines for trans-European energy infrastructure

Energy Infrastructure Package Regulation on guidelines for trans-European energy infrastructure

Energy Infrastructure Package Regulation on guidelines for trans-European energy infrastructure &amp; Connecting Europe Facility European Commission DG Energy Internal Market I: Networks and Regional initiatives CONTENT Do we need more

528 views • 21 slides
Computational Models of Discourse: Introduction to Discourse: Coherence and Cohesion, Lexical

Computational Models of Discourse: Introduction to Discourse: Coherence and Cohesion, Lexical

Computational Models of Discourse: Introduction to Discourse: Coherence and Cohesion, Lexical Chains Caroline Sporleder Universit at des Saarlandes Sommersemester 2009 29.04.2009 Caroline Sporleder csporled@coli.uni-sb.de Computational

1.33k views • 95 slides
HOWTO Reminders Basic Vulnerabilities and their Vulnerabilities Exploitation Security Samuel

HOWTO Reminders Basic Vulnerabilities and their Vulnerabilities Exploitation Security Samuel

HOWTO Basic Vulnerabilities and their Exploitation Samuel Angebault HOWTO Reminders Basic Vulnerabilities and their Vulnerabilities Exploitation Security Samuel Angebault staphylo@lse.epita.fr http://lse.epita.fr/ July 18, 2013 Table

969 views • 57 slides
Lecture 4: Linear filters Tuesday, Sept 11 Many slides by (or adapted from) D. Forsyth, Y.

Lecture 4: Linear filters Tuesday, Sept 11 Many slides by (or adapted from) D. Forsyth, Y.

Lecture 4: Linear filters Tuesday, Sept 11 Many slides by (or adapted from) D. Forsyth, Y. Boykov, L. Davis, W. Freeman, M. Hebert, D. Kreigman, P. Duygulu Image neighborhoods Q: What happens if we reshuffle all pixels within the image?

897 views • 72 slides
2008, nature Shlens et al 09 Pillow et al, 2008 Pillow et al, 2008 Whats role of coupling

2008, nature Shlens et al 09 Pillow et al, 2008 Pillow et al, 2008 Whats role of coupling

2008, nature Shlens et al 09 Pillow et al, 2008 Pillow et al, 2008 Whats role of coupling / correlations? No strong role in predicting single-cell spike probabilities Whats role of coupling / correlations? Decoding

326 views • 8 slides
Noise-adaptive Margin- based Active Learning, and Yining Wang , Aarti Singh Carnegie Mellon

Noise-adaptive Margin- based Active Learning, and Yining Wang , Aarti Singh Carnegie Mellon

Noise-adaptive Margin- based Active Learning, and Yining Wang , Aarti Singh Carnegie Mellon University Lower Bounds Machine Learning: the setup The machine learning problem Each data point consists of data and label ( x

421 views • 24 slides
Neural Networks for Machine Learning Lecture 9a Overview of ways to improve generalization

Neural Networks for Machine Learning Lecture 9a Overview of ways to improve generalization

Neural Networks for Machine Learning Lecture 9a Overview of ways to improve generalization Geoffrey Hinton Nitish Srivastava, Kevin Swersky Tijmen Tieleman Abdel-rahman Mohamed Reminder: Overfitting The training data contains information

1.17k views • 39 slides

More recommend