Re Recent nt trends nds in n Aut Autom omated ed Machi chine - PowerPoint PPT Presentation

Re Recent nt trends nds in n Aut Autom omated ed Machi chine ne Le Lear arni ning ng (AutoML) L) Su Summer r semester r 201 019 Ti Tim Meinhardt rdt and d Pro rof. Dr. r. Laura ra Leal-Ta Taixé

Ou Outline ne • What is AutoML? • Organization – General information – Course and paper matching – Presentations • Paper preview 2 24.01.19 AutoML seminar - Tim Meinhardt

Wha What is AutoML? L? Machine and Deep Learning Inputs • Tasks (Classification, Regression, etc.) • Datasets (research, real, non-vision) 3 24.01.19 AutoML seminar - Tim Meinhardt

Wha What is AutoML? L? Learn a task/dataset specific model: • Architecture design • Data processing • Optimization Hy Hyperparame meter optimi mization! 4 24.01.19 AutoML seminar - Tim Meinhardt

Wha What is AutoML? L? • Enhance progress on existing inputs • Produce state-of-the-art outputs for new inputs – Research – Industry Machine learning experts (or graduate student descent)! Automated Mach chine Learning (AutoML) 5 24.01.19 AutoML seminar - Tim Meinhardt

Ho How to Aut utoML ML? Classic optimization: • Grid or random searches • Bayesian optimization (TPE, Spearmint, SMAC, etc.) Learning to learn or Meta Learning 6 24.01.19 AutoML seminar - Tim Meinhardt

Me Meta Learning Leverage power of learning methods to improve learning: • Few shot learning • Pretraining on ImageNet • Multi-task initialization learning • Fast Reinforcement Learning • Learning architectures AutoML • Learning optimizers 7 24.01.19 AutoML seminar - Tim Meinhardt

Organi Or nization General information Website: https://dvl.in.tum.de/teaching/automl_ss19/ • Contact: tim.meinhardt@tum.de • Room: MI 02.09.023 • Time: 12 participants -> 6 sessions • Attendance is mandatory! • Schedule: 25 th January 1 – 3 pm Pre-course meeting: • 25 th April 2 – 4 pm Paper matching: • Presentations: Thursdays 2 - 4 pm, TBD • 8 24.01.19 AutoML seminar - Tim Meinhardt

Ma Match ching Course matching (https://matching.in.tum.de/) • – See FAQ for details 8 th - 13 th February – Registration period: – Preference: I2DL or DL4CV grade (contact us if external student) – Announcement: 20 th February Paper matching • – Study our list of suggested papers (website 8 th February) – Propose own paper until 20 th April – On the 25 th April • Match paper based on preferences (toss a coin if necessary) • Fix dates for the presentations 9 24.01.19 AutoML seminar - Tim Meinhardt

Bef Befor ore e th the e presen esenta tati tion on • Read and work through the paper • Note questions and difficulties Three weeks before: Arrange meeting to discuss and clarify paper One week before: Arrange meeting to discuss slides 10 24.01.19 AutoML seminar - Tim Meinhardt

Pr Present ntation • Duration: 20 minutes talk + 10 minutes discussions • Finish talk on time! • Explain in own words • Complement paper content with additional material and explanations (from an I2DL perspective) • Rule of thumb: 1-2 minutes per slide, i.e., 10-20 slides • Submit PDF until 1 week after presentation 11 24.01.19 AutoML seminar - Tim Meinhardt

Pa Pape per pr previ view As Asyn ynchron onou ous Methods ods for or Deep Reinforcement ng . Mnih et al. Le Learni ning • Q-Learning • Advantage Actor-Critic 12 24.01.19 AutoML seminar - Tim Meinhardt

Pa Pape per pr previ view Pr Proxi ximal Po Policy Optimization Algorithms . Schulman et al. • (Proximal)Policy gradient methods • Trust region methods 13 24.01.19 AutoML seminar - Tim Meinhardt

Pape Pa per pr previ view Ne Neural Architecture Search wi with Reinforcement ng . Zoph et al. Le Learni ning • Recurrent network to predict architectures (NAS) • Trained with RL 14 24.01.19 AutoML seminar - Tim Meinhardt

Pa Pape per pr previ view Le Learni ning ng Trans nsferable Archi hitectures for Scalable Image on . Zoph et al. Rec Recog ognition • Extension of NAS with new architecture search space • Applicable to large datasets 15 24.01.19 AutoML seminar - Tim Meinhardt

Pa Pape per pr previ view Le Learni ning ng to learn n by gradient nt descent nt by gradient nt descent . Andrychowicz et al. de • Design of optimizer casted as a learning problem • Generalizes to unseen tasks 16 24.01.19 AutoML seminar - Tim Meinhardt

Pa Pape per pr previ view Se Searching for Activation Fu Functions . Ramachandran et al. • Apply reinforcement learning to discover new activation • New activation function Swish 17 24.01.19 AutoML seminar - Tim Meinhardt

Pa Pape per pr previ view Le Learni ning ng Step Size Cont ntrollers for Robust Neural work Training . Daniel et al. Ne Netwo • Learned learning rate scheduler • Reinforcement Learning 18 24.01.19 AutoML seminar - Tim Meinhardt