learning how to active learn a deep reinforcement
play

Learning how to Active Learn: A Deep Reinforcement Learning Approach - PowerPoint PPT Presentation

Oct 10, 2023 •148 likes •708 views

Learning how to Active Learn: A Deep Reinforcement Learning Approach Meng Fang, Yuan Li, Trevor Cohn The University of Melbourne Presenter: Jialin Song April 05, 2018 April 05, 2018 1 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine

  1. Learning how to Active Learn: A Deep Reinforcement Learning Approach Meng Fang, Yuan Li, Trevor Cohn The University of Melbourne Presenter: Jialin Song April 05, 2018 April 05, 2018 1 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  2. Overview Introduction 1 Model 2 Algorithms 3 Numerical Experiments 4 April 05, 2018 2 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  3. Introduction: Active Learning 1 Annotation: April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  4. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  5. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) ⋄ then we can train a supervised learning model φ (classifier) April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  6. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) ⋄ then we can train a supervised learning model φ (classifier) ⋄ we hope to maximize the accuracy of the classification model April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  7. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) ⋄ then we can train a supervised learning model φ (classifier) ⋄ we hope to maximize the accuracy of the classification model 2 Active learning: April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  8. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) ⋄ then we can train a supervised learning model φ (classifier) ⋄ we hope to maximize the accuracy of the classification model 2 Active learning: ⋄ there is high cost annotating every sentence April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  9. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) ⋄ then we can train a supervised learning model φ (classifier) ⋄ we hope to maximize the accuracy of the classification model 2 Active learning: ⋄ there is high cost annotating every sentence ⋄ how to select raw data to add labels in order to maximize the accuracy of the classification model April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  10. Introduction: Active Learning 1 Annotation: ⋄ select a subset of data to annotate from a large unlabelled dataset (adding labels) ⋄ then we can train a supervised learning model φ (classifier) ⋄ we hope to maximize the accuracy of the classification model 2 Active learning: ⋄ there is high cost annotating every sentence ⋄ how to select raw data to add labels in order to maximize the accuracy of the classification model ⋄ active learning becomes a sequential decision: as each sentence arrives, annotate it or not (our action) April 05, 2018 3 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  11. Introduction: MDP 1 Markov Decision Process (MDP): April 05, 2018 4 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  12. Introduction: MDP 1 Markov Decision Process (MDP): ⋄ a framework to model a sequential decision process April 05, 2018 4 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  13. Introduction: MDP 1 Markov Decision Process (MDP): ⋄ a framework to model a sequential decision process ⋄ in each decision stage, agent observes state variables ( s ) and take a action ( a ) to maximize its current payoff April 05, 2018 4 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  14. Introduction: MDP 1 Markov Decision Process (MDP): ⋄ a framework to model a sequential decision process ⋄ in each decision stage, agent observes state variables ( s ) and take a action ( a ) to maximize its current payoff ⋄ after taking the action, a reward associated with the action and state ( r ( s, a ) ) is generated and current state transits to next state April 05, 2018 4 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  15. Introduction: MDP 1 Markov Decision Process (MDP): ⋄ a framework to model a sequential decision process ⋄ in each decision stage, agent observes state variables ( s ) and take a action ( a ) to maximize its current payoff ⋄ after taking the action, a reward associated with the action and state ( r ( s, a ) ) is generated and current state transits to next state ⋄ agent aims maximizing the expected value of rewards over all stages April 05, 2018 4 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  16. Introduction: Bellman Equation 1 The dynamics of MDP can be modeled in Bellman equations April 05, 2018 5 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  17. Introduction: Bellman Equation 1 The dynamics of MDP can be modeled in Bellman equations ⋄ Bellman equation 1: value function � � � P ss ′ ( a ) J ( s ′ ) J ( s ) = max r ( s, a ) + α ¯ a s ′ a ∗ s = argmax J ( s ) April 05, 2018 5 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  18. Introduction: Bellman Equation 1 The dynamics of MDP can be modeled in Bellman equations ⋄ Bellman equation 1: value function � � � P ss ′ ( a ) J ( s ′ ) J ( s ) = max r ( s, a ) + α ¯ a s ′ a ∗ s = argmax J ( s ) ⋄ Bellman equation 2 (more common!): Q-function � Q ( s ′ , u ) Q ( s, a ) = ¯ r ( s, a ) + α P ss ′ ( a ) max u s ′ a ∗ s = argmax Q ( s, a ) April 05, 2018 5 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  19. Introduction: Bellman Equation 1 The dynamics of MDP can be modeled in Bellman equations ⋄ Bellman equation 1: value function � � � P ss ′ ( a ) J ( s ′ ) J ( s ) = max r ( s, a ) + α ¯ a s ′ a ∗ s = argmax J ( s ) ⋄ Bellman equation 2 (more common!): Q-function � Q ( s ′ , u ) Q ( s, a ) = ¯ r ( s, a ) + α P ss ′ ( a ) max u s ′ a ∗ s = argmax Q ( s, a ) ⋄ where ¯ r ( s, a ) is the expected reward, P ss ′ ( a ) is the transition probability from state s to s ′ , α is the discount of reward April 05, 2018 5 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  20. Q-Learning 1 If P ss ′ ( a ) is known, then solve the Bellmen equations (VI/PI) to get the optimal policy. There is no need to ’learn’!!! April 05, 2018 6 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  21. Q-Learning 1 If P ss ′ ( a ) is known, then solve the Bellmen equations (VI/PI) to get the optimal policy. There is no need to ’learn’!!! 2 If P ss ′ ( a ) is not known, then how to compute Q-function becomes a learning problem April 05, 2018 6 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  22. Q-Learning 1 If P ss ′ ( a ) is known, then solve the Bellmen equations (VI/PI) to get the optimal policy. There is no need to ’learn’!!! 2 If P ss ′ ( a ) is not known, then how to compute Q-function becomes a learning problem 3 Q-learning: April 05, 2018 6 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  23. Q-Learning 1 If P ss ′ ( a ) is known, then solve the Bellmen equations (VI/PI) to get the optimal policy. There is no need to ’learn’!!! 2 If P ss ′ ( a ) is not known, then how to compute Q-function becomes a learning problem 3 Q-learning: � � ⋄ Q t +1 ( s t , a t ) = (1 − ǫ t ) Q t ( s t , a t ) + ǫ t r ( s t , a t ) + α max u Q t ( s t +1 , u ) ¯ April 05, 2018 6 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  24. Q-Learning 1 If P ss ′ ( a ) is known, then solve the Bellmen equations (VI/PI) to get the optimal policy. There is no need to ’learn’!!! 2 If P ss ′ ( a ) is not known, then how to compute Q-function becomes a learning problem 3 Q-learning: � � ⋄ Q t +1 ( s t , a t ) = (1 − ǫ t ) Q t ( s t , a t ) + ǫ t r ( s t , a t ) + α max u Q t ( s t +1 , u ) ¯ ⋄ where t is iteration and ǫ t is the learning rate April 05, 2018 6 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  25. Q-Learning 1 If P ss ′ ( a ) is known, then solve the Bellmen equations (VI/PI) to get the optimal policy. There is no need to ’learn’!!! 2 If P ss ′ ( a ) is not known, then how to compute Q-function becomes a learning problem 3 Q-learning: � � ⋄ Q t +1 ( s t , a t ) = (1 − ǫ t ) Q t ( s t , a t ) + ǫ t r ( s t , a t ) + α max u Q t ( s t +1 , u ) ¯ ⋄ where t is iteration and ǫ t is the learning rate ⋄ In practice, above is useless: | S | × | A | is huge April 05, 2018 6 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  26. Deep Q-Learning 1 Deep Q-learning: April 05, 2018 7 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

  27. Deep Q-Learning 1 Deep Q-learning: ⋄ use the output of a DNN parametrized by θ , i.e., f θ ( s, u ) to approximate Q ( s, a ) : April 05, 2018 7 / 17 Meng Fang, Yuan Li, Trevor Cohn CS 546 Machine Learning in NLP

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree

Deep Reinforcement Learning [Mastering the Game of Go with Deep Reinforcement Learning and Tree Search, Nature 2016] CS 486/686 University of Waterloo Lecture 21: July 12, 2017 Outline AlphaGo Supervised Learning of Policy Networks

540 views • 15 slides
Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays

Inverse Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Todays Lecture 1. So far: manually design reward function to define a task 2. What if we want to learn the reward function from observing an expert, and

650 views • 33 slides
1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement

1 Deep Reinforcement Learning Qianqian Li, Nayeon Koong, Langtian He What is deep reinforcement learning? Agent/Actor + Action + Environment + State + Reward How does reinforcement learning work?

792 views • 31 slides
Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and

Deep Neural Networks and Deep Reinforcement Learning Deep Neural Networks and Deep Reinforcement Learning Deep Learning, Goodfellow, Bengio and Courville [chapt. 6,7,8]; AIMA [sect. 21.1-21.3]; Sutton and Barto, Reinforcement Learning: an

527 views • 35 slides
Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature

Deep Reinforcement Learning [Human-Level Control through deep reinforcement learning, Nature 2015] CS 486/686 University of Waterloo Lecture 20: July 10, 2017 Outline Value Function Approximation Linear approximation Neural

706 views • 19 slides
Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning

Reproducibility and Replicability in Deep Reinforcement Learning (and Other Deep Learning Methods) Peter Henderson Statistical Society of Canada Annual Meeting 2018 Contributors Riashat Islam Phil Bachman Doina Precup David Meger Joelle

850 views • 37 slides
Deep Hep Reading Group 1611.05763 Learning To Reinforcement Learn 1611.02779 SchemaAc

Deep Hep Reading Group 1611.05763 Learning To Reinforcement Learn 1611.02779 SchemaAc

Deep Hep Reading Group 1611.05763 Learning To Reinforcement Learn 1611.02779 SchemaAc Approach for solving Markov Decision Process Agent interacts with environment Takes acAons to move from one state to another Is rewarded or

623 views • 19 slides
Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning

Deep he(a)p, big feat arXiv:1707.06887 A Distributional Perspective on Reinforcement Learning arXiv:1702.08165 Reinforcement Learning with Deep Energy-Based Policies 1 / 25 Reinforcement Learning Environment Action Reward Interpreter State

531 views • 25 slides
Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1.

Introduction to Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 1 is due next Wednesday! Remember that Monday is a holiday, so no office hours 2. Remember to start forming final project

877 views • 48 slides
Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey

Advanced Model-Based Reinforcement Learning CS 294-112: Deep Reinforcement Learning Sergey Levine Class Notes 1. Homework 3 is extended by one week, to Wednesday after next Todays Lecture 1. Managing overfitting in model-based RL

686 views • 40 slides
Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning

Deep Reinforcement Learning and Complex Environments Raia Hadsell End-to-end Deep Learning for robots? slide from V. Vanhoucke End-to-end Deep Learning for robots? 2010 : Speech Recognition Audio Acoustic Model Phonetic Model

1.05k views • 65 slides
CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe

DeepLoco by X. B. Peng, G. Berseth & M. van de Panne CMP784 DEEP LEARNING Lecture #12 Deep Reinforcement Learning Aykut Erdem // Hacettepe University // Spring 2018 Neural Face by Taehoon Kim Previously on CMP784 Generative

895 views • 71 slides
Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics:

Deep Reinforcement Learning for Robotics: Frontiers and Beyond Shixiang (Shane) Gu ( ) 2018.5.27 01 Deep RL: successes and limitations Computation-Constrained Data-Constrained

634 views • 21 slides
Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina

Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Introduction to Deep Reinforcement Learning and Control Spring 2019, CMU 10-403 Katerina Fragkiadaki Course Logistics Course website : all you need to

1.13k views • 88 slides
Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement

Reinforcement Learning Q-Learning Deep Q-Learning on Atari Reinforcement Learning Timothy Chou Charlie Tong Vincent Zhuang April 19, 2016 Reinforcement Learning Q-Learning Deep Q-Learning on Atari Table of Contents Reinforcement Learning

938 views • 63 slides
Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3.

Deep Reinforcement Learning 1 Outline 1. Overview of Reinforcement Learning 2. Policy Search 3. Policy Gradient and Gradient Estimators 4. Q-prop: Sample Efficient Policy Gradient and an Off-policy Critic 5. Model Based Planning in Discrete

769 views • 53 slides
Spring Framework 2.5: New and Notable Ben Alex, Principal Software Engineer, SpringSource

Spring Framework 2.5: New and Notable Ben Alex, Principal Software Engineer, SpringSource

Spring Framework 2.5: New and Notable Ben Alex, Principal Software Engineer, SpringSource GOAL> Learn whats new in Spring 2.5 and why it matters to you 2 springsource.com Agenda Goals of Spring 2.5 Support for new platforms

612 views • 45 slides
Gradual Typing with Inference Jeremy Siek University of Colorado at Boulder joint work with

Gradual Typing with Inference Jeremy Siek University of Colorado at Boulder joint work with

Gradual Typing with Inference Jeremy Siek University of Colorado at Boulder joint work with Manish Vachharajani Overview Motivation Background Gradual Typing Unification-based inference Exploring the Solution Space Type

1.25k views • 83 slides
TRECVID 2007 Collaborative Annotation using Active Learning Georges Qunot Multimedia

TRECVID 2007 Collaborative Annotation using Active Learning Georges Qunot Multimedia

TRECVID 2007 Collaborative Annotation using Active Learning Georges Qunot Multimedia Information Retrieval Group L aboratoire d' I nformatique de G renoble November 5, 2007 1 Outline Active learning. Previous work: evaluation of

381 views • 14 slides
Models of Annotation (II) Bob Carpenter, LingPipe, Inc. Massimo Poesio, Uni. Trento LREC 2010

Models of Annotation (II) Bob Carpenter, LingPipe, Inc. Massimo Poesio, Uni. Trento LREC 2010

Models of Annotation (II) Bob Carpenter, LingPipe, Inc. Massimo Poesio, Uni. Trento LREC 2010 (Malta) Mechanical Turk Examples (Carpenter, Jamison and Baldwin, 2008) Amazons Mechanical Turk Crowdsourcing Data Collection

1.12k views • 89 slides
Annotating Expressions of Opinion and Emotion in the Italian Content Annotation Bank (I-CAB)

Annotating Expressions of Opinion and Emotion in the Italian Content Annotation Bank (I-CAB)

About I-CAB Annotating private states Inter-annotator agreement (IAA) Conclusion: problems with the markup language Annotating Expressions of Opinion and Emotion in the Italian Content Annotation Bank (I-CAB) Andrea Esuli, Fabrizio Sebastiani

608 views • 21 slides
USING ADVENE TO USING ADVENE TO ACCOMPANY RESEARCH IN ACCOMPANY RESEARCH IN AUDIOVISUAL DIGITAL

USING ADVENE TO USING ADVENE TO ACCOMPANY RESEARCH IN ACCOMPANY RESEARCH IN AUDIOVISUAL DIGITAL

USING ADVENE TO USING ADVENE TO ACCOMPANY RESEARCH IN ACCOMPANY RESEARCH IN AUDIOVISUAL DIGITAL AUDIOVISUAL DIGITAL HUMANITIES HUMANITIES Olivier Aubert - www.olivieraubert.net Olivier Aubert - www.olivieraubert.net FOSDEM 2020 FOSDEM

447 views • 23 slides
Fine-Grained Temporal Relation Extraction Siddharth Vashishtha Benjamin Van Durme Aaron

Fine-Grained Temporal Relation Extraction Siddharth Vashishtha Benjamin Van Durme Aaron

Fine-Grained Temporal Relation Extraction Siddharth Vashishtha Benjamin Van Durme Aaron Steven White University of Rochester Johns Hopkins University University of Rochester Data and code available at: http://decomp.io Overarching

1.57k views • 130 slides
HORAE: an annotated dataset of books of hours Mlodie Boillet, Marie-Laurence Bonhomme,

HORAE: an annotated dataset of books of hours Mlodie Boillet, Marie-Laurence Bonhomme,

Horae project Pages selection process Annotation results Document layout analysis HORAE: an annotated dataset of books of hours Mlodie Boillet, Marie-Laurence Bonhomme, Dominique Stutzmann, Christopher Kermorvant Teklia SAS, Paris, France

267 views • 25 slides

More recommend