A Quick Look at the Reinforcement Learning course A. LAZARIC ( - - PowerPoint PPT Presentation

▶
a quick look at the reinforcement learning course
SMART_READER_LITE
LIVE PREVIEW

A Quick Look at the Reinforcement Learning course A. LAZARIC ( - - PowerPoint PPT Presentation

Feb 21, 2023 22 likes •487 views

A Quick Look at the Reinforcement Learning course A. LAZARIC ( SequeL Team @INRIA-Lille ) Ecole Centrale - Option DAD SequeL INRIA Lille EC-RL Course Why A. LAZARIC Introduction to Reinforcement Learning 2/18 Why: Important

slide-1
SLIDE 1

EC-RL Course

A Quick Look at the “Reinforcement Learning” course

  • A. LAZARIC (SequeL Team @INRIA-Lille)

Ecole Centrale - Option DAD

SequeL – INRIA Lille

slide-2
SLIDE 2

Why

  • A. LAZARIC – Introduction to Reinforcement Learning

2/18

slide-3
SLIDE 3

Why: Important Problems

  • A. LAZARIC – Introduction to Reinforcement Learning

3/18

slide-4
SLIDE 4

Why: Important Problems

◮ Autonomous robotics

  • A. LAZARIC – Introduction to Reinforcement Learning

4/18

slide-5
SLIDE 5

Why: Important Problems

◮ Autonomous robotics ◮ Elder care

  • A. LAZARIC – Introduction to Reinforcement Learning

4/18

slide-6
SLIDE 6

Why: Important Problems

◮ Autonomous robotics ◮ Elder care ◮ Exploration of

unknown/dangerous environments

  • A. LAZARIC – Introduction to Reinforcement Learning

4/18

slide-7
SLIDE 7

Why: Important Problems

◮ Autonomous robotics ◮ Elder care ◮ Exploration of

unknown/dangerous environments

◮ Robotics for entertainment

  • A. LAZARIC – Introduction to Reinforcement Learning

4/18

slide-8
SLIDE 8

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications

  • A. LAZARIC – Introduction to Reinforcement Learning

5/18

slide-9
SLIDE 9

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Trading execution algorithms

  • A. LAZARIC – Introduction to Reinforcement Learning

5/18

slide-10
SLIDE 10

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Trading execution algorithms ◮ Portfolio management

  • A. LAZARIC – Introduction to Reinforcement Learning

5/18

slide-11
SLIDE 11

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Trading execution algorithms ◮ Portfolio management ◮ Option pricing

  • A. LAZARIC – Introduction to Reinforcement Learning

5/18

slide-12
SLIDE 12

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management

  • A. LAZARIC – Introduction to Reinforcement Learning

6/18

slide-13
SLIDE 13

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Energy grid integration

  • A. LAZARIC – Introduction to Reinforcement Learning

6/18

slide-14
SLIDE 14

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Energy grid integration ◮ Maintenance scheduling

  • A. LAZARIC – Introduction to Reinforcement Learning

6/18

slide-15
SLIDE 15

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Energy grid integration ◮ Maintenance scheduling ◮ Energy market regulation

  • A. LAZARIC – Introduction to Reinforcement Learning

6/18

slide-16
SLIDE 16

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Energy grid integration ◮ Maintenance scheduling ◮ Energy market regulation ◮ Energy production

management

  • A. LAZARIC – Introduction to Reinforcement Learning

6/18

slide-17
SLIDE 17

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems

  • A. LAZARIC – Introduction to Reinforcement Learning

7/18

slide-18
SLIDE 18

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Web advertising

  • A. LAZARIC – Introduction to Reinforcement Learning

7/18

slide-19
SLIDE 19

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Web advertising ◮ Product recommendation

  • A. LAZARIC – Introduction to Reinforcement Learning

7/18

slide-20
SLIDE 20

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Web advertising ◮ Product recommendation ◮ Date matching

  • A. LAZARIC – Introduction to Reinforcement Learning

7/18

slide-21
SLIDE 21

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Social applications

  • A. LAZARIC – Introduction to Reinforcement Learning

8/18

slide-22
SLIDE 22

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Social applications ◮ Bike sharing optimization

  • A. LAZARIC – Introduction to Reinforcement Learning

8/18

slide-23
SLIDE 23

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Social applications ◮ Bike sharing optimization ◮ Election campaign

  • A. LAZARIC – Introduction to Reinforcement Learning

8/18

slide-24
SLIDE 24

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Social applications ◮ Bike sharing optimization ◮ Election campaign ◮ ER service optimization

  • A. LAZARIC – Introduction to Reinforcement Learning

8/18

slide-25
SLIDE 25

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Social applications ◮ Bike sharing optimization ◮ Election campaign ◮ ER service optimization ◮ Resource distribution

  • ptimization
  • A. LAZARIC – Introduction to Reinforcement Learning

8/18

slide-26
SLIDE 26

Why: Important Problems

◮ Autonomous robotics ◮ Financial applications ◮ Energy management ◮ Recommender systems ◮ Social applications ◮ And many more...

  • A. LAZARIC – Introduction to Reinforcement Learning

9/18

slide-27
SLIDE 27

What

  • A. LAZARIC – Introduction to Reinforcement Learning

10/18

slide-28
SLIDE 28

What: Decision-Making under Uncertainty

Agent Environment

state / actuation action / perception

  • A. LAZARIC – Introduction to Reinforcement Learning

11/18

slide-29
SLIDE 29

How: Reinforcement Learning

Reinforcement learning is learning what to do – how to map situations to actions – so as to maximize a numerical reward signal. The learner is not told which actions to take, as in most forms of machine learning, but instead must discover which actions yield the most reward by trying them (trial–and–error). In the most interesting and challenging cases, actions may affect not only the immediate reward but also the next situation and, through that, all subsequent rewards (delayed reward).

“An introduction to reinforcement learning”, Sutton and Barto (1998).

  • A. LAZARIC – Introduction to Reinforcement Learning

12/18

slide-30
SLIDE 30

How: the Course

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-31
SLIDE 31

How: the Course

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-32
SLIDE 32

How: the Course

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-33
SLIDE 33

How: the Course

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-34
SLIDE 34

How: the Course

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-35
SLIDE 35

How: the Course

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-36
SLIDE 36

How: the Course

Agent Environment

state / actuation action / perception

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-37
SLIDE 37

How: the Course

Agent Environment

state / actuation action / perception

Formal and rigorous approach to the RL’s way to decision-making under uncertainty

  • A. LAZARIC – Introduction to Reinforcement Learning

13/18

slide-38
SLIDE 38

What: the Highlights of the Course

How do we formalize the agent-environment interaction?

  • A. LAZARIC – Introduction to Reinforcement Learning

14/18

slide-39
SLIDE 39

What: the Highlights of the Course

How do we formalize the agent-environment interaction? How do we solve an MDP?

  • A. LAZARIC – Introduction to Reinforcement Learning

14/18

slide-40
SLIDE 40

What: the Highlights of the Course

How do we formalize the agent-environment interaction? How do we solve an MDP? How do we solve an MDP “online”?

  • A. LAZARIC – Introduction to Reinforcement Learning

14/18

slide-41
SLIDE 41

What: the Highlights of the Course

How do we formalize the agent-environment interaction? How do we solve an MDP? How do we solve an MDP “online”? How do we effectively trade-off exploration and exploitation?

  • A. LAZARIC – Introduction to Reinforcement Learning

14/18

slide-42
SLIDE 42

What: the Highlights of the Course

How do we formalize the agent-environment interaction? How do we solve an MDP? How do we solve an MDP “online”? How do we effectively trade-off exploration and exploitation? How do we solve a “huge” MDP?

  • A. LAZARIC – Introduction to Reinforcement Learning

14/18

slide-43
SLIDE 43

Who

Lectures and Practical Sessions Alessandro LAZARIC SequeL Team INRIA-Lille Nord Europe

alessandro.lazaric@inria.fr researchers.lille.inria.fr/˜lazaric/

  • A. LAZARIC – Introduction to Reinforcement Learning

15/18

slide-44
SLIDE 44

When/What/Where

See planning on the website.

  • A. LAZARIC – Introduction to Reinforcement Learning

16/18

slide-45
SLIDE 45

Evaluation

◮ Three homework (dynamic programming, multi-armed bandit,

approximate dynamic programming): 2.5 points each.

◮ Review of literature with oral presentation: 12.5 points.

  • A. LAZARIC – Introduction to Reinforcement Learning

17/18

slide-46
SLIDE 46

Reinforcement Learning

Alessandro Lazaric alessandro.lazaric@inria.fr sequel.lille.inria.fr