learned impatience dispersed reinforcement and time
play

Learned Impatience? Dispersed Reinforcement and Time Discounting - PowerPoint PPT Presentation

Jul 21, 2023 •21 likes •471 views

Learned Impatience? Dispersed Reinforcement and Time Discounting David Poensgen (Goethe University Frankfurt) February 22, 2019 Sloan-Nomis Workshop on the Cognitive Foundations of Economic Behavior Motivation 1. Individuals learn from

  1. Learned Impatience? Dispersed Reinforcement and Time Discounting David Poensgen (Goethe University Frankfurt) February 22, 2019 Sloan-Nomis Workshop on the Cognitive Foundations of Economic Behavior

  2. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 1

  3. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 1

  4. 2

  5. 2

  6. 2

  7. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 3

  8. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 3

  9. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 3

  10. 4

  11. Motivation 1. Individuals learn from consequences of past actions. 2. Actions often have a series of consequences: some follow soon, some later. 3. How does this ordering affect learning? Plausibly: Easiest to learn from soonest consequences. 4. Then: Immediate consequences will be over-weighted. Behavior biased towards impatience. May help explain why myopic behavior is so widespread and persistent. 5

  12. Background • Decreasing effectiveness of reinforcement with delay (e.g. Mazur 2002). • Typically not connected to time discounting, but speed of learning. • Explained via accumulation of noise by Commons, Woodford et al. (1982, 1991). • Feedback delay modulates neural circuitries involved in learning (Foerde/Shohamy 2011, Foerde et al. 2013, Arbel et al. 2017). • Associative learning tasks; singular feedback. Performance not affected. • Gabaix & Laibson (2017) also link time discounting and information frictions. • Formally applicable here; different interpretation on source of noise. • Melioration theory: Behavior guided by immediate, not overall reinforcement rate (Herrnstein et al.). • Important experimental paradigm: “Harvard game” (Review: Prelec 2014). • Critique by Sims et al. (2013): Bayesian algorithms need 1000s of trials for solution. Melioration as rational response to task complexity. 6

  13. Design: Overview • Payoff and feedback mechanism: • All rules and mechanisms clearly communicated to subjects. • All points rewarded simultaneously after the experiment. • Goal: Collect as many points as possible. • Choosing x has 2 consequences: • Values initially unknown, but can be learned. • 6 abstract options (= colors): 7 • Subjects faced with sequence of 105 binary choices. , , , , , { } • Each color x associated with a payoff vector ( x 1 , x 2 ) x 1 + ϵ points shown and awarded immediately. x 2 + ϵ ′ points shown and awarded with one round delay. • ϵ , ϵ ′ are disturbances drawn uniformly from { 1 , 2 , 3 , 4 } . • Total value of x is x 1 + x 2

  14. Design: Overview • Payoff and feedback mechanism: • All rules and mechanisms clearly communicated to subjects. • All points rewarded simultaneously after the experiment. • Goal: Collect as many points as possible. • Choosing x has 2 consequences: • Values initially unknown, but can be learned. • 6 abstract options (= colors): 7 • Subjects faced with sequence of 105 binary choices. , , , , , { } • Each color x associated with a payoff vector ( x 1 , x 2 ) x 1 + ϵ points shown and awarded immediately. x 2 + ϵ ′ points shown and awarded with one round delay. • ϵ , ϵ ′ are disturbances drawn uniformly from { 1 , 2 , 3 , 4 } . • Total value of x is x 1 + x 2

  15. Design: Overview • Payoff and feedback mechanism: • All rules and mechanisms clearly communicated to subjects. • All points rewarded simultaneously after the experiment. • Goal: Collect as many points as possible. • Choosing x has 2 consequences: • Values initially unknown, but can be learned. • 6 abstract options (= colors): 7 • Subjects faced with sequence of 105 binary choices. , , , , , { } • Each color x associated with a payoff vector ( x 1 , x 2 ) x 1 + ϵ points shown and awarded immediately. x 2 + ϵ ′ points shown and awarded with one round delay. • ϵ , ϵ ′ are disturbances drawn uniformly from { 1 , 2 , 3 , 4 } . • Total value of x is x 1 + x 2

  16. Design: Example Screen 8

  17. Design: Example Screen 8

  18. Design: Example Screen 8

  19. Design: Example Screen 8

  20. Design: Example Screen 8

  21. Design: Example Screen 8

  22. Design: Example Screen 8

  23. Design: Example Screen 8

  24. Design: Example Screen 8

  25. Design: Example Screen 8

  26. Design: Example Screen 8

  27. Design: Example Screen 8

  28. Design: Example Screen 8

  29. Design: Example Screen 8

  30. Design: Example Screen 8

  31. Design: Example Screen 8

  32. Design: Example Screen 8

  33. Design: Payoff Vectors Group A Hypotheses: Option Group B 9 Payoff Vectors color e.g. ( total value ) ( immediate, delayed ) ( 18 ) ( 11 , 7 ) A ( 7 , 11 ) B ( 16 ) ( 6 , 10 ) A ( 10 , 6 ) B ( 14 ) ( 9 , 5 ) A ( 5 , 9 ) B ( 12 ) ( 4 , 8 ) A ( 8 , 4 ) B ( 10 ) ( 7 , 3 ) A ( 3 , 7 ) B ( 8 ) ( 2 , 6 ) A ( 6 , 2 ) B ( 11 , 7 ) A chosen more often than ( 7 , 11 ) B ; ( 10 , 6 ) B more than ( 6 , 10 ) A ; ... ( 11 , 7 ) A and ( 6 , 10 ) A further apart than ( 6 , 10 ) A and ( 9 , 5 ) A . Potentially even: ( 9 , 5 ) A preferred to ( 6 , 10 ) A .

  34. Design: Payoff Vectors Group A Hypotheses: Option Group B 9 Payoff Vectors color e.g. ( total value ) ( immediate, delayed ) ( 18 ) ( 11 , 7 ) A ( 7 , 11 ) B ( 16 ) ( 6 , 10 ) A ( 10 , 6 ) B ( 14 ) ( 9 , 5 ) A ( 5 , 9 ) B ( 12 ) ( 4 , 8 ) A ( 8 , 4 ) B ( 10 ) ( 7 , 3 ) A ( 3 , 7 ) B ( 8 ) ( 2 , 6 ) A ( 6 , 2 ) B ( 11 , 7 ) A chosen more often than ( 7 , 11 ) B ; ( 10 , 6 ) B more than ( 6 , 10 ) A ; ... ( 11 , 7 ) A and ( 6 , 10 ) A further apart than ( 6 , 10 ) A and ( 9 , 5 ) A . Potentially even: ( 9 , 5 ) A preferred to ( 6 , 10 ) A .

  35. Results: Choice Frequencies 10

  36. Results: Choice Frequencies 10

  37. Results: Choice Frequencies 10

  38. Results: Choice Frequencies 10

  39. Results: Choice Frequencies 10

  40. Results: Bias over time 11

  41. Summary: Further Results • Elicited beliefs are in accordance with choice behavior. • Considerable heterogeneity in degree of biasedness. • Correlated to impatience in hypothetical intertemporal choice. • (To do: Incentivized choice or field measures of impatience.) • Treatment: Learning by observation • Subjects passively presented with feedback for 63 rounds. • Directly afterwards: 42 own decisions. • Bias attenuated; low right after the learning phase, then gradually increasing. • Suggests emergence of bias is connected to active decision making. 12 • Estimated latent utility function: u ( x ) = x 1 + 0 . 4 x 2

  42. Summary: Further Results • Elicited beliefs are in accordance with choice behavior. • Considerable heterogeneity in degree of biasedness. • Correlated to impatience in hypothetical intertemporal choice. • (To do: Incentivized choice or field measures of impatience.) • Treatment: Learning by observation • Subjects passively presented with feedback for 63 rounds. • Directly afterwards: 42 own decisions. • Bias attenuated; low right after the learning phase, then gradually increasing. • Suggests emergence of bias is connected to active decision making. 12 • Estimated latent utility function: u ( x ) = x 1 + 0 . 4 x 2

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Class Structure Last time: Midterm! This time: Exploration and Exploitation Next time: Batch RL

Lecture 11: Fast Reinforcement Learning 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 11: Fast Reinforcement Learning 3 Emma Brunskill (CS234 Reinforcement Learning. )

887 views • 66 slides
How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness

How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness

Automatic Programming: How Far Can Machines Go? Hila Peleg Technion Laziness Laziness Impatience Laziness Impatience Hubris Automatic Programming Clean up my spreadsheet! Right away, boss! Program synthesis to the rescue Program

1.01k views • 72 slides
Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Class Structure Last time: Midterm This time: Fast Learning Next time: Fast Learning Lecture 11:

Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill CS234 Reinforcement Learning Winter 2019 1 With many slides from or derived from David Silver, Examples new Lecture 11: Fast Reinforcement Learning 1 Emma Brunskill (CS234 Reinforcement

870 views • 65 slides
AONT-RS: Blending Security and Performance in Dispersed Storage Systems Jason Resch James Plank

AONT-RS: Blending Security and Performance in Dispersed Storage Systems Jason Resch James Plank

AONT-RS: Blending Security and Performance in Dispersed Storage Systems Jason Resch James Plank Cleversafe, Inc. University of Tennessee Chicago, IL Knoxville, TN 1 Topics Appeals of Dispersed Storage Methods for Securing Dispersed

659 views • 26 slides
Set Up Your Dev Setup {srinu, sayan} Three Chief Virtues of a Programmer: Laziness Impatience

Set Up Your Dev Setup {srinu, sayan} Three Chief Virtues of a Programmer: Laziness Impatience

Set Up Your Dev Setup {srinu, sayan} Three Chief Virtues of a Programmer: Laziness Impatience Hubris Three Focus Areas for Today: Package Management Writing and Executing Code Common Frameworks This is very subjective!

674 views • 23 slides
Oregon Behavioral Health Integration: What weve heard, seen and learned Integration

Oregon Behavioral Health Integration: What weve heard, seen and learned Integration

Oregon Behavioral Health Integration: What weve heard, seen and learned Integration Environmental Scan OHSU qualitative research, five geographically dispersed CCO communities. Additional 30+ site visits, CCO and provider interviews.

207 views • 18 slides
Experimental choices and field behavior on impatience, saving and smoking Matthias Sutter

Experimental choices and field behavior on impatience, saving and smoking Matthias Sutter

Latsis Symposium 2012 Economics on the Move Experimental choices and field behavior on impatience, saving and smoking Matthias Sutter University of Innsbruck and University of Gothenburg 1 Overview of talk Main paper (forthcoming

554 views • 34 slides
Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Class Structure Last time: Fast Learning, Exploration/Exploitation Part 1 This Time: Fast

Lecture 12: Fast Reinforcement Learning Part II 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver, Worked Examples New Lecture 12: Fast Reinforcement Learning Part II 3 Emma Brunskill

1.04k views • 70 slides
Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2019 Reinforcement Learning

Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2019 Reinforcement Learning

Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2019 Reinforcement Learning t r t R = max : S A t =0 MDPs Agent interacts with an environment At each time t: Receives sensor signal s t

717 views • 40 slides
THERMOELECTRIC PROPERTIES OF Cu-DISPERSED Bi 2 Te 2.7 Se 0.3 NANOCOMPOSITES Il-Ho Kim 1 *, Soon-Mok

THERMOELECTRIC PROPERTIES OF Cu-DISPERSED Bi 2 Te 2.7 Se 0.3 NANOCOMPOSITES Il-Ho Kim 1 *, Soon-Mok

18 TH INTERNATIONAL CONFERENCE ON COMPOSITE MATERIALS THERMOELECTRIC PROPERTIES OF Cu-DISPERSED Bi 2 Te 2.7 Se 0.3 NANOCOMPOSITES Il-Ho Kim 1 *, Soon-Mok Choi 2 , Won-Seon Seo 2 , Dong-Ik Cheong 3 , Hyung Kang 3 1 Department of Materials Science

100 views • 5 slides
Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David

Reinforcement Learning A (almost) quick (and very incomplete) introduction Slides from David Silver, Dan Klein, Mausam, Dan Weld Reinforcement Learning At each time step t : Agent executes an action A t Environment emits a reward Rt

1.1k views • 71 slides
Reinforcement Learning, cont. Models an agent trying to learn what to do / how to act

Reinforcement Learning, cont. Models an agent trying to learn what to do / how to act

Reinforcement Learning, cont. Models an agent trying to learn what to do / how to act in a complex environment. So far, we have seen passive learning We learned utility values for a fixed world with fixed transitions (no real

737 views • 29 slides
RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem

Introduction to Reinforcement Learning RL Overview of topics About Reinforcement Learning The Reinforcement Learning Problem Inside an RL agent Temporal difference learning Many faces of Reinforcement Learning What is

551 views • 35 slides
Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 16:

Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 16:

Lecture 16: MCTS 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 1 With many slides from or derived from David Silver Lecture 16: MCTS 1 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 / 57 Class Structure Last time:

1.28k views • 58 slides
Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 14:

Class Structure Last time: Batch RL This Time: MCTS Next time: Human in the Loop RL Lecture 14:

Lecture 14: MCTS 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 2 With many slides from or derived from David Silver Lecture 14: MCTS 3 Emma Brunskill (CS234 Reinforcement Learning. ) Winter 2018 1 / 57 Class Structure Last time:

817 views • 57 slides
Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine

Reinforcement Learning Reinforcement Learning Reinforcement Learning in a nutshell g Imagine playing a new game whose rules you dont know; after a hundred or so moves your don t know; after a hundred or so moves, your opponent announces, You

511 views • 30 slides
1 Northwest SEED is a non-profit that has completed solar and DHP group purchase programs in the

1 Northwest SEED is a non-profit that has completed solar and DHP group purchase programs in the

1 Northwest SEED is a non-profit that has completed solar and DHP group purchase programs in the past. They can provide more information on the program and DHPs. Rogue Climate is the local outreach partner that is organizing volunteers and

1.07k views • 31 slides
Recent Developments, Old Complexities Remembered Samantha Cheng (BSc Hons) Sales &

Recent Developments, Old Complexities Remembered Samantha Cheng (BSc Hons) Sales &

Right to Buy Recent Developments, Old Complexities Remembered Samantha Cheng (BSc Hons) Sales & Acquisitions Manager London Borough of Southwark www.southwark.gov.uk Housing Act 1980 RTB introduced Minimum qualification period 3

589 views • 20 slides
available to individuals and groups. We maintain a B+ (Good) rating from A.M. Best Company,

available to individuals and groups. We maintain a B+ (Good) rating from A.M. Best Company,

Founded over 50 years ago as an insurance agency focused on meeting the insurance needs of financial institution customers, IAC has grown through the years by remaining Focused on what matters most. Today, we operate in 43 states and

704 views • 31 slides
Insurance Commissioner John D. Doak Building Resilient Homes A Better Way Forward May 19, 2013

Insurance Commissioner John D. Doak Building Resilient Homes A Better Way Forward May 19, 2013

Insurance Commissioner John D. Doak Building Resilient Homes A Better Way Forward May 19, 2013 Shawnee, Oklahoma May 20, 2013 Moore, Oklahoma May 30, 2013 Broken Arrow, Oklahoma May 31, 2013 El Reno, Oklahoma 2013 Aftermath

241 views • 20 slides
Effectiveness Threshold for Cancer Care in Alberta: Eldon Spackman, PhD Assistant Professor

Effectiveness Threshold for Cancer Care in Alberta: Eldon Spackman, PhD Assistant Professor

Estimating the Cost- Effectiveness Threshold for Cancer Care in Alberta: Eldon Spackman, PhD Assistant Professor Contributors Mike Paulden, PhD: University of Alberta Chris McCabe, PhD: University of Alberta Petros Pechlivanoglou,

351 views • 13 slides
Combining Expert Advice on Social Discounting Moritz A. Drupp 1 , Mark C. Freeman 2 , Ben Groom 3 ,

Combining Expert Advice on Social Discounting Moritz A. Drupp 1 , Mark C. Freeman 2 , Ben Groom 3 ,

Combining Expert Advice on Social Discounting Moritz A. Drupp 1 , Mark C. Freeman 2 , Ben Groom 3 , and Frikk Nesje 4 1 University of Kiel and University of Freiburg, Germany 2 University of York, UK 3 London School of Economics and Political

380 views • 16 slides
Angle Lake Station TOD Suitability for housing and development strategy 3/5/2020 Why we are

Angle Lake Station TOD Suitability for housing and development strategy 3/5/2020 Why we are

Angle Lake Station TOD Suitability for housing and development strategy 3/5/2020 Why we are here Seeking action on declaring surplus properties as suitable for the development of housing and on the development strategy. 2 Property

612 views • 8 slides
#68 Joint Development Policy PORTFOLIO-WIDE GOAL Establish a portfolio-wide goal of 35%

#68 Joint Development Policy PORTFOLIO-WIDE GOAL Establish a portfolio-wide goal of 35%

ADOPT Updated Metro #68 Joint Development Policy PORTFOLIO-WIDE GOAL Establish a portfolio-wide goal of 35% affordable housing units Allow flexibility across the region for housing types Current portfolio is at 31% but was largely

208 views • 3 slides

More recommend