University of Genoa CINEF - Duopolistic competition in an Duopolistic competition in an artificial power exchange artificial power exchange with learning agents with learning agents Speaker: ERIC GUERCI Joint work with: Silvano Cincotti, Stefano Ivaldi, Marco Raberto “Complex Markets” Meeting, Marseille, 6-7 th October 2006
Outline Outline • Electricity Market outlook Electricity Market outlook • • Agent Agent- -based Artificial Power Exchange based Artificial Power Exchange • • Stochastic Games Framework Stochastic Games Framework • • Reinforcement Learning Reinforcement Learning • • Computational Results Computational Results • • Conclusions Conclusions • 10/20/2006 "Complex Markets" meeting, Marseille 2\35
Electricity Markets Electricity Markets • Worldwide deregulation in the 90’s: from monopolistic state- - • Worldwide deregulation in the 90’s: from monopolistic state owned suppliers to “competitive” electricity market owned suppliers to “competitive” electricity market • Active Eureopan Eureopan Power Power Exchanges Exchanges: : • Active – Nord Pool (Finnish countries), 1996 Nord Pool (Finnish countries), 1996 – PPE (Poland ), 2000 PPE (Poland ), 2000 – – – OMEL (Spain), 1998 OMEL (Spain), 1998 – Opcom Opcom (Romania), 2001 (Romania), 2001 – – – APX (The Netherlands), 1999 APX (The Netherlands), 1999 – Powernext Powernext (France), 2001 (France), 2001 – – – NETA (UK), 2000 NETA (UK), 2000 – Borzen Borzen (Slovenia), 2002 (Slovenia), 2002 – – – EEX Frankfurt (Germany), 2000 EEX Frankfurt (Germany), 2000 – EXAA (Austria), 2003 EXAA (Austria), 2003 – – – LPX LPX Lipsia Lipsia (Germany), 2000 (Germany), 2000 – IPEX (Italy), 2004 IPEX (Italy), 2004 – – 10/20/2006 "Complex Markets" meeting, Marseille 3\35
Actors in a Power Exchange Actors in a Power Exchange • Producers (generators and suppliers) • Producers (generators and suppliers) – produce and have customers that consume physical quantities produce and have customers that consume physical quantities – of energy of energy • Traders • Traders – buy and sell electrical energy under contract buy and sell electrical energy under contract – • Clients • Clients – consuming energy customers consuming energy customers – • Market Operator • Market Operator – organization and management of the electricity market organization and management of the electricity market – • System Operator • System Operator – determines what actions need to be taken in order to maintain determines what actions need to be taken in order to maintain – the required national and local balances of generation and the required national and local balances of generation and consumption consumption 10/20/2006 "Complex Markets" meeting, Marseille 4\35
Electricity Markets - - Clearinghouse Clearinghouse Electricity Markets • Day- -Ahead Market Ahead Market – – DAM (Energy market) DAM (Energy market) • Day – Collection of offers and bids for next day hours Collection of offers and bids for next day hours – – Construction of demand and supply curves Construction of demand and supply curves – – Market clearing for every hour of the next day Market clearing for every hour of the next day – – Zonal splitting, in the case of congestion Zonal splitting, in the case of congestion – • Adjustment Market – – AM (Energy market) AM (Energy market) • Adjustment Market – Allows revision of trading activities Allows revision of trading activities – – Starts after DAM and considers separately every our of the next Starts after DAM and considers separately every our of the next – day day • Ancillary Services Market – – ASM (Service market) ASM (Service market) • Ancillary Services Market – Procures resources for dispatching, i.e. management, operation – Procures resources for dispatching, i.e. management, operation and control of the power system and control of the power system – Planned grid congestion relief, purchase of operating reserve Planned grid congestion relief, purchase of operating reserve – for the next day, electricity for real- -time balancing of the system time balancing of the system for the next day, electricity for real 10/20/2006 "Complex Markets" meeting, Marseille 5\35
Artificial Power Exchange Artificial Power Exchange Main Aims and opportunities: • To understand and to simulate the micro To understand and to simulate the micro- -structure of a structure of a • real power exchange real power exchange • To overcome analytical intractability To overcome analytical intractability • • To perform What To perform What- -if Analysis if Analysis • • To develop a framework for market design and To develop a framework for market design and • validation validation 10/20/2006 "Complex Markets" meeting, Marseille 6\35
Stochastic Game Framework (I) Stochastic Game Framework (I) ( , , n S A , , T R ) • Stochastic Game Stochastic Game • 1... n 1... n n Number of agents S Set of States = × Α A i A Set of actions available to agent i i i [ ] × × → Α T S S 0,1 Transition function R Reward function of agent i i 10/20/2006 "Complex Markets" meeting, Marseille 7\35
Strategic form of the game Strategic form of the game R-bimatrix game (one-shot game) ∈ ∈ a A , b A actions of the matrix game j 1 k 2 i R istantaneous reward for player i lm b b 1 2 a 1 2 1 2 R , R R , R 1 11 11 12 12 a 1 2 1 2 R , R R , R 2 21 21 22 22 10/20/2006 "Complex Markets" meeting, Marseille 8\35
Strategic form of the game Strategic form of the game Q-bimatrix game (repeated games) { } { } α = β = a , a ,..., a ,... , b b , ,,..., b ,... stationary policies j j 1 j 2 jt k k k jt i Q expected sum of discounted delayed rewards for player i lm ⎡ ⎤ ∞ ∑ β β π = γ i t i Q E R ⎢ ⎥ 1 2 lm t ⎣ ⎦ = t 0 α π α β 1 2 1 2 Q , Q Q , Q is a joint policy( , ) 1 11 11 12 12 l m α 1 2 1 2 Q , Q Q , Q 2 21 21 22 22 10/20/2006 "Complex Markets" meeting, Marseille 9\35
Solution Concepts Concepts Solution = ∈ * * * * Joint set of actions x ( a a , ), a A − i i i i Π is the generic payoff either R or Q i i i Nash equilibria equilibria: Competitive Solution : Competitive Solution Nash = Π ≥ Π ∀ * * * * * * x ( a a , ) is Nash if ( a a , ) ( a a , ), i − − − i i i i i i i i Pareto optima optima: : Tacit Tacit collusive collusive solution solution Pareto ∃ Π ≥ Π ∀ * * x is not Pareto if x : ( ) x ( x ), i i i 10/20/2006 "Complex Markets" meeting, Marseille 10\35
Why learning? Why learning? • Real electricity markets are characterized by: Real electricity markets are characterized by: • – Small number of suppliers, possible Small number of suppliers, possible oligopolistic oligopolistic – scenario scenario – Few big producers exercising market power Few big producers exercising market power – – Portfolio generators, with different market shares Portfolio generators, with different market shares – – Repeated interaction among the same sellers may Repeated interaction among the same sellers may – produce collusive behavior produce collusive behavior 10/20/2006 "Complex Markets" meeting, Marseille 11\35
Reinforcement Learning Reinforcement Learning 1. Adaptive evolutionary - - Marimon Marimon and and McGrattan McGrattan 1. Adaptive evolutionary (1995) (1995) • Matrix game payoffs (R- -matrix game) matrix game) • Matrix game payoffs (R • Learning instantaneous rewards • Learning instantaneous rewards • Adaptive Stochastic algorithms • Adaptive Stochastic algorithms 2. Q- -Learning Learning – – Watkins (1989) Watkins (1989) 2. Q • Repeated game payoffs (Q- -matrix game) matrix game) • Repeated game payoffs (Q • Learning from delayed rewards • Learning from delayed rewards • Sequential decision task • Sequential decision task • Markov Decision Process Framework • Markov Decision Process Framework 10/20/2006 "Complex Markets" meeting, Marseille 12\35
Recommend
More recommend
