How to Make Profit: Exploiting Fluctuating Electricity Prices with - - PowerPoint PPT Presentation
How to Make Profit: Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters Timo Hnig, Christopher Eibel, Adam Wagenhuser, Maximilian Wagner, and Wolfgang Schrder-Preikschat 8th
Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, and Wolfgang Schröder-Preikschat 8th International Workshop on Runtime and Operating Systems for Supercomputers (ROSS 2018), Tempe, AZ, USA. June 12, 2018
Gode Wind Farm: 582 MW North Sea, Europe
Photo: DONG Energy
Solana Generating Station: 280 MW Arizona, USA
Photo: Abengoa Energy
Solana Generating Station: 280 MW Arizona, USA
Photo: Abengoa Energy, Google
Arizona State University: 24 MW Arizona, USA
Photo: Arizona State University
a great impact on the grid.
Availability of renewables leads to
fmuctuating electricity prices.
How can we design and build an HPC runtime
system to exploit dynamic electricity pricing?
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5
a great impact on the grid.
Availability of renewables leads to
fmuctuating electricity prices.
How can we design and build an HPC runtime
system to exploit dynamic electricity pricing?
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5
a great impact on the grid.
Availability of renewables leads to
fmuctuating electricity prices.
How can we design and build an HPC runtime
system to exploit dynamic electricity pricing?
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 5
Others 1 % Renewables 38 % Natural Gas 9 % Non-renewables 62 % Nuclear Power 13 %
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 6 ■ non-renewables: 339.93 TWh, renewables: 209.97 TWh
dynamic availability of renewables fmuctuation electricity prices
Others 1 % Renewables 38 % Natural Gas 9 % Non-renewables 62 % Nuclear Power 13 %
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 6 ■ non-renewables: 339.93 TWh, renewables: 209.97 TWh ■ dynamic availability of renewables fmuctuation electricity prices
Mo Tu We Th Fr Sa Su
20 40 60 80
Time [day of week] Power [GW]
20 40 60 80 100
Price [EUR/MWh]
non-renewable electricity sources non-renewable plus renewable electricity sources electricity price
[week 43/2017]
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 7
Others 1 % Renewables 38 % Natural Gas 9 %Non-Renewables
Nuclear Power 13 %■ non-renewables can be planned…
…but renewables are often unpredictable consequence: strong fmuctuation of the electricity price extreme imbalance of supply & demand negative prices
Mo Tu We Th Fr Sa Su
20 40 60 80
Time [day of week] Power [GW]
20 40 60 80 100
Price [EUR/MWh]
non-renewable electricity sources non-renewable plus renewable electricity sources electricity price
[week 43/2017]
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 7
Others 1 %Renewables
Natural Gas 9 %Non-Renewables
Nuclear Power 13 %■ non-renewables can be planned… ■ …but renewables are often unpredictable
consequence: strong fmuctuation of the electricity price extreme imbalance of supply & demand negative prices
Mo Tu We Th Fr Sa Su
20 40 60 80
Time [day of week] Power [GW]
20 40 60 80 100
Price [EUR/MWh]
negative price non-renewable electricity sources non-renewable plus renewable electricity sources electricity price
[week 43/2017]
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 7
non-renewables can be planned… …but renewables are often unpredictable
■ consequence: strong fmuctuation of the electricity price ■ extreme imbalance of supply & demand negative prices
Building a System that Benefjts from Dynamic Electricity Prices
10:00 11:00 12:00 13:00 14:00 15:00 16:00 2 4 6 8
Time [hours] Power Demand [kW]
HPC Cluster Power Demand Excess Demand / Under Supply Penalty zone I Target zone Under Demand / Excess Supply Penalty zone II
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 8 ■ challenge: take advantage of low and negative prices ■ consider contracts (i.e., penalties enforced by grid operators) ■ integrate with existing HPC cluster infrastructure
Building a System that Benefjts from Dynamic Electricity Prices
10:00 11:00 12:00 13:00 14:00 15:00 16:00 2 4 6 8
Time [hours] Power Demand [kW]
HPC Cluster Power Demand Excess Demand / Under Supply Penalty zone I Target zone Under Demand / Excess Supply Penalty zone II
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 8 ■ challenge: take advantage of low and negative prices ■ consider contracts (i.e., penalties enforced by grid operators) ■ integrate with existing HPC cluster infrastructure
build a fmexible runtime system that integrates data on electricity pricing into its operating decisions reduce energy demand when prices are high increase energy demand when prices are low (or negative)
Building a System that Benefjts from Dynamic Electricity Prices
10:00 11:00 12:00 13:00 14:00 15:00 16:00 2 4 6 8
Time [hours] Power Demand [kW]
With Albatross Without Albatross Excess Demand / Under Supply Penalty zone I Target zone Under Demand / Excess Supply Penalty zone II
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 8
Our proposed runtime system Albatross:
■ implements low- and high-power operation modes ■ considers heterogeneity aspects of the cluster hardware ■ respects QoS and non-functional requirements of workloads
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9 ■ foobar
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
■ generic system model of an HPC cluster without Albatross
Job1 Job2 Jobm
. . .
Master node Cluster manager Job scheduler Execution unit1
Compute node1
Execution unit2
Compute node2
. . .
Execution unitn
Compute noden
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9 ■ extended system model of an HPC cluster with Albatross
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
■ generic system model of an HPC cluster without Albatross
Job1 Job2 Jobm
. . .
Master node Cluster manager Job scheduler Execution unit1
Compute node1
Execution unit2
Compute node2
. . .
Execution unitn
Compute noden
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9 ■ extended system model of an HPC cluster with Albatross
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Resource governor Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
■ input interfaces for job submission and control
■ submission interface for electricity-pricing data ■ real-time data as provisioned by grid operator ■ detailed job description, added QoS constraints
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9 ■ extended system model of an HPC cluster with Albatross
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Resource governor Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
■ power monitoring and control interfaces
■ probes operate power-management features (e.g., power capping) ■ hardware-specifjc metering (e.g., RAPL, measuring devices) ■ delivery of base data for feedback loop to the resource governor
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9 ■ extended system model of an HPC cluster with Albatross
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Resource governor Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
■ runtime control of the HPC system by the resource governor
■ job submission based on QoS data and current electricity prices ■ control and measurement of power demand for individual jobs ■ consider heterogeneity aspects (system, component, and confjguration)
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 9 ■ extended system model of an HPC cluster with Albatross
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Resource governor Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
■ runtime control of the HPC system by the resource governor
■ job submission based on QoS data and current electricity prices ■ control and measurement of power demand for individual jobs ■ consider heterogeneity aspects (system, component, and confjguration)
Heterogeneity Scope System Component
Feature Moab Tivoli Univa Slurm Job pinning
○ ○
CPU allocation
○
Quality of service
○
Generic resources
○
Cluster status
Heterogeneity aware
○ ○
Power/price aware
○ ○ ○ ○ Table: Feature comparison of Albatross with other workload managers.
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 10 ■ core of Albatross implementation is based on Slurm ■ constraint-aware job-to-node–assignment strategy
■ evaluation scenario
NAS Parallel Benchmarks (NPB) suite, Albatross runtime system heterogeneous cluster with Intel & ARM CPUs, Nvidia & Intel GPUs, and various power meters
■ evaluation experiments and goals
energy demand and execution times
energy–delay product and power demand
to exploit dynamic electricity pricing
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 11
1 MCP39F511 1 3 Current probe 3 2 PCIe riser card 2 4 Red Pitaya 4 C Intel Xeon C A / B ARM ODROID-C1+/C2 A B D Intel HD GPU D E Nvidia GPU E
evaluation scenario
NAS Parallel Benchmarks (NPB) suite, Albatross runtime system heterogeneous cluster with Intel & ARM CPUs, Nvidia & Intel GPUs, and various power meters
evaluation experiments and goals
energy demand and execution times
energy–delay product and power demand
to exploit dynamic electricity pricing
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 11
1 MCP39F511 1 3 Current probe 3 2 PCIe riser card 2 4 Red Pitaya 4 C Intel Xeon C A / B ARM ODROID-C1+/C2 A B D Intel HD GPU D E Nvidia GPU E
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 12 ■ impact of system-level and confjguration-level heterogeneity
CG EP FT IS MG
100 200
Energy Demand [J] ODROID-C1+ ODROID-C2 Xeon Xeoncap
Execution Time
9.1 s 8.0 s 0.6 s 0.6 s 27.9 s 12.6 s 1.9 s 2.6 s 55.1 s 22.8 s 1.2 s 1.4 s 3.4 s 1.5 s 0.3 s 0.3 s 14.8 s 7.8 s 1.5 s 1.5 s
■ Xeon and Xeoncap dominate with performance ■ energy demand in some cases lower for ARM platforms ■ occasional relationship between process energy demand
and execution time
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 12 ■ impact of system-level and confjguration-level heterogeneity
CG EP FT IS MG
100 200
Energy Demand [J] ODROID-C1+ ODROID-C2 Xeon Xeoncap
Execution Time
9.1 s 8.0 s 0.6 s 0.6 s 27.9 s 12.6 s 1.9 s 2.6 s 55.1 s 22.8 s 1.2 s 1.4 s 3.4 s 1.5 s 0.3 s 0.3 s 14.8 s 7.8 s 1.5 s 1.5 s
■ Xeon and Xeoncap dominate with performance ■ energy demand in some cases lower for ARM platforms ■ occasional relationship between process energy demand
and execution time Key take-away race to sleep does not always reduce energy demand knowledge on systems’ impact is absolutely necessary for
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 13 ■ impact of component-level heterogeneity
0.2 0.4 0.6 0.8 1
Normalized EDP
CG EP FT IS MG
20 40 60
Power Demand [W] Xeon Intel HD Nvidia
■ EDP is best for Nvidia for all but one benchmark ■ Intel HD has the worst EDP results, but lowest average power
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 13 ■ impact of component-level heterogeneity
0.2 0.4 0.6 0.8 1
Normalized EDP
CG EP FT IS MG
20 40 60
Power Demand [W] Xeon Intel HD Nvidia
■ EDP is best for Nvidia for all but one benchmark ■ Intel HD has the worst EDP results, but lowest average power
Key take-away both, EDP and average power must be considered to balance the cluster’s power drain Albatross trades average power for EDP (and vice versa)
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 14
10 20 30 40 50 60 40 80 120 160
Time [min]
Power Drain [W]
Albatross SLURM upper limit lower limit
0.1 Time [day of week]
Cost [EUR]
50 100
Price [EUR/MWh]
Th Fr Sa Su negative price Albatross SLURM price
[week 43/2017]
Albatross, an HPC runtime system that
successfully exploits dynamic electricity pricing. Albatross…
■ considers QoS of jobs and electricity prices for
deploying workloads
■ exploits heterogeneity at system-level,
component-level, and confjguration-level
■ prepares HPC clusters for future settings where
energy is just another operating resource
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 15
(Wednesday Evening)
Others 1 %Renewables
Natural Gas 9 %Non-Renewables
Nuclear Power 13 % interface Albatross_Cluster_Control { /* Get cluster configuration */ NodeList get_nodes(); /* Get all running jobs and their allocation */ JobList get_jobs(); /* Submit job with attached job−resource info */ JobResult submit_job(Job j); }
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 17
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Resource governor Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
interface Albatross_Power_Control { /* Triggers power measuring for an exec. unit */ void start_measurement(Node n, ExecUnit u); PowerValues stop_measurement(Node n, ExecUnit u); /* Sets the value of a power−management feature */ void set_pm(Node n, ExecUnit u, PowerConfig c); }
Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters 18
Job1 Job2 Jobm QoS constraints
. . .
Electricity & price constraints Master node
Resource governor Cluster manager Job scheduler
Resource group1 Resource groupl
. . .
Exec1 Execn1 Probe1
Compute node1
…
Exec1 Execn2 Probe2
Compute node2
… . . .
Exec1 Execnk Probek
Compute nodek
…
Power meters
Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, and Wolfgang Schröder-Preikschat: How to Make Profjt: Exploiting Fluctuating Electricity Prices with Albatross, A Runtime System for Heterogeneous HPC Clusters Proceedings of the 8th International Workshop on Runtime and Operating Systems for Supercomputers (ROSS’18), 12 June 2018, Tempe, AZ, USA ACM, New York, NY, USA, ISBN: 978-1-4503-5864-4 DOI: 10.1145/3217189.3217193 Timo Hönig, Christopher Eibel, Adam Wagenhäuser, Maximilian Wagner, and Wolfgang Schröder-Preikschat: Making Profjt with ALBATROSS: A Runtime System for Heterogeneous High-Performance–Computing Clusters Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing (HPDC’18), 11 - 15 June 2018, Tempe, AZ, USA ACM, New York, NY, USA, ISBN: 978-1-4503-5899-6 DOI: 10.1145/3220192.3220457