MIP Model: Minimum Shipment of 100 cases β’ Shipment volume: x (continuous variable) β’ Discrete decision: ship (binary variable) Cost ($) ship = 0 ship = 1 Not possible x (Number of cases) 0 100 add constraints: (if ship=1, then ship at least 100) π¦ π,π β₯ 100 β π‘βππ π,π β π, π (if ship=0, then do not ship at all) π¦ π,π β€ ππππ β π‘βππ π,π β π, π Hands-On π‘βππ π,π β 0,1 45
MIP Model: GAMS Syntax 46
MIP Model: Results 47
MIP Model: Solution shipments Markets (demand) Canning Plants (supply) (Number of cases) 300 275 325 Freight: $90 case / thousand miles Total cost: $153,675 48
LP MIP MINLP Scenarios β’ Determine β’ Discrete β’ Non-linearity β’ SolveLink minimum decisions β’ E.g.: Decrease β’ Grid Facility transportation cost β’ E.g.: Ship at in unit cost β’ GUSS β’ Result: city to city least 100 with growing shipment volumes cases volumes 49
MINLP: Cost Savings Cost ($) The cost per case decreases with an increasing shipment volume ship = 0 ship = 1 Not possible x (Number of cases) 0 100 Replace: (Minimize total transportation cost) π π π ππ β π¦ ππ min With π π π ππ β π¦ ππ ππππ (Minimize total transportation cost) min Hands-On 50
MINLP Model: GAMS Syntax 51
MINLP Model: Results 52
MINLP Model: Solution shipments Markets (demand) Canning Plants (supply) (Number of cases) 300 275 325 Freight: $90 case / thousand miles Total cost: $115,438 53
LP MIP MINLP Scenarios β’ Determine β’ Discrete β’ Non-linearity β’ SolveLink minimum decisions β’ E.g.: Decrease β’ Grid Facility transportation cost β’ E.g.: Ship at in unit cost β’ GUSS β’ Result: city to city least 100 with growing shipment volumes cases volumes 54
Motivation β’ Solving challenging real-world problems often involves the solution of many optimization problems β’ Decomposition Methods β’ Scenario Analysis β’ Heuristics β’ β¦ β’ Such approaches are often chosen, if solving the problem at hand does not work with a single monolithic model, e.g. β’ Due to itβs size and the required resources (e.g. memory) β’ Due to time restrictions (Problem should be solved in minutes but it takes days) β’ β¦ β GAMS Grid Facility β Gather-Update-Solve-Scatter 55
SolveLink Option Controls GAMS function when linking to solver. Model transport /all/ ; transport.solveLink={ 0 %Solvelink.ChainScript%, 1 %Solvelink.CallScript%, 2 %Solvelink.CallModule%, 3 %Solvelink.AsyncGrid%, 4 %Solvelink.AsyncSimulate%, 5 %Solvelink.LoadLibrary%, 6 %solveLink.aSyncThreads%, %solveLink.threadsSimulate% }; 7 solve transport using lp minimizing z; 56
SolveLink Option β Sequential Solves β’ ChainScript [0]: Solver process, GAMS vacates memory + Maximum memory available to solver + protection against solver failure ( hostile link) - swap to disk β’ Call{Script [1]/Module [2]}: Solver process, GAMS stays live + protection against solver failure ( hostile link) + no swap of GAMS database - file based model communication β’ LoadLibrary [5]: Solver DLL in GAMS process + fast memory based model communication + update of model object inside the solver (hot start) - not supported by all solvers 57
SolveLink Option Sequential β Exercise β’ Generate 100 distance scenarios with random data β’ Solve these scenarios with the solveLink valuesβ¦ β’ ChainScript [0] β’ CallModule [2] β’ LoadLibrary [5] β’ Compare the execution time of solving all scenarios with different solveLink settings β’ Hint: Look at the GAMS function jNow Hands-On 58
SolveLink Option β Sequential Solves ---- 88 PARAMETER time time for 100 scenarios ChainScript 6.710, CallModule 2.694, LoadLibrary 0.578 59
SolveLink Option β Asynchronous Solves β’ aSyncGrid [3]: GAMS starts the solution and continues in a Grid computing environment β’ aSyncThreads [6]: The problem is passed to the solver in core without use of temporary files, GAMS does not wait for the solver to come back 60
SolveLink Option Asynchronous β Exercise β’ Generate 100 distance scenarios with random data β’ Solve these scenarios with the solveLink valuesβ¦ β’ aSyncGrid [3] β’ aSyncThreads [6] β’ Compare the execution time of solving all scenarios with different solveLink settings β’ Hint: Check the log for output about solveLink β’ β Use solver CplexD instead of Cplex β’ Hint: Look at the following GAMS functions: β’ readyCollect β’ handleCollect β’ handleDelete Hands-On 61
SolveLink Option β Asynchronous Solves ChainScript 6.710 CallModule 2.694 LoadLibrary 0.578 aSyncGrid 4.259 aSyncThreads 0.496 ---- 96 PARAMETER time time for 100 scenarios aSyncGrid 4.259, aSyncThreads 0.496 62
SolveLink Option β Asynchronous Solves β’ Helpful, if large ratio of solver time / GAMS time ChainScript 29.807 CallModule 30.004 LoadLibrary 28.864 aSyncGrid 9.112 aSyncThreads 7.901 63
GUSS β Gather-Update-Solve-Scatter aka Scenario Solver Simple Serial Scenario Solver/GUSS Solve Loop Generation Loop Generation Solution Solution Update Update Generates model once and updates the algebraic model keeping the model βhotβ inside the solver. 64
GUSS β Gather-Update-Solve-Scatter aka Scenario Solver ChainScript 6.710 CallModule 2.694 LoadLibrary 0.578 aSyncGrid 4.259 aSyncThreads 0.496 GUSS 0.273 65
Grid & GUSS β Examples from the model library β’ trnsgrid: https://www.gams.com/latest/gamslib_ml/libhtml/gamslib_trnsgrid.html β’ Simple asynchronous solves in a loop, separate collection loop β’ tgridmix : https://www.gams.com/latest/gamslib_ml/libhtml/gamslib_tgridmix.html β’ Asynchronous solves in combined submission & collection loop. Keep number of submitted models <= #threads β’ gussgrid : https://www.gams.com/latest/gamslib_ml/libhtml/gamslib_gussgrid.html β’ Asynchronous GUSS-solves in combined submission & collection loop. Keep number of submitted models <= #threads 66
A Model is a Model is a Model* * Freely adapted from the poetry of Gertrude Stein, 1874-1946, American writer
What is a Model? 68
What is a Model? β’ A mathematical model is a description of a system using mathematical language (from: Wikipedia) β Mathematical Programming (MP) Model: List of Equations 69
What is a Model in GAMS? β’ A mathematical model is a description of a system using mathematical language (from: Wikipedia) β Mathematical Programming (MP) Model: List of Equations β’ Collection of several intertwined MP Models β’ Data Preparation β’ Data Calibration β’ βSolutionβ Module (e.g. sequential, parallel loop) β’ Report Module 70
Model as Communication Vehicle β’ Defining scope of a (part of a) project/model β’ IT, analysts, managers, model builders have different views β’ Misunderstandings common with verbal descriptions β’ Use a model to define the scope β’ Requirements for such a model β’ Rapid prototyping (max. 1-2 man days) β’ Standard IO interface (Excel) β’ Remote execution (e.g. through Web UI) 71
Model as Analytic Framework β’ Optimization models do not allow for any type of vagueness β’ Input data requirements β’ Objectives and constraints β’ Results β’ Misunderstandings result in failure of the model β’ Compilation/execution errors β’ Infeasible/unbounded MP models β’ Model as a contract 72
Model as Contract β’ Good models do not rely on contract (input data) β’ Input Module (handles bad data) β’ Simple error checks β’ Analyzing and reporting complex data problems β’ Good models (modeling systems) provide access to results via independent result analyzers for non model experts β’ Analytic framework help define result metric β’ e.g. violations of soft constraints 73
Model as Cost Saver β’ Most convincing and obvious reason for using an optimization model β’ Science of better (INFORMS) β’ Often exaggerated/difficult to estimate Model Roles over Time Analytic Communication Contract Cost Saver Vehicle Framework 74
Outlook: Parallelization on HPC Systems From simple sequential to highly parallel solve statements
Available Computing Resources Multi-core shared memory standard core core core core memory Distributed (shared) memory Off-the-shelf Optimization software core core core core Node 0 (speedup via Problem memory parallelization) core core core core Node 1 HPC memory . . . How does GAMS support problem Node n-1 core core core core specific solution approaches that are well suited/customized for HPC? memory 76
Solve sequence of problems Parallelize (e.g. in Benders Decomp.) β¦ 1 β¦ β¦ Decompose time β¦ time β¦ β¦ β¦ β¦ β¦ GAMS supports different levels of parallelization Optimization Problem GAMS provides annotation facilities and HPC solver links Non-zero Plot of matrix User Annotation to define block structure 2 Use structure exploiting solver 77
Sequential Solve Statements in Loops β’ loop body code in sequence, often with an expensive solve statement: Model generation Solution process Model generation Solution process time Model generation Solution process β¦ 78
Parallel Solves β GAMS Grid Facility SolveLink option specifies the solver linking β’ conventions Split loop in submission & collection loop: β’ MG MG S MG S time S Model generation and loop body code in sequence β’ Either file based IO or limited to shared memory β’ 79
Excursus 1: Message Passing Interface (MPI) gams myfile gams myfile mpiexec β n 5 gams myfile gams myfile gams myfile gams myfile 80
Excursus 1: Message Passing Interface (MPI) gams myfile <-- %sysenv.PMI_RANK%=0 gams myfile <-- %sysenv.PMI_RANK%=1 mpiexec β n 5 gams myfile gams myfile <-- %sysenv.PMI_RANK%=2 gams myfile <-- %sysenv.PMI_RANK%=3 gams myfile <-- %sysenv.PMI_RANK%=4 myfile.gms β pseudo code broadcast(db,0) 0 1 2 3 4 gather(db,0) 0 1 2 3 4 81
Excursus 1: Message Passing Interface (MPI) gams myfile <-- %sysenv.PMI_RANK%=0 gams myfile <-- %sysenv.PMI_RANK%=1 mpiexec β n 5 gams myfile gams myfile <-- %sysenv.PMI_RANK%=2 gams myfile <-- %sysenv.PMI_RANK%=3 gams myfile <-- %sysenv.PMI_RANK%=4 myfile.gms β pseudo code MG MG MG β¦ S S S time 82
Excursus 1: Message Passing Interface (MPI) gams myfile <-- %sysenv.PMI_RANK%=0 gams myfile <-- %sysenv.PMI_RANK%=1 mpiexec β n 5 gams myfile gams myfile <-- %sysenv.PMI_RANK%=2 gams myfile <-- %sysenv.PMI_RANK%=3 gams myfile <-- %sysenv.PMI_RANK%=4 myfile.gms β pseudo code β’ Requires reorganization of the code but allows parallel solve β’ Distribute/merging data easy (part of MPI) β’ Network based communication β’ Need to make GAMS aware of MPI β Embedded Code 83
Excursus 2: Embedded Code Facility 84
Example: Sequential Benders Decomposition set scen 'scenario set' / scen1*scen100 / s(scen) 'dynamic secenario subset' k 'benders iterations' / k1*k1000 /; ... // preparatory work loop (k$( NOT done ), ... // setup model for master-problem solve master min obj_master use lp; ... // fix first stage variables loop (scen, ... // setup model for sub-problem option clear=s; s(scen) = yes ; solve sub min obj_sub use lp; ... // process results ); ... // compute cuts for next master ... // free fixed first stage variables ... // set done=1 if convergence criterion is met ); ... // reporting 85
Example: Parallel Benders with mpi4py PMI_RANK=0 PMI_RANK>=1 set scen 'scenario set' / scen1*scen100 / set scen 'scenario set' / scen1*scen100 / s(scen) 'dynamic secenario subset' s(scen) 'dynamic secenario subset' k 'benders iterations' / k1*k1000 /; k 'benders iterations' / k1*k1000 /; ... ... embeddedCode Python: embeddedCode Python: from mpi4py import * from mpi4py import * comm = MPI.COMM_WORLD comm = MPI.COMM_WORLD ... ... pauseEmbeddedCode pauseEmbeddedCode ... // preparatory work ... // preparatory work $ifthen.MPI 0==%sysenv.PMI_RANK% $else.MPI loop (k$( NOT done ), s(scen) = ord (scen)=%sysenv.PMI_RANK%; ... // setup model for master-problem while (1, solve master min obj_master use lp; continueEmbeddedCode : ... // fix first stage variables primal_solution = comm.bcast(None, root=0) // broadcasted data β GAMS data struct. continueEmbeddedCode: comm.bcast([[done]] + <data for sub>, root=0) pauseEmbeddedCode <GAMS data struct.> cut = comm.gather(None, root=0)[1:] abort .noerror$done 'terminating subprocess'; ... // gathered data β GAMS data struct. solve sub min obj_sub use lp; pauseEmbeddedCode <load GAMS data struct.> ... // process results ... // compute cuts continueEmbeddedCode : ... // free fixed first stage variables comm.gather(<subproblem results>), root=0 ) ... // set done=1 if convergence criterion is met pauseEmbeddedCode ); ); continueEmbeddedCode: $endif.MPI comm.bcast([[done],<empty>], root=0) endEmbeddedCode ... // reporting $else.MPI 86
Computational Result(s) β’ Two-stage stochastic problem emerged from energy system model β’ 100 scenarios β’ Deterministic Equivalent: 21,029,101 rows, 23,217,077 columns, 85,721,477 non-zeroes β’ Benders: Master: up to 553 rows, 177 columns, 24,911 non-zeroes β’ Sub: 210,282 rows 232,161 columns 696,461 non-zeroes β’ 19 lines of Python Code + some refactorization of GAMS code for MPI version β’ TIME [sec] sub- master- Method problems problem total 4059.00 Deterministic Equivalent 1 2394.92 0.18 2395.10 Seq. Benders 2 28.35 0.16 28.51 MPI Benders 3 All runs were made with GAMS 25.1.2 on JURECA@JSC with 24 cores per node, 2.5 GHz, (Intel Xeon E5-2680 v3 Haswell), 128 GB RAM 1: single node, 16 cores, CPLEX barrier, no crossover 2: single node, 4 cores per solve statement, CPLEX barrier, advind 0 3: 17 nodes, 404 cores in total, 4 cores per solve statement, CPLEX barrier, advind 0 87
Deployment of GAMS models
Calling GAMS from your Application GAMS SOLVER Application GDX Creating Input for GAMS Model β Data handling using GDX API Callout to GAMS β GAMS option settings using Option API β Starting GAMS using GAMS API Reading Solution from GAMS Model β Data handling using GDX API 89
Low level APIs β Object Oriented API β’ Low level APIs β’ GDX, OPT, GAMSX, GMO, β¦ β’ High performance and flexibility β’ Automatically generated imperative APIs for several languages (C, C++, C#, Delphi, Java, Python, VBA, β¦) β’ Object Oriented GAMS API β’ Additional layer on top of the low level APIs β’ Object Oriented β’ Written by hand to meet the specific requirements of different Object Oriented languages 90
GAMS comes with several OO APIs (Python, Java, C++, C#, β¦) to β’ develop applications β Programming required to build your applications 91
From GAMS Model to Visual Web User Interface 92
From GAMS Model to Visual Web User Interface Basic setup: 1 β Annotating GAMS model (defining the input and output data to be displayed in the WebUI) 93
Basic Setup β GAMS Model Annotations Fully functional interface by only β specifying input and output data Tabular view of input (editable) β and output data Graphical visualization via pivot β charts 94
Basic Setup β GAMS Model Annotations 95
Basic Setup β GAMS Model Annotations 96
Basic Setup β GAMS Model Annotations 97
From GAMS Model to Visual Web User Interface Basic setup: 1 β Annotating GAMS model (defining the input and output data to be displayed in the WebUI) Advanced setup: β Configuration of standard graphics and UI 2 β Sophisticated (custom) graphics (R API) β Scenario management with internal database 98
Advanced Setup β Configuration Configuration via JSON file β Access to a number of pre- β implemented tools for graphical representation Focus on configuration, not β programming 99
Advanced Setup β Sophisticated graphics Sophisticated graphics created in β R can be included as modules Access to the entire R ecosystem β Easily extendable with the wide β spectrum of the R programming language 100
From GAMS Model to Visual Web User Interface Initialization: 1 β Annotating GAMS model (defining the input and output data to be displayed in the WebUI) Advanced setup: β Configuration of standard graphics and UI 2 β Sophisticated (custom) graphics (R API) β Scenario management with internal database Enterprise setup: 3 β User- and Application management 101
Recommend
More recommend
