Xcrypt Highly-Product ive Parallel Script Language Hiroshi - PDF document

Thanks, Cost in ! Xcrypt Highly-Product ive Parallel Script Language Hiroshi Nakashima (ACCMS, Kyot o U.) who is really working wit h t he cooperat ion of Takeshi I washit a and Tasuku Hiraishi Cont ent s � Yet Anot her HPC Programming � Not Only by XcalableMP � Capabilit y and Capacit y Comput ing wit h Script Language � PDCA (Plan-Do-Check-Act ion) Cycle � HP&P Script Language � Design Goal and Concept � Language Syst em Archit ect ure � Current St at us and Near Fut ure Work � Concluding Remark

Yet Anot her HPC Programming Not Only by XcalableMP � Use of a HPC syst em f or R&D ... � is not j ust a single run of a HPC program plan-do-check-act ion � but has many PDCA cycles wit h many runs � HPC applicat ion programming ... � is not limit ed t o f rom-scrat ch wit h Fort ran, C(++), J ava, ... and wit h MPI , OpenMP, XMP... � but includes glue-programming f or; � do-parallel execut ions of a program � int erf acing programs and t ools � PDCA cycle management � ... Yet Anot her HPC Programming Example of C&C Comput ing � Oceanographic Simulat ion � Capabilit y Comput ing � Navier-St okes + Convect ive Heat Xf er + .... � Fort ran + MPI , of course � Capacit y Comput ing � Ensemble Simulat ion wit h various init ial/ boundary condit ions � Fort ran + MPI , why??? � Not only unnecessary but also inef f icient � Do it wit h Script Language !!!

Yet Anot her HPC Programming C&C wit h Script Language � Two-Layered Million-Scale Programming � 10 3 capabilit y x 10 3 capacit y = 10 6 Script Program f or do-parallel exec lower layer of parallel programs = capabilit y t ype = XscalableMP upper layer = capacit y t ype = Highly-Product ive Parallel Script Lang. Yet Anot her HPC Programming Goal=Aut omat ed PDCA Cycle � e.g. Ensemble-Based Dat a Assimilat ion = repeat ed sim t o f ind opt paramet er P: creat e huge size of input dat a D: submit huge number of j obs qsub sim p1 qsub sim p2 qsub sim p3 ... ? ? ? A: f ind t he way t o go next C: check huge size of out put dat a

Yet Anot her HPC Programming Goal=Aut omat ed PDCA Cycle � e.g. Ensemble-Based Dat a Assimilat ion = repeat ed sim t o f ind opt paramet er P: creat e huge size of input dat a D: submit huge number of j obs qsub sim p1 @params= @results= qsub sim p2 qsub sim p3 create_param(@space) submit($job,@params) ... use a_smart_search search(’sim’, ...) ? ? ? @space= @eval= explore(@results) evaluate(@results) A: f ind t he way t o go next C: check huge size of out put dat a HP&P Script Language Why HP & P � Script Language � inherent ly suit able f or programming t o run programs � rich f unct ionalit y f or gluing programs � easy-t o-writ e f or comput er scient ist s � Parallel Script Language � f unct ions t o run programs in parallel e.g. submit many j obs and wait f or t heir complet ion � Highly-Product ive Parallel Script Language � easy-t o-writ e f or comput at ional scient ist s � creat e input f iles f rom a t emplat e easily � ext ract desired lines/ words f rom out put f iles easily

HP&P Script Language Design Goal � Easy-t o-Writ e � even f or guys who never hear of regular expressions, obj ect -orient ed, ... � not requiring more t han 10 lines f or simple paramet er sweeps. � Rich Funct ionalit y � t o implement easy-t o-writ e magic by wizards who support s Muggles. � t o glue applicat ions and GUI (if you love it ), visualizat ion t ools, dat a capt ure t ools, ... HP&P Script Language Design Concept � Must not be MS-Word � easy-t o-writ e does not mean not hing-t o- writ e. � Must not be TEX � rich f unct ionalit y does not mean do-it - yourself f or everyt hing. A � So L TEX in some sense � reasonably easy-t o-writ e and reasonably cust omizable. � encourages st yle-f ile wizards wit h powerf ul built -ins and well-designed st andard int erf aces.

HP&P Script Language Syst em Archit ect ure 1st step 2nd step user script � j ob & j ob set def init ion job interf ace library classes � opt ional def init ion of param � j ob invocat ion/ surveillance/ kill space explorat ion � pre- and (asynchronous) post - processing opt ional script f or j ob input � dat a passing bet ween script & j obs generat ion � abst ract ion of j ob input s/ out put s opt ional script f or j ob out put evaluat ion exploration library job management modules （ J O-engine + α ) =P NQS j ob script generat or j ob scheduler Torque j ob submit t er int erf ace PBS Pro LSF j ob execut ion vigilance SGE j ob complet ion report er HP&P Script Language How I t Looks Like Now package example; use restrict; # module to control job concurrency use parallel; # module to execute jobs in parallel $myjobs=example->new(); $myjobs->{Jobset_exec}= " pathname of program executable "; $myjobs->{Jobset_args}= " printf string for arguments " or & function_to_create_arg_string ; $myjobs->{Jobset_before}= optional pre-job handler* ; $myjobs->{Jobset_after}= optional post-job handler* ; $myjobs->{Par_njob}= number of jobs to submit ; $myjobs->{Par_after_all}= optional finalize handler* ; $myjobs->{Restrict_max}= number of jobs to run concurrently ; $myjobs->start(); # start jobs st ill OO & busy immat ure yet *handler : " string to eval " or & function_to_invoke

HP&P Script Language How I t Can Look Like package example; use restrict; # module to control job concurrency use parallel; # module to execute jobs in parallel example->start( exec=> " pathname of program executable ", args=> " printf string for arguments " or & function_to_create_arg_string , before=> optional pre-job handler* , after=> optional post-job handler* , njob=> number of jobs to submit , after_all=> optional finalize handler* , max=> number of jobs to run concurrently , ); *handler : " string to eval " or & function_to_invoke HP&P Script Language How Built Now package job; j ob scheduler via sub new {...} j ob management sub submit {...} module sub do {...} package jobset; use job; sub new {...} sub before {...} sub after {...} sub start {...} package restrict; package parallel; use restrict; use jobset; use parallel; sub new {...} sub new {...} sub before {...} sub before {...} sub after {...} sub after {...} sub after_all {...} sub start {...} package example; use restrict; use parallel; $myjobs=example->new(); ... $myjobs->start();

HP&P Script Language Next St ep: I / O I nt erf ace For a Paramet er-Sweep of a Simulat ion � What t hey are doing � edit input f ile f or each j ob � submit each j ob manually wit h t he input f ile f or it � view out put f ile t o get what t hey want � What I am doing = writ e perl script s ... � t o creat e j ob-specif ic input f rom t emplat e � t o do qsub wit h various input paramet ers � t o ext ract what I want t o make CSV � What we have t o do = f ind easiest applicat ion-specif ic way ... � t o creat e input , t o submit j ob, t o examine out put � wit h smallest ef f ort t o learn CS sorcery � wit h most sophist icat ed t ech of CS warlock Concluding Remark Let ' s Discuss I ssues on ... � Funct ionalit y v.s. Simplicit y � How do we design a language gradually leading programmers f rom "hello world" t o t he level at which t hey f eel sat isf act ion? � Learning v.s. Teaching � How do we design a language which programmers easily learn and/ or designers easily t each t o t hem? � Can you give a good name t o our script language syst em? We got , Xcrypt , f rom Cost in.