Xcrypt Highly-Product ive Parallel Script Language Hiroshi - - PDF document

Xcrypt Highly-Product ive Parallel Script Language

Hiroshi Nakashima (ACCMS, Kyot o U.)

wit h t he cooperat ion of Takeshi I washit a and Tasuku Hiraishi

who is really working Thanks, Cost in !

Cont ent s

Yet Anot her HPC Programming

Not Only by XcalableMP Capabilit y and Capacit y Comput ing wit h

Script Language

PDCA (Plan-Do-Check-Act ion) Cycle

HP&P Script Language

Design Goal and Concept Language Syst em Archit ect ure Current St at us and Near Fut ure Work

Concluding Remark

Yet Anot her HPC Programming

Not Only by XcalableMP

Use of a HPC syst em f or R&D ...

is not j ust a single run of a HPC program but has many PDCA cycles wit h many runs

HPC applicat ion programming ...

is not limit ed t o f rom-scrat ch wit h Fort ran,

C(++), J ava, ... and wit h MPI , OpenMP, XMP...

but includes glue-programming f or;

do-parallel execut ions of a program int erf acing programs and t ools PDCA cycle management ...

plan-do-check-act ion

Yet Anot her HPC Programming

Example of C&C Comput ing

Oceanographic Simulat ion

Capabilit y Comput ing

Navier-St okes +

Convect ive Heat Xf er + ....

Fort ran + MPI , of course

Capacit y Comput ing

Ensemble Simulat ion wit h

various init ial/ boundary condit ions

Fort ran + MPI , why???

Not only unnecessary but also inef f icient

Do it wit h Script Language !!!

Yet Anot her HPC Programming

C&C wit h Script Language

Script Program f or do-parallel exec

• f parallel programs

lower layer = capabilit y t ype = XscalableMP upper layer = capacit y t ype = Highly-Product ive Parallel Script Lang.

Two-Layered Million-Scale Programming

103 capabilit y x 103 capacit y = 106

Yet Anot her HPC Programming

Goal=Aut omat ed PDCA Cycle

qsub sim p1 qsub sim p2 qsub sim p3 ...

D: submit huge number of j obs C: check huge size of out put dat a A: f ind t he way t o go next

? ? ?

P: creat e huge size of input dat a

e.g. Ensemble-Based Dat a Assimilat ion

= repeat ed sim t o f ind opt paramet er

Yet Anot her HPC Programming

Goal=Aut omat ed PDCA Cycle

qsub sim p1 qsub sim p2 qsub sim p3 ...

D: submit huge number of j obs C: check huge size of out put dat a A: f ind t he way t o go next

? ? ?

P: creat e huge size of input dat a

@eval= evaluate(@results) @params= create_param(@space) @space= explore(@results) use a_smart_search search(’sim’, ...) @results= submit($job,@params)

e.g. Ensemble-Based Dat a Assimilat ion

= repeat ed sim t o f ind opt paramet er HP&P Script Language

Why HP & P

Script Language

inherent ly suit able f or programming t o run programs rich f unct ionalit y f or gluing programs easy-t o-writ e f or comput er scient ist s

Parallel Script Language

f unct ions t o run programs in parallel

e.g. submit many j obs and wait f or t heir complet ion

Highly-Product ive Parallel Script Language

easy-t o-writ e f or comput at ional scient ist s

creat e input f iles f rom a t emplat e easily ext ract desired lines/ words f rom out put f iles easily

HP&P Script Language

Design Goal

Easy-t o-Writ e

even f or guys who never hear of regular

expressions, obj ect -orient ed, ...

not requiring more t han 10 lines f or simple

paramet er sweeps.

Rich Funct ionalit y

t o implement easy-t o-writ e magic by wizards

who support s Muggles.

t o glue applicat ions and GUI (if you love it ),

visualizat ion t ools, dat a capt ure t ools, ... HP&P Script Language

Design Concept

Must not be MS-Word

easy-t o-writ e does not mean not hing-t o-

writ e.

Must not be TEX

rich f unct ionalit y does not mean do-it -

yourself f or everyt hing.

So L TEX in some sense

reasonably easy-t o-writ e and reasonably

cust omizable.

encourages st yle-f ile wizards wit h powerf ul

built -ins and well-designed st andard int erf aces. A

HP&P Script Language

Syst em Archit ect ure

user script

j ob & j ob set def init ion

• pt ional def init ion of param

space explorat ion

• pt ional script f or j ob input

generat ion

• pt ional script f or j ob out put

evaluat ion

job interf ace library classes

j ob invocat ion/ surveillance/ kill pre- and (asynchronous) post -

processing

dat a passing bet ween script & j obs abst ract ion of j ob input s/ out put s

exploration library

job management modules

（ =P J O-engine + α) j ob script generat or j ob submit t er j ob execut ion vigilance j ob complet ion report er

j ob scheduler int erf ace

NQS Torque PBS Pro LSF SGE

1st step 2nd step

HP&P Script Language

How I t Looks Like Now

package example; use restrict; # module to control job concurrency use parallel; # module to execute jobs in parallel $myjobs=example->new(); $myjobs->{Jobset_exec}= "pathname of program executable"; $myjobs->{Jobset_args}= "printf string for arguments" or &function_to_create_arg_string; $myjobs->{Jobset_before}= optional pre-job handler*; $myjobs->{Jobset_after}= optional post-job handler*; $myjobs->{Par_njob}= number of jobs to submit; $myjobs->{Par_after_all}= optional finalize handler*; $myjobs->{Restrict_max}= number of jobs to run concurrently; $myjobs->start(); # start jobs *handler : "string to eval" or &function_to_invoke

st ill OO & busy immat ure yet

HP&P Script Language

How I t Can Look Like

package example; use restrict; # module to control job concurrency use parallel; # module to execute jobs in parallel example->start( exec=> "pathname of program executable", args=> "printf string for arguments" or &function_to_create_arg_string, before=> optional pre-job handler*, after=> optional post-job handler*, njob=> number of jobs to submit, after_all=> optional finalize handler*, max=> number of jobs to run concurrently, ); *handler : "string to eval" or &function_to_invoke

HP&P Script Language

How Built Now

package example; use restrict; use parallel; $myjobs=example->new(); ... $myjobs->start(); package restrict; use restrict; use parallel; sub new {...} sub before {...} sub after {...} package jobset; use job; sub new {...} sub before {...} sub after {...} sub start {...} package job; sub new {...} sub submit {...} sub do {...} package parallel; use jobset; sub new {...} sub before {...} sub after {...} sub after_all {...} sub start {...}

j ob scheduler via j ob management module

HP&P Script Language

Next St ep: I / O I nt erf ace

For a Paramet er-Sweep of a Simulat ion

What t hey are doing

edit input f ile f or each j ob submit each j ob manually wit h t he input f ile f or it view out put f ile t o get what t hey want

What I am doing = writ e perl script s ...

t o creat e j ob-specif ic input f rom t emplat e t o do qsub wit h various input paramet ers t o ext ract what I want t o make CSV

What we have t o do

= f ind easiest applicat ion-specif ic way ...

t o creat e input , t o submit j ob, t o examine out put wit h smallest ef f ort t o learn CS sorcery wit h most sophist icat ed t ech of CS warlock

Concluding Remark

Let ' s Discuss I ssues on ...

Funct ionalit y v.s. Simplicit y

How do we design a language gradually

leading programmers f rom "hello world" t o t he level at which t hey f eel sat isf act ion?

Learning v.s. Teaching

How do we design a language which

programmers easily learn and/ or designers easily t each t o t hem?

Can you give a good name t o our script

language syst em?

We got , Xcrypt , f rom Cost in.