Employing Task Parallelism to Facilitate Dynamic Comparison of - - PowerPoint PPT Presentation

Employing Task Parallelism to Facilitate Dynamic Comparison of Model Output

Ha nna h Aize nma n (CUNY Gra d ua te Ce nte r, NOAA-CRE ST a t CCNY) Me nto rs: Da ve Bro wn, Ma ry Ha le y (NCAR-CISL / VE T S) Co ntrib uto rs: She ri Mic ke lso n (ANL ), Susa n Ba te s, Go kha n Da na b a so g lu (NCAR-CGD)

OMWG POP Diagnostics

Swift POP Diagnostics

• Pa ra lle lize s e xe c utio n o f NCL

sc ripts

• 3X spe e d up o ve r No n-Swift ve rsio n

Only supported on DASG machines

Even on DASG…

NCAR’s mission is “to support, enhance, and extend the capabilities

• f the university community and the

broader scientific community, nationally and internationally”

• http://ncar.ucar.edu/about-ncar

Service Oriented Architecture

Why SOA?

• Se rvic e s sta y inde pe nde nt
• Dia g no stic s c a n b e a dde d inde pe nde nt o f o the r se rvic e s
• Ne w se rvic e s c a n use o ld o ne s witho ut c ha ng ing the o lds o ne s
• Multiple c lie nts c a n ta lk to the sa me se rve r
• CL

I , mo b ile , de skto p, …

• Ca n ke e p o ld UI

up while ro lling o ut ne w o ne

• Clie nts a nd se rvic e s do no t ne e d to b e o n the sa me

ma c hine

• Slo w se rvic e s (the dia g no stic s) do no t inte rfe re with

fa st o ne s (da ta , o b se rva tio ns)

RESTful SOA

Start a run

Get the results

URL->File

Why pydiag?

Automate running the diagnostics

• T

urn use r de fine d se tting s into inputs

• c a se / run, o b se rva tio ns, ye a rs, a nd plo ts
• I

nputs a re pytho n dic tio na rie s

• Ge ne ra te Swift c o nfig ura tio n file s
• site s.xml c ha ng e s with e ve ry ne w o utput dire c to ry
• site s.xml a nd tc .da ta a re ma c hine de pe nde nt
• Co nfig s a re dic tio na rie s a nd lists o f dic ts
• F

ile s g e ne ra te xml, tsv, c dm, a nd c o nfig ura tio n

• I

so la te de pe nde nc ie s

• T

urn it into a n impo rta b le lib ra ry

CSH->Python

Deployment

• Re q uire s a ny lib ra rie s use d b y the dia g no stic s
• Se lf-c o nta ine d e nviro nme nt
• I

so la te d thro ug h use o f virtua le nv

• All the pa c ka g e s a re in PyPi
• Only o the r de pe nde nc ie s
• lib e ve nt a nd a c c e ss to the da ta se rve rs (g la de , hpss)
• Wo rks b e tte r o n c luste r
• Ma inte na nc e a nd e xte nsib ility:
• Mo dula r a rc hite c ture
• Unit, inte g ra tio n, a nd func tio na l re g re ssio n te sts
• Do c ume nta tio n c a n b e c o mpile d into we b pa g e s, b o o ks, e tc
• pydia g , whic h is sta nd-a lo ne , is c o mpa tib le with

Pytho n2.4 thro ug h Pytho n2.7

Server Side

mik.ucar.edu Use any of the documented/sample urls

Client Side

Acknowledgments

• Da ve Bro wn a nd Ma ry Ha ly (NCAR/ CI

SL )

• She ri Mic ke lso n (ANL

)

• Susa n Ba te s a nd Go kha n Da na b a so g lu

(NCAR/ CGD)

• Pa ul Go o dma n a nd Na tha n Wilhe lmi (NCAR-CI

SL )

• We i Hua ng a nd Ric k Bro wnrig g (NCAR/ CI

SL )