Tutorial: Advanced (Batch) Job Scriptjng Robert Barthel, SCC, KIT - PowerPoint PPT Presentation

Tutorial: Advanced (Batch) Job Scriptjng Robert Barthel, SCC, KIT Steinbuch Centre for Computjng (SCC) www.bwhpc-c5.de Funding:

How to read the following slides Abbreviatjon/Colour code Full meaning prompt of the interactjve shell $ c o m m a n d - o p t i o n $ = The full prompt may look like: v a l u e u s e r @ m a c h i n e : p a t h $ The command has been entered in the interactjve shell session Placeholder for integer, string etc < i n t e g e r > < > = < s t r i n g > Metasyntactjc variables f o o , b a r $ { W O R K S H O P } / o p t / b w h p c / k i t / w o r k s h o p / 2 0 1 6 - 1 2 - 0 6 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 2

Goal Be descriptjve! Header Header Comment your code e.g. via headers sectjons of script and functjons. Declaratjons Declaratjons (MSUB + defaults) Decipherable names for variables and functjons (MSUB + defaults) Functjons Functjons Organise and structure! Input handling Break complex scripts into simpler blocks Input handling e.g. use functjons Use exit codes Use standardized parameter fmags for script Main sectjon Main sectjon invocatjon. Write job script that runs interactjvely Footer → Then add part for MOAB Footer 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 3

Best Practjses – Common Problems (1) Do not Run your code/applicatjon/job in $ { H O M E } Multjnode Job: use $ { W O R K } Producing Tbyte of scratch fjles + <10000 File: Change your applicatjon code if you do not know how: Apply for Tiger Team Support (htups://www.bwhpc-c5.de/en/zas_info_tjgerteamsupport.php) Singlenode Job: use $ : HowTo → Case 1 { T M P D I R } Chain jobs HowTo → Case 2 Many sequentjal tjny jobs: Bundle to one big job: HowTo → Case 3 Use of MPI/OpenMP Parallelisatjon in jobs: Tutorial „best practjce - parallelizatjon and compiling“ 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 4

Rev: MOAB variables htup://www.bwhpc-c5.de/wiki/index.php/Batch_Jobs#Moab_Environment_Variables MSUB variables: # ! / b i n / b a s h # M S U B - N t e s t # M S U B - l a d v r e s = w o r k s h o p . 8 , n o d e s = 1 : p p n = 1 , m e m = 5 0 m b # M S U B - l w a l l t i m e = 0 0 : 0 5 : 0 0 # M S U B - m n # M S U B - v m y _ o w n _ v a r i a b l e s = “ a r g u m e n t s “ 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 5

Case 1: Jobs @ $TMPDIR (1) If temporary fjles of job > Gbyte → Run your job at $ { T M P D I R } but ONLY if single node jobs What to do: Generate subdirectory under $ { T M P D I R } = > $ { r u n _ D I R } Copy to $ { r u n _ D I R } Change to $ & program executjon { r u n _ D I R } Copy results to start DIR How? Start with templates: $ { W O R K S H O P } / e x e r c i s e s / 0 7 / 0 1 _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h $ { W O R K S H O P } / e x e r c i s e s / 0 7 / 0 1 _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h + + $ { W O R K S H O P } / e x e r c i s e s / 0 7 / { 0 1 _ g e n _ f i l e s , 0 1 _ g e n _ f i l e s . i n p } $ { W O R K S H O P } / e x e r c i s e s / 0 7 / { 0 1 _ g e n _ f i l e s , 0 1 _ g e n _ f i l e s . i n p } 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 6

Case 1: Jobs @ $TMPDIR (2) Code snip: $ { W O R K S H O P } / e x e r c i s e s / 0 7 / 0 1 _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h # ! / b i n / b a s h . . . T A S K / T o D o : 1 0 m i n T A S K / T o D o : 1 0 m i n # # a ) T u t o r i a l T o D o : l o a d m o d u l e s I N T E L + M K L * G e n e r a l i s e b l u e c o d e i f n o t l o a d e d * G e n e r a l i s e b l u e c o d e a v o i d i n g r e p e t i t i o n a v o i d i n g r e p e t i t i o n # # b ) D e f i n e y o u r r u n d i r e c t o r y u n d e r t m p d i r * W r i t e c o d e f o r a - h * W r i t e c o d e f o r a - h # # i n c o r p o r a t i n g u s e r n a m e a n d J o b I D / P I D * R e d i r e c t o u t p u t o f b i n a r y * R e d i r e c t o u t p u t o f b i n a r y m k d i r - p v " $ { T M P } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " # # c ) T u t o r i a l T o D o : C h e c k e x i s t e n c e o f r u n d i r e c t o r y # # d ) C o p y f i l e s f r o m s u b m i t d i r e c t o r y # # t o r u n d i r e c t o r y c d $ M O A B _ S U B M I T D I R c p - p v g e n _ f i l e s . x " $ { T M P } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " # # C h e c k i f c o p y s u c c e e d e d c p - p v g e n _ f i l e s . i n p " $ { T M P } / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " # # e ) C h a n g e t o r u n d i r e c t o r y ( c h e c k i f s u c c e e d e d ) a n d s t a r t b i n a r y + i n p u t f i l e c d " $ { T M P } / $ { U S E R } . $ { M O A B _ J O B I D } " . / 0 1 _ g e n _ f i l e s . x 0 1 _ g e n _ f i l e s . i n p # # f ) T u t o r i a l T o D o : c h e c k r u n s t a t u s # # g ) t r a n s f e r f i l e s t o s u b m i t d i r e c t o r y c p - p v f i l e s _ * . o u t " $ { M O A B _ S U B M I T D I R } " # # h ) T u t o r i a l T o D o : c l e a n u p r u n _ D I R 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 7

Case 1: Jobs @ $TMP (3) Decl. + a-c: $ { W O R K S H O P } / s o l u t i o n s / 0 7 / 0 1 _ g e n e r a l i s e d _ j o b _ r u n _ u n d e r _ l o c a l _ t m p d i r . s h # # 1 ) D e f i n e f u l l p a t h o f y o u r b i n a r y E X E = " 0 1 _ g e n _ f i l e s . x " S o l u t i o n ! S o l u t i o n ! # # 2 ) D e f i n e o u t p u t f i l e # # = N a m e o f e x e c u t a b l e + J O B I D o r P I D o u t p u t = " $ ( b a s e n a m e $ { E X E } ) _ $ { M O A B _ J O B I D : - $ $ } . l o g " # # 3 ) D e f i n e i n p u t f i l e s i n p u t = " 0 1 _ g e n _ f i l e s . i n p " # # 4 ) D e f i n e i n p u t f i l e s t o b e c o p i e d c o p y _ l i s t = " $ { E X E } $ { i n p u t } " # # 5 ) D e f i n e f i l e s t o b e c o p i e d b a c k a f t e r r u n , i . e . o u t p u t f i l e s a v e _ l i s t = " $ { o u t p u t } f i l e s _ * . o u t " # # a ) L o a d m o d u l e s I N T E L + M K L i f n o t l o a d e d f o r m o d i n c o m p i l e r / i n t e l n u m l i b / m k l ; d o m o d u l e l i s t 2 > & 1 | g r e p " $ { m o d } " > / d e v / n u l l | | m o d u l e l o a d " $ { m o d } " d o n e # # b ) D e f i n e y o u r r u n d i r e c t o r y a n d a n d g e n e r a t e v i a m k d i r r u n _ D I R = " $ T M P / $ { U S E R } . $ { M O A B _ J O B I D : - $ $ } " m k d i r - p v " $ { r u n _ D I R } " # # c ) C h e c k e x i s t e n c e o f r u n d i r e c t o r y i f [ ! - d " $ { r u n _ D I R } " ] ; t h e n e c h o " E R R O R : R u n D I R = $ { r u n _ D I R } d o e s n o t e x i s t " ; e x i t 1 f i 06/12/16 Tutorial: Adv. Batch Job Scriptjng / R. Barthel 8