Jozef Cernak, Marek Kocan, Eva Cernakova (P. J. Safarik University - PowerPoint PPT Presentation

ARC tools for revision and nightly functional tests Jozef Cernak, Marek Kocan, Eva Cernakova (P. J. Safarik University in Kosice, Kosice, E M Slovak Republic) I N I F S O - R I - E 2 M 6 1 6 I N 1 I

Outline • Testing strategy in ARC • ARC-EMI testing tools – Build testing • Nightly (every night) • Revision (after each commit of developers) – Automatic functional testing of nightly builds • Future plans – Increasing number of automatic functional tests – Automatic performance testing • Conclusion E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 2 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Testing practice in ARC • Several roles in project: developers, testers and ARC release manager • Adopted tests: – Unit tests(developers) – Build tests (testers) • Revision (Kosice) • Nightly (Copenhagen) • EMI-RC (ETICS) – Functional tests (testers, developers) • Manual • Automatic (Kosice) – Regression testing (testers, developers) – Performance testing (testers) • Manual • Automatic in development phase – Large scale testing (in preparation phase) E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 3 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

ARC-EMI testing tools Relation between ARC and EMI testing tools E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 4 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Block diagram of automatic revision and Functional tools Simple database server with PHP and Python scripts E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 5 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Technical details • Infrastructure (local cluster) – 2 SUN 2200M2 servers (2x4GB RAM, 250GB disk, AMD 64 bits) – 2 desktops (2x 1GB RAM, 500GB disk 32 bits) – 1 MAC mini (2GB RAM, 160GB, 32 bits) • Software – MySQL, PHP, Python – Code is free • Distributed data processing – SVN E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 6 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Revision automatic testing • Features – Supported are several platforms of OS: • CentOS (32,64), • Debian(32,64), • and MAX OS (64) – Detailed identification of errors during building process • Autogen • Configuration • make • make dist • Build E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 7 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

…continued. – User friendly interface to define search criteria • Daily • Weakly • Monthly • Custom – Revision number, etc (see next examples) • Direct connection with SVN • Warning messages for developers if code is broken – CCCC metric – Quick mechanisms to identify breaking event E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 8 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Example of www interface for automatic testing tools Home page: http://arc-emi.grid.upjs.sk/ (Revision database is slow) E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 9 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Several criteria to search results of revision testing E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 10 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Example of the results of daily revisions E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 11 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Automatic functional tests • Features – Automatic download code from SVN based on several criteria: • trunk, • revision code, • nightly builds – Automatic build – Automatic deployment – User friendly interface for: • submitting proposals of test cases • to search the results of functional tests – Tests are grouped into two main groups: • functional tests of server, • functional tests of client. E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 12 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Interface to enter search criteria E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 13 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

The results of automatic functional tests E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 14 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Functional test scenaria • client_arcsub_to_arex_gridmap_simplejob_jsdl – Tries to submit a simple JSDL job to a grid-map secured A-REX service. • client_arcsub_to_grid-manager_gridmap_simplejob_jsdl – Tries to submit a simple JSDL job to a grid-map secured ARC0 CE. • client_arcsub_to_GIIS_simplejob_jsdl – Submission of a simple JSDL job to a GIIS service. • client_arcsub_to_ISIS_simplejob_jsdl – Submission of a simple JSDL job to an ISIS cloud. • client_arcsub_random_broker_isis_simplejob_jsdl – Submission of a simple JSDL job to an ISIS cloud using random broker. • client_arcsub_random_broker_giis_simplejob_jsdl – Submission of a simple JSDL job to an GIIS cloud using random broker. • client_arcsub_fastestqueue_broker_isis_simplejob_jsdl – Submission of a simple JSDL job to an ISIS cloud using fastest queue broker. • client_arcsub_fastestqueue_broker_giis_simplejob_jsdl – Submission of a simple JSDL job to a GIIS service using fastest queue broker. • Failed – There was some error before we managed to start testing. The error could have been in code E M retrieval, installation or somewhere else. I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 15 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Automatic performance tests • Requirements: – several servers and clients – Monitoring of several distributed processes: • resources: – Memory – CPU – Network usage • Performance – Scalability – Number of concurrent requests • Reliability of services: – Ratio between successful tasks and total tasks. Examples of tasks: job submission, file transfer and etc. E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 16 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

… continued. • Proposed infrastructure: – Grid servers and clients, we plan to use existing infrastructure – Communication layer • XML-RPC client-server architecture – Central database server – Common Python library on each server and client (for example start stop service) – Central control program E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 17 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Conclusions • Automatic testing covers: – Build process (ARC, ETICS), – Functional, – Performance tests (in progress). • Utilization of automatic test tools increases probability to find weak part of the software. On the other hand, manual E tests are important. M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 18 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

References • Testing in ARC – http://wiki.nordugrid.org/index.php/Testing • Revision tests – http://download.nordugrid.org/builds/ – http://arc-emi.grid.upjs.sk/revisionTests.php • Functional tests – http://arc-emi.grid.upjs.sk/functionalTests.php • Code: – http://svn.nordugrid.org/trac/workarea/browser/ARCT estScriptshttp://svn.nordugrid.org/trac/workarea/bro wser/ARCTestScripts E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 19 I - E Footer to set 2 M 6 1 this field 6 I N 1 I

Thank you EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611 E M I N I F S Use Insert O - Header & R Use Insert Header & Footer to set this field 20 I - E Footer to set 2 M 6 1 this field 6 I N 1 I