DING T EST A UTOMATION XTENDI OR V IR ON FOR IRTU TUALI LIZATION - PDF document

11/1/2013 C HALLENGES E XTE DING T EST A UTOMATION XTENDI OR V IR ON FOR IRTU TUALI LIZATION Sharookh Daru aruwalla Inte tel Corp rporati tion Intel Confidential — Do Not Forward C ONT ONTEXT  Automation essential to manage today’s validation workloads  Focus mainly for Traditional OS validation (Windows, Linux)  Extending for Virtualization OSes is very different  In this presentation, the focus is on: – Challenges when extending Traditional automation frameworks to support Virtualization OSes – Mechanisms useful to overcome them 1

11/1/2013 A GEND NDA  Background  Virtualization Automation Challenges  Virtualization Automation Design  Key Points & Experience A GENDA  Ba Back ckground  Virtualization Automation Challenges  Virtualization Automation Design  Key Points & Experience 2

11/1/2013 T EST A UT UTOMATION F RAM RKS AMEWORK Test Scheduler Test Automation Frameworks include 3 components  Test Schedulers trigger test execution Test Execution Engi gine  Test Executi tion Engines run test scripts on SUT  Syst stem Under Test (SUT) = System + OS System under Test Virtualization OS Traditional OS T RAD RADITIONAL V S . V IRTUAL ALIZATION  Traditional OS – direct and dedicated control CPU MEM NIC of system hardware Sys ystem  Virtualization – system hardware is shared Virtua Tradit uali dition lizatio ional ion OS l OS  Syst stem to to OS OS ratio change from 1:1 to 1:n VM 1 VM 2 VM n OS 1 OS 2 OS n  Automation Frameworks stuck on 1:1 3

11/1/2013 A GEND NDA  Background  Virtualization Automation Ch Challenges  Virtualization Automation Design  Key Points & Experience V IRTUAL ALIZATION A UT UTOMATION : : C HAL ALLENGES Test Execution  Consider a simple test scenario Engin gine Chal allenge 1:  SUT – Virtualization OS How to to man anag age dyn ynamic re resources? s??!! ? Virtua ual l Host VM 1 VM 2 VM n 4

11/1/2013 V IRTUAL ALIZATION A UT UTOMATION : : C HAL ALLENGES Test Execution  Test Execution Engines support Windows or Linux Engin gine  But what OS are running on the VMs?? Virtua ual l Host Chal allenge 2: Need to to support rt multi tiple OSes! s! ? ? Win2 VM 1 n2k8 VM 2 RHEL6 ? Win2 Win2k12 VM n V IRTUAL ALIZATION A UT UTOMATION : : C HAL ALLENGES Test Execution  Automation uses “Setup -Test- Teardown” per test Engin gine  Setup-Teardown times dependent on number of VMs  Virtualization ≈ 2 * Traditional OS Virtua ual l Host Chal allenge 3: How to to man anag age long setu tup-teardown times? s? Win2 n2k8 RHEL6 Win2 Win2k12 5

11/1/2013 V IRTUAL ALIZATION A UT UTOMATION : : C HAL ALLENGES Chal allenge 1: Test Execution Engin gine How to to man anag age dyn ynamic re resources? s??!! Chal allenge 2: Virtua ual l Host Need to to support rt multi tiple OSes! s! Chal allenge 3: How to to man anag age long setu tup-teardown times? s? Win2 n2k8 RHEL6 Win2k12 Win2 A GENDA  Background  Virtualization Automation Challenges  Virtualization Automation Desi sign  Key Points & Experience 6

11/1/2013 D ESIGN GN : D YNA YNAMIC R ESOURCES Dynamic Resource Management: Test Execution Engin gine  Record systems in a test execution  Record systems in a test execution  Pointer to control various systems • System Table holds key information  Pointer helps control test execution correctly ? Virtua ual l Host • Operations executed on system pointed by Token Host OS, Auth, IPAdd, . . . VM1 OS, Auth, IPAdd, . . . VM2 OS, Auth, IPAdd, . . . VM n OS, Auth, IPAdd, . . . VM 1 VM 2 VM n D ESIGN GN : M UL ULTI -OS S UPP UPPORT  Libraries categorized into 2 types  Common Librari ries:  Features common to all OSes – traditional and virtualization  E.g. SendSystemCommand()  Virtu tualizati tion Libraries:  Virtualization specific features only  E.g. NewVM() 7

11/1/2013 D ESIGN GN : T EST C ONT ONTINUATION M ECHAN ANISM  Virtualization Setups include:  Host Setup  Virtual Machine setup  Teardown is just the reverse of setup  Setup-teardown times dependent on number of VMs  Setup-teardown can dominate total test time D ESIGN GN : T EST C ONT ONTINUATION M ECHAN ANISM Setup Test Teardown Test 1  Group tests with similar setups Test 2 Setup Test Teardown Test 3 Setup Test Teardown  Full setup for FIRST test Test 4 Setup Test Teardown  Full teardown for LAST test Test 5 Setup Test Teardown Setup Test Teardown Test 6  Minimal setup-cleanup within tests Setup Test Teardown Test 7 Time 8

11/1/2013 A GEND NDA  Background  Virtualization Automation Challenges  Virtualization Automation Design  Key Points & Experience ce ITE TE-BERTA T EST A UT UTOMATION F RAM RK AMEWORK  Built using in-house tools:  Integrated Test Environment (ITE)  Berta - test-scheduler  Framework extended to support VMware ESXi  Other hypervisor (HyperV, KVM) support planned  Libraries extended to support Linux  Libraries support Dynamic Resource Management  Test Continuation in-progress 9

11/1/2013 P ER ERSONAL E XP RIENCE XPERI  Three Firsts – Validation, Automation and Virtualization  Re-using existing framework to create common is possible  Scripts satisfy dual purpose: Full and Selective Automation  Script-in-script execution slow; debugging very painful T HANK HANK Y OU OU 10

11/1/2013 B ACK -U P S LID LIDES ITE TE-BERTA T EST A UT UTOMATION F RAM RK AMEWORK  Built using two in-house tools:  Integrated Test Environment (ITE) BERTA TA  Berta  ITE TE: • Test Execution Engine ITE ITE ITE • Windows OS Validation  Bert rta: • Test Scheduler SUT UT SUT UT SUT UT • Maintains test database (SUT, tests, etc.) • Provides metrics, send notification, etc. 11

11/1/2013 C HAL ALLENGES : C OM OMPLEX S ET ETUP -T EARDOWN Traditional OS Virtua ualization n OS (mins) (mins ns) Establish connection between SUT and Target 25 25 Add required LUNs 4 (2 min/LUN) 10 Setup VMs (DataStores, VMs, OS, LUNs) - (10 min/VM) 50 Setup Time: 29 29 85 85 Ru Run Test: (SA SAN IO fo for various packet sizes) 30 30 30 30 Remove VM Setup (delete DataStores, VMs, etc.) - (4 min/VM) 20 Release LUNs, SUT and Target connections 24 (4 min/LUN) 20 Teardown Time: 24 24 40 40 Total Test Time: 83 83 155 155 D ESIGN GN : D YNA YNAMIC R ESOURCE M AN ANAGEMENT  Implement in two stages:  Recording new resources  Hand control to correct system (token)  Extend supports for 3 types of systems:  SUT UT – Initially host  PEER – Traffic Partners  VM VM – Test Virtual Machine  Toggle control to correct system:  Token ( controlSystem ) hands control 12

11/1/2013 D ESIGN GN : T EST C ONT ONTINUATION M ECHAN ANISM  Need to reduce setup-teardown times for Start virtualization testing! Run Test  Takes 2x Time versus Traditional OS Scenario  Test Continuation helps reduce test 1 st Test in Y Run Setup times for test groups Scenario? Procedure N  The Idea involves: Run Test  Classify/Group tests with similar setups  Full setup for FIRST test Run Minor N Y Run Teardown Last Test in Clean-up Procedure Scenario?  Minimal clean-up within group End  Full teardown for LAST test O TH THER S LID LIDES 13

11/1/2013 C ONT ONTEXT  Automation been mainly focused for Traditional OSes (Windows, Linux)  Extending automation for Virtualization OSes/Hypervisors is very different  In this presentation, we present: – Challenges with extending automated validation for virtualization OSes – Mechanisms used to overcome them  We present our internal ITE-Berta Automation Framework used to:  Extend a Windows-only automation framework to support Virtualization Hypervisors (VMware) S TOR ORAGE N ET ETWORKING  Involves Storage Area Networks (SAN)  SANs are dedicated networks to access consolidated data storage  Initiators are servers initiating RD/WR data requests  Target ets are data storage devices that service these requests  LUNs are data storage hard-drive (logical unit numbers)  Protocols include Fiber Channel (FC), iSCSI, FC over Ethernet (FCoE) Fiber Channel FCoE Fiber Channel Fiber Channel Initiator Target Forwarder (FCF) Switch (FSW) 14

11/1/2013 ITE TE-BERTA T EST A UT UTOMATION F RAM RK AMEWORK  No-Touch Automation Solution • Monitors for new component and project builds • Triggers BAT testing on new builds • Triggers Regression testing on BAT-certified builds • Notifies key stakeholders E XAM LE T EST -C AS ASE S CENA NARIO XAMPLE  Comparing Setup, Runtime, Teardown for Traditional vs. Virtualization Traditional OS Virtualization OS Setup SUT and Target (FCF-FSW setup) Add 1 LUN of required size Add 5 LUNs of required size n/a Create a DataStore on the LUNs n/a Create 5 VMs with Windows/Linux OSes n/a Attach a LUN to each VM Run traffic between SUT and Target Run traffic between VMs and Target Clean-up Setup 15