DING T EST A UTOMATION XTENDI OR V IR ON FOR IRTU TUALI LIZATION - - PDF document

▶
ding t est a utomation
SMART_READER_LITE
LIVE PREVIEW

DING T EST A UTOMATION XTENDI OR V IR ON FOR IRTU TUALI LIZATION - - PDF document

Jul 01, 2023 163 likes •395 views

11/1/2013 C HALLENGES E XTE DING T EST A UTOMATION XTENDI OR V IR ON FOR IRTU TUALI LIZATION Sharookh Daru aruwalla Inte tel Corp rporati tion Intel Confidential Do Not Forward C ONT ONTEXT Automation essential to manage todays

slide-1
SLIDE 1

11/1/2013 1

Intel Confidential — Do Not Forward

CHALLENGES EXTE

XTENDI DING TEST AUTOMATION FOR OR VIR IRTU TUALI LIZATION ON

Sharookh Daru aruwalla Inte tel Corp rporati tion

CONT

ONTEXT

  • Automation essential to manage today’s validation workloads
  • Focus mainly for Traditional OS validation (Windows, Linux)
  • Extending for Virtualization OSes is very different
  • In this presentation, the focus is on:

– Challenges when extending Traditional automation frameworks to support Virtualization OSes – Mechanisms useful to overcome them

slide-2
SLIDE 2

11/1/2013 2

AGEND

NDA

  • Background
  • Virtualization Automation Challenges
  • Virtualization Automation Design
  • Key Points & Experience

AGENDA

  • Ba

Back ckground

  • Virtualization Automation Challenges
  • Virtualization Automation Design
  • Key Points & Experience
slide-3
SLIDE 3

11/1/2013 3

TEST AUT

UTOMATION FRAM AMEWORK RKS

Test Automation Frameworks include 3 components

  • Test Schedulers trigger test execution
  • Test Executi

tion Engines run test scripts on SUT

  • Syst

stem Under Test (SUT) = System + OS

Test Scheduler Test Execution Engi gine System under Test Traditional OS Virtualization OS

TRAD

RADITIONAL VS. VIRTUAL ALIZATION

  • Traditional OS – direct and dedicated control
  • f system hardware
  • Virtualization – system hardware is shared
  • Syst

stem to to OS OS ratio change from 1:1 to 1:n

  • Automation Frameworks stuck on 1:1

Tradit dition ional l OS

OS 1 OS 2 OS n VM 1 VM 2 VM n

Virtua uali lizatio ion OS

CPU MEM NIC

Sys ystem

slide-4
SLIDE 4

11/1/2013 4

AGEND

NDA

  • Background
  • Virtualization Automation Ch

Challenges

  • Virtualization Automation Design
  • Key Points & Experience

VIRTUAL

ALIZATION AUT UTOMATION:

: CHAL

ALLENGES

  • Consider a simple test scenario
  • SUT – Virtualization OS

Virtua ual l Host VM 1 VM 2 VM n

Test Execution Engin gine

Chal allenge 1: How to to man anag age dyn ynamic re resources? s??!!

?

slide-5
SLIDE 5

11/1/2013 5

VIRTUAL

ALIZATION AUT UTOMATION:

: CHAL

ALLENGES

Virtua ual l Host VM 1 VM 2 VM n

Test Execution Engin gine

  • Test Execution Engines support Windows or Linux
  • But what OS are running on the VMs??

Win2 n2k8 RHEL6 Win2 Win2k12 ? ?

Chal allenge 2: Need to to support rt multi tiple OSes! s!

?

VIRTUAL

ALIZATION AUT UTOMATION:

: CHAL

ALLENGES

Virtua ual l Host

Test Execution Engin gine

Win2 n2k8 RHEL6 Win2 Win2k12

  • Automation uses “Setup-Test-Teardown” per test
  • Setup-Teardown times dependent on number of VMs
  • Virtualization ≈ 2 * Traditional OS

Chal allenge 3: How to to man anag age long setu tup-teardown times? s?

slide-6
SLIDE 6

11/1/2013 6

VIRTUAL

ALIZATION AUT UTOMATION:

: CHAL

ALLENGES

Virtua ual l Host

Test Execution Engin gine

Win2 n2k8 RHEL6 Win2 Win2k12

Chal allenge 3: How to to man anag age long setu tup-teardown times? s? Chal allenge 1: How to to man anag age dyn ynamic re resources? s??!! Chal allenge 2: Need to to support rt multi tiple OSes! s!

AGENDA

  • Background
  • Virtualization Automation Challenges
  • Virtualization Automation Desi

sign

  • Key Points & Experience
slide-7
SLIDE 7

11/1/2013 7

DESIGN

GN: DYNA YNAMIC RESOURCES

Virtua ual l Host VM 1 VM 2 VM n

Test Execution Engin gine

?

Dynamic Resource Management:

  • Record systems in a test execution
  • Pointer to control various systems

Host OS, Auth, IPAdd, . . . VM1 OS, Auth, IPAdd, . . . VM2 OS, Auth, IPAdd, . . . VMn OS, Auth, IPAdd, . . .

  • Record systems in a test execution
  • System Table holds key information
  • Pointer helps control test execution correctly
  • Operations executed on system pointed by Token

DESIGN

GN: MUL ULTI-OS SUPP UPPORT

  • Libraries categorized into 2 types
  • Common Librari

ries:

  • Features common to all OSes – traditional and

virtualization

  • E.g. SendSystemCommand()
  • Virtu

tualizati tion Libraries:

  • Virtualization specific features only
  • E.g. NewVM()
slide-8
SLIDE 8

11/1/2013 8

DESIGN

GN: TEST CONT ONTINUATION MECHAN ANISM

  • Virtualization Setups include:
  • Host Setup
  • Virtual Machine setup
  • Teardown is just the reverse of setup
  • Setup-teardown times dependent on number of VMs
  • Setup-teardown can dominate total test time

DESIGN

GN: TEST CONT ONTINUATION MECHAN ANISM

Setup Test Teardown

Test 1

Setup Test Teardown

Test 2

Setup Test Teardown

Test 3

Setup Test Teardown

Test 4

Setup Test Teardown

Test 5

Setup Test Teardown

Test 6 Time

  • Group tests with similar setups
  • Full setup for FIRST test
  • Full teardown for LAST test
  • Minimal setup-cleanup within tests

Setup Test Teardown

Test 7

slide-9
SLIDE 9

11/1/2013 9

AGEND

NDA

  • Background
  • Virtualization Automation Challenges
  • Virtualization Automation Design
  • Key Points & Experience

ce

ITE TE-BERTA TEST AUT

UTOMATION FRAM AMEWORK RK

  • Built using in-house tools:
  • Integrated Test Environment (ITE)
  • Berta - test-scheduler
  • Framework extended to support VMware ESXi
  • Other hypervisor (HyperV, KVM) support planned
  • Libraries extended to support Linux
  • Libraries support Dynamic Resource Management
  • Test Continuation in-progress
slide-10
SLIDE 10

11/1/2013 10

PER

ERSONAL EXP XPERI RIENCE

  • Three Firsts – Validation, Automation and Virtualization
  • Re-using existing framework to create common is possible
  • Scripts satisfy dual purpose: Full and Selective Automation
  • Script-in-script execution slow; debugging very painful

THANK

HANK YOU OU

slide-11
SLIDE 11

11/1/2013 11

BACK-UP SLID

LIDES

ITE TE-BERTA TEST AUT

UTOMATION FRAM AMEWORK RK

  • Built using two in-house tools:
  • Integrated Test Environment (ITE)
  • Berta
  • ITE

TE:

  • Test Execution Engine
  • Windows OS Validation
  • Bert

rta:

  • Test Scheduler
  • Maintains test database (SUT, tests, etc.)
  • Provides metrics, send notification, etc.

BERTA TA ITE SUT UT ITE SUT UT ITE SUT UT

slide-12
SLIDE 12

11/1/2013 12

CHAL

ALLENGES: COM OMPLEX SET ETUP-TEARDOWN

Traditional OS (mins) Virtua ualization n OS (mins ns) Establish connection between SUT and Target 25 25 Add required LUNs 4 (2 min/LUN) 10 Setup VMs (DataStores, VMs, OS, LUNs)

  • (10 min/VM) 50

Setup Time: 29 29 85 85 Ru Run Test: (SA SAN IO fo for various packet sizes) 30 30 30 30 Remove VM Setup (delete DataStores, VMs, etc.)

  • (4 min/VM) 20

Release LUNs, SUT and Target connections 24 (4 min/LUN) 20 Teardown Time: 24 24 40 40 Total Test Time: 83 83 155 155

DESIGN

GN: DYNA YNAMIC RESOURCE MAN ANAGEMENT

  • Implement in two stages:
  • Recording new resources
  • Hand control to correct system (token)
  • Extend supports for 3 types of systems:
  • SUT

UT – Initially host

  • PEER – Traffic Partners
  • VM

VM – Test Virtual Machine

  • Toggle control to correct system:
  • Token (controlSystem) hands control
slide-13
SLIDE 13

11/1/2013 13

DESIGN

GN: TEST CONT ONTINUATION MECHAN ANISM

  • Need

to reduce setup-teardown times for virtualization testing!

  • Takes 2x Time versus Traditional OS
  • Test Continuation helps reduce test

times for test groups

  • The Idea involves:
  • Classify/Group tests with similar setups
  • Full setup for FIRST test
  • Minimal clean-up within group
  • Full teardown for LAST test

Start 1st Test in Scenario? Run Setup Procedure Last Test in Scenario? Run Teardown Procedure End Run Minor Clean-up Run Test Run Test Scenario Y N Y N

OTH

THER SLID LIDES

slide-14
SLIDE 14

11/1/2013 14

CONT

ONTEXT

  • Automation been mainly focused for Traditional OSes (Windows, Linux)
  • Extending automation for Virtualization OSes/Hypervisors is very different
  • In this presentation, we present:

– Challenges with extending automated validation for virtualization OSes – Mechanisms used to overcome them

  • We present our internal ITE-Berta Automation Framework used to:
  • Extend a Windows-only automation framework to support Virtualization

Hypervisors (VMware)

STOR

ORAGE NET ETWORKING

  • Involves Storage Area Networks (SAN)
  • SANs are dedicated networks to access consolidated data storage
  • Initiators are servers initiating RD/WR data requests
  • Target

ets are data storage devices that service these requests

  • LUNs are data storage hard-drive (logical unit numbers)
  • Protocols include Fiber Channel (FC), iSCSI, FC over Ethernet (FCoE)

Initiator Fiber Channel Forwarder (FCF) Fiber Channel Switch (FSW) Target Fiber Channel FCoE

slide-15
SLIDE 15

11/1/2013 15

ITE TE-BERTA TEST AUT

UTOMATION FRAM AMEWORK RK

  • No-Touch Automation Solution
  • Monitors for new component and project builds
  • Triggers BAT testing on new builds
  • Triggers Regression testing on BAT-certified builds
  • Notifies key stakeholders

EXAM

XAMPLE LE TEST-CAS ASE SCENA NARIO

  • Comparing Setup, Runtime, Teardown for Traditional vs. Virtualization

Traditional OS Virtualization OS Setup SUT and Target (FCF-FSW setup) Add 1 LUN of required size Add 5 LUNs of required size n/a Create a DataStore on the LUNs n/a Create 5 VMs with Windows/Linux OSes n/a Attach a LUN to each VM Run traffic between SUT and Target Run traffic between VMs and Target Clean-up Setup

slide-16
SLIDE 16

11/1/2013 16

EXAM

XAMPLE LE TEST-CAS ASE SCENA NARIO

ESX Host (SUT) Fiber Channel Forwarder (FCF) Fiber Channel Switch (FSW) Target VM 1 Windows (SUT) Fiber Channel Forwarder (FCF) Fiber Channel Switch (FSW) Target VM 2 VM 5 Fiber Channel FCoE VM Connections

Trad raditional Vi Virtu tualizati tion

AUT

UTOMATION INF NFRA RASTRUCTURE RE SET ETUP

Berta ITE E Controlle ller

MGM GMT Network rk Fiber r Channel

Test case Management System

FSW1 FSW2 FSW4 FSW3 FCF1 FCF2 FCF2

FCoE

Targe gets System under Test (SUT)

slide-17
SLIDE 17

11/1/2013 17

VIRTUAL

ALIZATION AUT UTOMATION:

: CHAL

ALLENGES

  • Virtualization OSes or Hosts allow VMs to work independently
  • Virtualization OS validation involves both Host and VM testing
  • The main challenges:

STATIC RESOU

SOURCE MAN ANAGE GEMENT

SINGL

GLE OS SUPPORT

COMP

OMPLEX AND ND TIME ME-CONSUMING SETUP AND ND TEARD ARDOWN

EXAM

XAMPLE LE TEST SCENARI RIO

ESX X Host (SUT) Fiber Channel Forwa warder (FCF) Fiber Channel Switch ch (FSW) Target VM VM VM VM VM VM

Virtu tualizati tion Trad raditional

Windows ws (SUT) LAN/SAN Traffi fic

slide-18
SLIDE 18

11/1/2013 18

CHAL

ALLENGES:

: STATIC RESOURCE MAN

ANAGEMENT

  • Test execution engines use STATIC resource management

– Resources selected before tests – Resources cannot change within tests – No communication-control with new resources

  • Problems for Virtualization
  • VMs are virtual test resources (SUTs)
  • VMs created within tests
  • Need to communicate with VMs within tests

CHAL

ALLENGES:

: STATIC RESOURCE MAN

ANAGEMENT

Example:

  • Before VMs, SUT  ESXi HOST
  • Later, VMs become the SUT (Host becomes

manager)

  • Run IO to-from the VMs
  • Test execution engine cannot manage a

system that it is not aware of!

Virtua ual l Host (SUT) T) VM 1 VM 2 VM n

slide-19
SLIDE 19

11/1/2013 19

CHAL

ALLENGES:

: STATIC RESOURCE MAN

ANAGEMENT

  • Example Test Scenario:
  • Before VMs, SUT  ESXi HOST
  • Later, VMs become the SUT (Host becomes manager)
  • Test requires running IO between the VMs and Target
  • Test execution engine cannot manage a system that it is not aware of!

ESX X Host (SUT) Fiber Channel Forwa warder (FCF) Fiber Channel Switch ch (FSW) Target VM VM VM VM VM VM

CHAL

ALLENGES:

: SING

NGLE LE OS

S SUPP

UPPORT

  • Test Execution Engines typically support one

OS

  • Most support either Windows or Linux
  • However, VMs created can run either Windows
  • r Linux OS
  • Need to communicate-control with all VMs to

complete test (start IO)

  • Need to extend support for multiple OSes
  • Need to add support for Windows, Linux, ESXi
  • Complicated feature – Compiler support and/or

Library support?

Virtua ual l Host (SUT) T) VM 1 VM 2 VM n

slide-20
SLIDE 20

11/1/2013 20

CHAL

ALLENGES: COM OMPLEX SET ETUP-TEARDOWN

ESX X Host (SUT) Fiber Channel Forwa warder (FCF) Fiber Channel Switch ch (FSW) Target VM VM VM VM VM VM

  • Setup and Teardown occurs for each test in automation
  • Setup/Teardown is complex and time-consuming for virtualization
  • Setup/Remove Host, VMs, etc.
  • Directly dependent on number of VMs

CHAL

ALLENGES: COM OMPLEX SET ETUP-TEARDOWN

  • Setup/Teardown for each test in automation
  • Directly dependent on number of VMs
  • Traditional vs. Virtualization (on-average):
  • Setup takes 3x longer
  • Teardown takes 2x longer
  • Total test time 2x longer for virtualization

Virtua ual l Host (SUT) T) VM 1 VM 2 VM n

In current form, Virtualization Automation would require 2x hardware to run similar number of tests in the same time as Traditional Automation

slide-21
SLIDE 21

11/1/2013 21

CHAL

ALLENGES: COM OMPLEX SET ETUP-TEARDOWN

  • Traditional vs. Virtualization Times (on-average):
  • Setup takes ~3x longer
  • Teardown takes ~2x longer
  • Total test time ~2x longer for virtualization

In current form rm, , Virtu tualizati tion Automati tion would re require 2x har ardware to to run similar r number of te tests s in the sam ame time as as Tra raditi tional Automati tion

VIRTUAL

ALIZATION AUT UTOMATION:

: DESIGN

GN

PowerC rCLI LI C# Wrap rapper Hy Hybrid rid PowerCLI for infrastructure Existing C#, VB for infrastructure Existing C#, VB for infrastructure PowerCLI for all Host Ops PowerCLI in C# for Host Ops PowerCLI in VB for Host Ops PowerCLI for all tests VB for tests VB for tests Native ESX support Leverage existing framework Leverage existing framework Lower Maintenance Specific to ESX Virtualization Double wrapping of ESX cmdlets PowerCLI execution slower Re-invents a new framework Higher Maintenance PowerCLI debugging painful High Maintenance

slide-22
SLIDE 22

11/1/2013 22

VIRTUAL

ALIZATION AUT UTOMATION: DESIGN GN

Test Scheduler

Libraries

Infrastructure (VB Script, C#) Host (PowerCLI)

Test Scripts

Test Execution Engine System under Test

Virtua ualiz lization ion OS Virtua ualiz lization ion OS

SUMM

UMMARY

  • Extending automation for virtualization has challenges:
  • Static resource management
  • Single OS support
  • Long setup and teardown times
  • We presented mechanisms that support:
  • Automation for both traditional and virtualization OSes
  • Dynamic resource management
  • Multiple traditional and virtualization OSes
  • Test Continuation Mechanism (to reduce setup-teardown times)