Integration, System and Regression Testing Automated testing and - PowerPoint PPT Presentation

Integration, System and Regression Testing Automated testing and J.P .Galeotti Alessandra Gorla verification Thursday, January 31, 13

Actual Needs and Delivered User Acceptance (alpha, beta test) Constraints Package Review System System System Test Specifications Integration Analysis / Review Subsystem Integration Test Subsystem Design/Specs Analysis / Review Unit/Component Unit/ Module Test Specs Components User review of external behavior as it is determined or becomes visible Validation Legend Verification (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

What is integration testing? Module test Integration test System test Interface specs, module Requirements Specification: Module interface breakdown specification Modular structure Visible structure: Coding details — none — (software architecture) Scaffolding Some Often extensive Some required: Looking for faults Interactions, System Modules in: compatibility functionality (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Integration versus Unit Testing • Unit (module) testing is a necessary foundation – Unit level has maximum controllability and visibility – Integration testing can never compensate for inadequate unit testing • Integration testing may serve as a process check – If module faults are revealed in integration testing, they signal inadequate unit testing – If integration faults occur in interfaces between correctly implemented modules, the errors can be traced to module breakdown and interface specifications (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Integration Faults • Inconsistent interpretation of parameters or values – Example: Mixed units (meters/yards) in Martian Lander • Violations of value domains, capacity, or size limits – Example: Bu ff er overflow • Side e ff ects on parameters or resources – Example: Conflict on (unspecified) temporary file • Omitted or misunderstood functionality – Example: Inconsistent interpretation of web hits • Nonfunctional properties – Example: Unanticipated performance issues • Dynamic mismatches – Example: Incompatible polymorphic method calls (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Example: A Memory Leak Apache web server, version 2.0.48 Response to normal page request on secure (https) port static void ssl io filter disable(ap filter t *f) { No obvious error, but Apache bio filter in ctx t *inctx = f->ctx; leaked memory slowly (in normal inctx->ssl = NULL; use) or quickly (if exploited for a DOS attack) inctx->filter ctx->pssl = NULL; } (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Example: A Memory Leak Apache web server, version 2.0.48 Response to normal page request on secure (https) port static void ssl io filter disable(ap filter t *f) { bio filter in ctx t *inctx = f->ctx; The missing code is for a structure defined and SSL_free(inctx -> ssl); created elsewhere , inctx->ssl = NULL; accessed through an opaque pointer. inctx->filter ctx->pssl = NULL; } (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Example: A Memory Leak Apache web server, version 2.0.48 Response to normal page request on secure (https) port static void ssl io filter disable(ap filter t *f) { bio filter in ctx t *inctx = f->ctx; Almost impossible to find with unit testing. (Inspection and SSL_free(inctx -> ssl); some dynamic techniques inctx->ssl = NULL; could have found it.) inctx->filter ctx->pssl = NULL; } (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Maybe you’ve heard ... Yes, I implemented ⟨ module A ⟩ , but I didn’t test it thoroughly yet. It will be tested along with ⟨ module B ⟩ when that’s ready. (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Translation... Yes, I implemented I didn’t think at all ⟨ module A ⟩ , but I didn’t about the strategy for testing. I didn’t design test it thoroughly yet. It ⟨ module A ⟩ for will be tested along with testability and I didn’t ⟨ module B ⟩ when that’s think about the best ready. order to build and test modules ⟨ A ⟩ and ⟨ B ⟩ . (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Integration Plan + Test Plan • Integration test plan drives and is driven ... by the project “build ... plan” Build Plan Test Plan – A key feature of the system architecture ... and project plan System Architecture (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Big Bang Integration Test An extreme and desperate approach: Test only after integrating all modules + Does not require sca ff olding • The only excuse, and a bad one - Minimum observability, diagnosability, e ffi cacy, feedback - High cost of repair • Recall: Cost of repairing a fault rises as a function of time between error and repair (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Structural and Functional Strategies • Structural orientation : Modules constructed, integrated and tested based on a hierarchical project structure – Top-down, Bottom-up, Sandwich • Functional orientation : Modules integrated according to application characteristics or features – Threads, Critical module (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Top down . Top stub A stub B stub C Working from the top level (in terms of “use” or “include” relation) toward the bottom. No drivers required if program tested from top- level interface (e.g. GUI, CLI, web app, etc.) (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Top down .. Top A stub B stub C Write stubs of called or used modules at each stub X stub Y step in construction (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Top down ... As modules replace stubs, more Top functionality is testable A B C stub X stub Y (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Top down ... complete Top A B C ... until the program is complete, and all functionality can be X Y tested (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Bottom Up . Starting at the leaves of the Driver “uses” hierarchy, we never need stubs X (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Bottom Up .. ... but we must Driver Driver construct drivers for each module (as in unit testing) ... X Y (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Bottom Up ... Driver A ... an intermediate module replaces a driver, and needs its own driver ... X Y (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Bottom Up .... Driver Driver A B X Y (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Bottom Up ..... Driver Driver Driver A B C ... so we may have several working subsystems ... X Y (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Bottom Up (complete) Top A B C ... that are eventually integrated into a X Y single system. (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Sandwich . Top (parts) Stub C Working from the extremes (top and bottom) toward center, we may use fewer Y drivers and stubs (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Sandwich .. Top (more) A C Sandwich integration is flexible and adaptable, X Y but complex to plan (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Thread ... Top A C A “thread” is a portion of several modules that together provide a X user-visible program feature. (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Thread ... Top A B C Integrating one thread, then another, etc., we maximize X Y visibility for the user (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Thread ... Top A B C As in sandwich integration testing, we can minimize stubs and drivers, but the integration X Y plan may be complex (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Critical Modules Strategy: Start with riskiest modules • Risk assessment is necessary first step • May include technical risks (is X feasible?), process risks (is schedule for X realistic?), other risks • May resemble thread or sandwich process in tactics for flexible build order – E.g., constructing parts of one module to test functionality in another • Key point is risk-oriented process – Integration testing as a risk-reduction activity, designed to deliver any bad news as early as possible (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13

Choosing a Strategy • Functional strategies require more planning – Structural strategies (bottom up, top down, sandwich) are simpler – But thread and critical modules testing provide better process visibility, especially in complex systems • Possible to combine – Top-down, bottom-up, or sandwich are reasonable for relatively small components and subsystems – Combinations of thread and critical modules integration testing are often preferred for larger subsystems (c) 2007 Mauro Pezzè & Michal Young Thursday, January 31, 13