important contributions
play

Important Contributions JD 100 Program Reduction 80 Execution - PowerPoint PPT Presentation

Dec 07, 2022 •177 likes •1.2k views

Introduction Regression Testing Empirical Evaluation Conclusion Practical Suggestions for Improving and Empirically Studying Greedy Test Suite Reduction and Prioritization Methods Gregory M. Kapfhammer Department of Computer Science

  1. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of a Test Suite Program Calculating Coverage Test Suite Use instrumentation probes Adequacy Criterion to capture and analyze a Instrumented Program test suite’s coverage of the Instrumentation program state and structure Instrumented Test Suite Regression Testing Test Coverage Coverage Monitoring Results The coverage results and Test Adequacy adequacy measurements can Requirements Calculation support both test suite Adequacy reduction and prioritization Measurements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 6 / 30

  2. Introduction Regression Testing Empirical Evaluation Conclusion Calculating the Coverage of a Test Suite Program Calculating Coverage Test Suite Use instrumentation probes Adequacy Criterion to capture and analyze a Instrumented Program test suite’s coverage of the Instrumentation program state and structure Instrumented Test Suite Regression Testing Test Coverage Coverage Monitoring Results The coverage results and Test Adequacy adequacy measurements can Requirements Calculation support both test suite Adequacy reduction and prioritization Measurements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 6 / 30

  3. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 1 R 3 R 4 R 6 R 7 R 5 R 2 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 7 / 30

  4. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 1 R 3 R 4 R 6 R 7 R 5 R 2 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 7 / 30

  5. Introduction Regression Testing Empirical Evaluation Conclusion Finding the Overlap in Coverage R R 2 R 1 R 3 R 4 R 7 R 5 R 6 T 8 T 2 T 4 T 12 T 1 T 3 T 11 T 7 T 9 T 10 T 5 T 6 R j → T i means that requirement R j is covered by test T i Test suite reduction discards the test cases that redundantly cover the test requirements T = � T 2 , T 3 , T 6 , T 9 � covers all of the test requirements Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 7 / 30

  6. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  7. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  8. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  9. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Approaches to Regression Testing Original Test Suite Harrold, Gupta, Soffa (HGS) T 1 T 2 T 4 T 3 Delayed Greedy (DGR) Traditional Greedy (GRD) Reduction Technique 2-Optimal Greedy (2OPT) First Output First Residual Second Output Hypothesis : Using the exe- T 1 T 4 T 2 T 3 T 2 T 3 cution time of a test case can Prioritized Test Suite improve the reduced and pri- T 1 T 4 T 2 T 3 oritized test suites Compare (i) greedy choices (cost, coverage, and ratio) and (ii) algorithms Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 8 / 30

  10. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [ 0 , 1 ] Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 9 / 30

  11. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Prioritizers Cover R ( T 1 ) Cover � n − 1 i = 1 R ( T i ) Cover R ( T ) C ( T , t ) T n Done Covered Test Reqs . . . � t ( n ) Area C ( T , t ) 0 Testing Time ( t ) T 1 Done T n − 1 Done Prioritize to increase the CE of a test suite CE = Actual Ideal ∈ [ 0 , 1 ] Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 9 / 30

  12. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers Reduction Factor for Size (RFFS): How small is the reduced test suite? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 10 / 30

  13. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers Reduction Factor for Time (RFFT): How fast is the reduced test suite? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 10 / 30

  14. Introduction Regression Testing Empirical Evaluation Conclusion Evaluating Test Suite Reducers Common Rate (CR): How similar are differently reduced test suites? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 10 / 30

  15. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  16. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  17. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  18. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  19. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  20. Introduction Regression Testing Empirical Evaluation Conclusion Greedy Choices Impact Effectiveness R 1 R 2 R 3 R 4 R 5 Execution Time T 1 4 � � � � T 2 1 � � T 3 1 � T 4 1 � � Greedy-by T r time ( T r ) T p CE coverage � T 1 , T 4 � 5 � T 1 , T 4 , T 2 , T 3 � 0.400 time � T 2 , T 3 , T 4 � 3 � T 2 , T 3 , T 4 , T 1 � 0.714 ratio � T 2 , T 4 , T 3 � 3 � T 2 , T 4 , T 3 , T 1 � 0.743 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 11 / 30

  21. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Tree Models: Use recursive partitioning to create hierarchical view of data Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 12 / 30

  22. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Explanatory Variable: Configuration of the testing methods (e.g., GCM) Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 12 / 30

  23. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Tree Models metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Response Variable: One of the evaluation metrics (e.g., CE or RFFT) Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 12 / 30

  24. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  25. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  26. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 metric: cost Mean alg: 2OPT, GRD 0.4889 Mean Mean 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  27. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  28. Introduction Regression Testing Empirical Evaluation Conclusion Analysis Method: Random Forests metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 metric: cost metric: cost Mean Mean alg: 2OPT, GRD alg: 2OPT, GRD 0.4889 0.4889 Mean Mean Mean Mean 0.2101 0.4946 0.2101 0.4946 Many Trees: Randomly construct a large collection of trees in order to avoid bias and identify the most important explanatory variables Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 13 / 30

  29. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  30. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  31. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  32. Introduction Regression Testing Empirical Evaluation Conclusion Case Study Applications Name | T | |R ( T ) | CCN NCSS DS 110 40 1.35 1243.00 GB 51 88 2.60 1455.00 JD 54 783 1.64 2716.00 LF 13 6 1.40 215.00 RM 13 19 2.13 569.00 SK 27 117 2.00 628.00 TM 27 46 2.21 748.00 RP 76 221 2.65 6822.00 Do the greedy reducers and prioritizers efficiently identify test suites that improve effectiveness? What are the fundamental trade-offs ? Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 14 / 30

  33. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation T 4 T 2 T 1 T 3 R 1 R 2 R 3 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal : generate coverage and timing information that will fool the greedy technique into creating T p = � T n , . . . , T 1 � even though CE ( T p ) < CE ( T ) for T = � T 1 , . . . , T n � Inspiration : Vazirani’s construction of a tight example for the greedy minimal set cover algorithm Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 15 / 30

  34. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation T 4 T 2 T 1 T 3 R 1 R 2 R 3 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal : generate coverage and timing information that will fool the greedy technique into creating T p = � T n , . . . , T 1 � even though CE ( T p ) < CE ( T ) for T = � T 1 , . . . , T n � Inspiration : Vazirani’s construction of a tight example for the greedy minimal set cover algorithm Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 15 / 30

  35. Introduction Regression Testing Empirical Evaluation Conclusion “Greedy Fooling” Coverage Generation T 4 T 2 T 1 T 3 R 1 R 2 R 3 Generation Procedure The greedy test prioritizer iteratively selects test cases according to the (coverage / cost) ratio Goal : generate coverage and timing information that will fool the greedy technique into creating T p = � T n , . . . , T 1 � even though CE ( T p ) < CE ( T ) for T = � T 1 , . . . , T n � Inspiration : Vazirani’s construction of a tight example for the greedy minimal set cover algorithm Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 15 / 30

  36. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test Suites T 4 T 2 T 1 T 3 R 1 R 2 R 3 Approach : use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost ( T n ) Construction : set cost ( T 1 ) = cost ( T 2 ) = cost ( T 3 ) = 1 and then determine the bounds for cost ( T 4 ) ∈ [ C min , C max ] Example : cost ( T 4 ) ∈ [ 2 . 138803 , 2 . 472136 ] so that CE min ( T p ) = . 5838004 CE min ( T ) = . 6108033 CE max ( T p ) = . 5482172 CE max ( T ) = . 6345125 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 16 / 30

  37. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test Suites T 4 T 2 T 1 T 3 R 1 R 2 R 3 Approach : use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost ( T n ) Construction : set cost ( T 1 ) = cost ( T 2 ) = cost ( T 3 ) = 1 and then determine the bounds for cost ( T 4 ) ∈ [ C min , C max ] Example : cost ( T 4 ) ∈ [ 2 . 138803 , 2 . 472136 ] so that CE min ( T p ) = . 5838004 CE min ( T ) = . 6108033 CE max ( T p ) = . 5482172 CE max ( T ) = . 6345125 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 16 / 30

  38. Introduction Regression Testing Empirical Evaluation Conclusion Constructing “Greedy Fooling” Test Suites T 4 T 2 T 1 T 3 R 1 R 2 R 3 Approach : use one dimensional optimization (e.g., golden section search and successive parabolic interpolation) to pick a value for cost ( T n ) Construction : set cost ( T 1 ) = cost ( T 2 ) = cost ( T 3 ) = 1 and then determine the bounds for cost ( T 4 ) ∈ [ C min , C max ] Example : cost ( T 4 ) ∈ [ 2 . 138803 , 2 . 472136 ] so that CE min ( T p ) = . 5838004 CE min ( T ) = . 6108033 CE max ( T p ) = . 5482172 CE max ( T ) = . 6345125 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 16 / 30

  39. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites Objective: Simulate test suite execution on a centralized server Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 17 / 30

  40. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites Objective: Simulate test suite execution on a centralized server Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 17 / 30

  41. Introduction Regression Testing Empirical Evaluation Conclusion Constructing Build/Test Machine Suites Construction: Combine all of the test suites and coverage reports Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 17 / 30

  42. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  43. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  44. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  45. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  46. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Importance: Random prioritization serves as a valuable experimental control and often produces orderings better than the initial suite Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  47. Introduction Regression Testing Empirical Evaluation Conclusion Random Test Suite Prioritization Random Number Input Output { 1 , 2 , 3 , 4 } � t 1 , t 2 , t 3 , | t 4 � � t 1 , t 4 , t 3 , t 2 � 2 { 1 , 2 , 3 } � t 1 , t 4 , | t 3 , t 2 � � t 1 , t 4 , t 3 , t 2 � 3 { 1 , 2 } � t 1 , | t 4 , t 3 , t 2 � � t 4 , t 1 , t 3 , t 2 � 1 Strategy: Use the modern and efficient implementation of the Fisher-Yates shuffle to produce the reordered test suite T p = � t 4 , t 1 , t 3 , t 2 � Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 18 / 30

  48. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends Reduction Factor for Time (RFFT) metric: cost | alg: 2OPT,GRD 0.4889 0.2101 0.4946 The myopic focus on cost leads to low RFFT values for 2OPT and GRD Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 19 / 30

  49. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFT Trends Reduction Factor for Time (RFFT) metric: cost | alg: 2OPT,GRD 0.4889 0.2101 0.4946 The myopic focus on cost leads to low RFFT values for 2OPT and GRD Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 19 / 30

  50. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends Reduction Factor for Size (RFFS) alg: 2OPT,GRD | metric: cost 0.6136 metric: coverage 0.1130 0.5967 0.4959 DGR and HGS are the best at creating test suites that improve RFFS Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 20 / 30

  51. Introduction Regression Testing Empirical Evaluation Conclusion Overview of RFFS Trends Reduction Factor for Size (RFFS) alg: 2OPT,GRD | metric: cost 0.6136 metric: coverage 0.1130 0.5967 0.4959 DGR and HGS are the best at creating test suites that improve RFFS Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 20 / 30

  52. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends Coverage Effectiveness (CE) alg: HGS | metric: coverage 0.7520 alg: DGR 0.8231 0.8344 0.9388 Using ratio and cost improves the CE of the prioritized test suite Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 21 / 30

  53. Introduction Regression Testing Empirical Evaluation Conclusion Overview of CE Trends Coverage Effectiveness (CE) alg: HGS | metric: coverage 0.7520 alg: DGR 0.8231 0.8344 0.9388 Using ratio and cost improves the CE of the prioritized test suite Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 21 / 30

  54. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time - SK cost coverage ratio SK 0.4 Reduction Factor for Time (RFFT) 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique For 2OPT and GRD, ratio and coverage create the best test suites Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 22 / 30

  55. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Time - SK cost coverage ratio SK 0.4 Reduction Factor for Time (RFFT) 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique For 2OPT and GRD, ratio and coverage create the best test suites Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 22 / 30

  56. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size - SK cost coverage ratio SK 0.6 Reduction Factor for Size (RFFS) 0.5 0.4 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique It is often easy to construct test suites with high RFFS values Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 23 / 30

  57. Introduction Regression Testing Empirical Evaluation Conclusion Reduction Factor for Size - SK cost coverage ratio SK 0.6 Reduction Factor for Size (RFFS) 0.5 0.4 0.3 0.2 0.1 0.0 2OPT DGR GRD HGS Reduction Technique It is often easy to construct test suites with high RFFS values Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 23 / 30

  58. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  59. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique DGR and HGS exhibit lackluster performance when reordering Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  60. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique Greedily reordered test suites are better than randomly prioritized ones Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  61. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique Greedy fooling test suites cause GRD and DGR to make low CE suites Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  62. Introduction Regression Testing Empirical Evaluation Conclusion Coverage Effectiveness Results - RP cost coverage ratio SK 0.9 Coverage Effectiveness (CE) 0.8 0.7 0.6 0.5 0.4 2OPT DGR GRD HGS Prioritization Technique 2OPT uses lookahead and can construct high CE test prioritizations Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 24 / 30

  63. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  64. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  65. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  66. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  67. Introduction Regression Testing Empirical Evaluation Conclusion Common Rate Scores Application CommonRate (Υ) Reminder 0.700 ReduceAndPrioritize 0.361 Sudoku 0.571 TransactionManager 0.450 DataStructures 0.171 GradeBook 0.747 JDepend 0.606 LoopFinder 0.500 Average 0.513 Value of the common rate is relatively stable across methods Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 25 / 30

  68. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Permutation Importance (PI) CE RFFS RFFT 30 20 10 0 alg metric alg metric alg metric Parameter Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 26 / 30

  69. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Permutation Importance (PI) CE RFFS RFFT 30 20 10 0 alg metric alg metric alg metric Parameter Algorithm choice is most important for improving the CE of ordering Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 26 / 30

  70. Introduction Regression Testing Empirical Evaluation Conclusion Parameter Importance Values Permutation Importance (PI) CE RFFS RFFT 30 20 10 0 alg metric alg metric alg metric Parameter Greedy choice metric has the greatest impact on the test suite reducers Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 26 / 30

  71. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements JD 100 80 Execution Time (ms) 60 40 20 0 2OPT DGR GRD HGS Prioritization Technique For the chosen case study applications, the techniques are fast Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 27 / 30

  72. Introduction Regression Testing Empirical Evaluation Conclusion Efficiency Measurements JD 100 80 Execution Time (ms) 60 40 20 0 2OPT DGR GRD HGS Prioritization Technique For the chosen case study applications, the techniques are fast Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 27 / 30

  73. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  74. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  75. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  76. Introduction Regression Testing Empirical Evaluation Conclusion Alternative Evaluation Metrics Like APFD Mutation Faults Real Faults Use mutation and real faults to support the calculation of fault detection effectiveness ( FDE ) and average percentage of faults detected ( APFD ). Consider search-based testing methods. Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 28 / 30

  77. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - R educe A nd pr I ortize S uit E s http://raise.googlecode.com/ provides tools, data sets, and resources Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 29 / 30

  78. Introduction Regression Testing Empirical Evaluation Conclusion RAISE - R educe A nd pr I ortize S uit E s Interactive visualization methods enable testers to find best ordering Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 29 / 30

  79. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

  80. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

  81. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

  82. Introduction Regression Testing Empirical Evaluation Conclusion Concluding Remarks JD 100 Program Reduction 80 Execution Time (ms) Analysis 60 Technique 40 20 Test Suite Prioritization 0 2OPT DGR GRD HGS Prioritization Technique Regression Testing Techniques Detailed Empirical Results Implementation and empirical evaluation of methods for test suite reduction and prioritization Freely available data sets and free/open source tools http://www.cs.allegheny.edu/~gkapfham/research/kanonizo/ Practical Suggestions for Improving and , Empirically Studying Greedy Test Suite , Reduction and Prioritization Methods 30 / 30

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Why This Matters Bayesian networks have been one of the most important contributions to the

Why This Matters Bayesian networks have been one of the most important contributions to the

Why This Matters Bayesian networks have been one of the most important contributions to the field of CS 331: Artificial Intelligence AI in the last 10-20 years Bayesian Networks Provide a way to represent knowledge in an uncertain

114 views • 7 slides
Accounting for Contributions, Pledges, Grants and Donations to Non-Profits Navigating the Gray

Accounting for Contributions, Pledges, Grants and Donations to Non-Profits Navigating the Gray

Accounting for Contributions, Pledges, Grants and Donations to Non-Profits Navigating the Gray Areas Surrounding FAS 116 and FAS 117 TUESDAY, MAY 13, 2014, 1:00-2:50 pm Eastern IMPORTANT INFORMATION This program is approved for 2 CPE credit

1.38k views • 88 slides
ASU 2018-08 Clarifying the Scope and the Accounting Guidance for Contributions Received and

ASU 2018-08 Clarifying the Scope and the Accounting Guidance for Contributions Received and

ASU 2018-08 Clarifying the Scope and the Accounting Guidance for Contributions Received and Contributions Made Kimberly Griffith, CPA Audit Shareholder Danielle Miceli, CPA Audit Manager Objectives Distinguish between contributions

442 views • 40 slides
Module 4: 2016 Reinsurance Contributions Updating Contributions Filings October 18, 2016

Module 4: 2016 Reinsurance Contributions Updating Contributions Filings October 18, 2016

Module 4: 2016 Reinsurance Contributions Updating Contributions Filings October 18, 2016 November 8, 2016 Payment Policy &amp; Financial Management Group, Division of Reinsurance Operations Training Series HTTPS://WWW.REGTAP.INFO/ Session

991 views • 64 slides
Middle author dilemma: how to recognize critical contributions of multidisciplinary teams Melissa

Middle author dilemma: how to recognize critical contributions of multidisciplinary teams Melissa

Middle author dilemma: how to recognize critical contributions of multidisciplinary teams Melissa Gymrek University of California San Diego COASP 2016 Outline Why is authorship order so important? The sandwich model of authorship It

325 views • 15 slides
Contributions to the impedance budget impedance of pumping holes and interconnects Bernard

Contributions to the impedance budget impedance of pumping holes and interconnects Bernard

Contributions to the impedance budget impedance of pumping holes and interconnects Bernard Riemann FCC-hh impedance and beamscreen workshop 2017-03-30 1 / 17 Contributions to impedance budget A list of (passive) contributions beam pipe

335 views • 17 slides
ICAR-NICRA ICAR-NICRA Contributions and Impacts Contributions and Impacts CRIDA Dr.

ICAR-NICRA ICAR-NICRA Contributions and Impacts Contributions and Impacts CRIDA Dr.

ICAR-NICRA ICAR-NICRA Contributions and Impacts Contributions and Impacts CRIDA Dr. Ch.Srinivasa Rao Director ICAR-Central Research Institute for Dryland Agriculture Hyderabad Climate Change Impacts on Agriculture Sector Thus, currently,

231 views • 22 slides
CTA Design Study CTA Design Study - Swiss Hardware Contributions - Swiss Hardware Contributions

CTA Design Study CTA Design Study - Swiss Hardware Contributions - Swiss Hardware Contributions

CTA Design Study CTA Design Study - Swiss Hardware Contributions - Swiss Hardware Contributions Isabel Braun Insttut for Partcle Physics, ETH Zrich CHIPP Plenary Meeting 2009, Appenberg Participating Institutes Participating Institutes

442 views • 16 slides
CSE 510: Advanced Topics in HCI Contributions in HCI James Fogarty Daniel Epstein

CSE 510: Advanced Topics in HCI Contributions in HCI James Fogarty Daniel Epstein

CSE 510: Advanced Topics in HCI Contributions in HCI James Fogarty Daniel Epstein Tuesday/Thursday 10:30 to 12:00 CSE 403 Today Contributions in HCI Project Discussions Abstract (from original version) Research in human-computer

707 views • 41 slides
Planora Oy www.planora.fi Solutions and Contributions to the implementation of Helen Energy

Planora Oy www.planora.fi Solutions and Contributions to the implementation of Helen Energy

Planora Oy www.planora.fi Solutions and Contributions to the implementation of Helen Energy Model in China Beijing 15- 23.5.2107 Katja Granlund Planoras main sectors of activities Consulting Technical design Contributions of Planora to

452 views • 22 slides
DIANA Contributions Update Brian Bockelman Including work from Jim Pivarski, Oksana Shadura, and

DIANA Contributions Update Brian Bockelman Including work from Jim Pivarski, Oksana Shadura, and

DIANA Contributions Update Brian Bockelman Including work from Jim Pivarski, Oksana Shadura, and Zhe Zhang DIANA Contributions (Since July) Since the F2F meeting, DIANA contributions have focused on the following areas: Parallelism :

342 views • 16 slides
Why is this important? Why is this important? Anyone know what song this is? Yes thats right,

Why is this important? Why is this important? Anyone know what song this is? Yes thats right,

Why is this important? Why is this important? Anyone know what song this is? Yes thats right, Crowded House Weather With You UK release in 1992. I was 5 years old. Why is this important? The estimated average annual economic

332 views • 16 slides
Two-loop fermionic contributions to polarized Moller scattering asymmetries Yong Du In

Two-loop fermionic contributions to polarized Moller scattering asymmetries Yong Du In

Two-loop fermionic contributions to polarized Moller scattering asymmetries Yong Du In collaboration with Ayres Freitas, Hiren Patel, Michael J. Ramsey-Musolf LoopFest XVIII, August 13, 2019 FermiLab, IL Two-loop fermionic contributions to

414 views • 30 slides
o All functionalities appear at least once o 5 functionalities stick out with 4 mentions Mobility

o All functionalities appear at least once o 5 functionalities stick out with 4 mentions Mobility

Workshop Cross-border Cooperation for CCAM Testing o 8 contributions received, still limited data but some trends appear o (Understandably) not all contributions cover all aspects (i.e. functionalities and all enablers) o Focus on elements that

416 views • 5 slides
Update on Nationally Determined Contributions (Paris Agreement on Climate Change) Energy Sector

Update on Nationally Determined Contributions (Paris Agreement on Climate Change) Energy Sector

Update on Nationally Determined Contributions (Paris Agreement on Climate Change) Energy Sector 7-26-2018 1 Conditional Contributions Completed Independent review of the Amaila Falls Project with the assistance of the Kingdom of

781 views • 25 slides
Q S QoS and DLC in d DLC i IaaS INDIGO- DataCloud Presenter : Patrick Fuhrmann Contributions

Q S QoS and DLC in d DLC i IaaS INDIGO- DataCloud Presenter : Patrick Fuhrmann Contributions

Q S QoS and DLC in d DLC i IaaS INDIGO- DataCloud Presenter : Patrick Fuhrmann Contributions by: Giacinto Donvito, INFN Marcus Hardt, KIT Paul Millar, DESY Alvaro Garcia, CSIC Alvaro Garcia, CSIC With ki d With kind contributions by t

760 views • 38 slides
Final Presentation Outline NORTHEAST HOSPITAL EXPANSION 123 Medical Lane, USA Joshua Miller

Final Presentation Outline NORTHEAST HOSPITAL EXPANSION 123 Medical Lane, USA Joshua Miller

Courtesy of Smith Group JJR Final Presentation Outline NORTHEAST HOSPITAL EXPANSION 123 Medical Lane, USA Joshua Miller Construction Management Option Advisor: Craig Dubler March 30, 2015 Final Presentation Outline March 30, 2014 I.

194 views • 6 slides
The Pennington School Daniel Brady Principal Jessica Maracallo Acting Assistant Principal

The Pennington School Daniel Brady Principal Jessica Maracallo Acting Assistant Principal

The Pennington School Daniel Brady Principal Jessica Maracallo Acting Assistant Principal School Composition for the 2019-2020 School Year Pre-Kindergarten (2) Fifth Grade (2) Kindergarten (2) Sixth Grade (2) First

447 views • 9 slides
WELCOME Lake Lanier Association Annual Membership Meeting June 5, 2013 1 Agenda Welcome

WELCOME Lake Lanier Association Annual Membership Meeting June 5, 2013 1 Agenda Welcome

WELCOME Lake Lanier Association Annual Membership Meeting June 5, 2013 1 Agenda Welcome Jackie Joseph Retirement Business Meeting Year in Review Corps of Engineers Report Q&amp;A Adjourn 2 Jackie Joseph Retirement LLA Member for 29

689 views • 17 slides
Ci City ty Pl Plan an Co Commissio mission Allan an Fung Michael hael Smith th Mayor

Ci City ty Pl Plan an Co Commissio mission Allan an Fung Michael hael Smith th Mayor

Ci City ty Pl Plan an Co Commissio mission Allan an Fung Michael hael Smith th Mayor Chair Jason on Pezzul ullo lo Freder erick ick Vincent ncent Planning Director Vice-Chair hair Robert Strom Robert Coupe Joseph Morales

671 views • 20 slides
THE INTERSECTION OF GLOBAL FRAGILITY AND CLIMATE RISKS Ashley Moran, University of Texas at Austin

THE INTERSECTION OF GLOBAL FRAGILITY AND CLIMATE RISKS Ashley Moran, University of Texas at Austin

THE INTERSECTION OF GLOBAL FRAGILITY AND CLIMATE RISKS Ashley Moran, University of Texas at Austin Research team: Ashley Moran, Joshua W. Busby, Clionadh Raleigh, T odd G. Smith, Roudabeh Kishi, Nisha Krishnan, Charles Wight USAID Adaptation

654 views • 28 slides
Overview of the TN Debit Tool Compensatory Mitigation webpage Questions Jimmy Smith,

Overview of the TN Debit Tool Compensatory Mitigation webpage Questions Jimmy Smith,

Highlights from the new rules Process to develop the Stream Mitigation Guidelines Highlights from the 2019 Stream Mitigation Guidelines Overview of the TN Debit Tool Compensatory Mitigation webpage Questions Jimmy Smith,

715 views • 54 slides
JBS Primo Wacol Enterprise Agreement 2019 EXPLANATION OF TERMS AND EFFECT 21 OCTOBER 2019 -

JBS Primo Wacol Enterprise Agreement 2019 EXPLANATION OF TERMS AND EFFECT 21 OCTOBER 2019 -

JBS Primo Wacol Enterprise Agreement 2019 EXPLANATION OF TERMS AND EFFECT 21 OCTOBER 2019 - UPDATED Part rt 2 - Appli licatio ion and Operatio ion Agreement applies to Primo Foods Pty Ltd and all employees at Primo facilities in Wacol

376 views • 22 slides
West Virginia Retiree Health Benefit Trust Fund and PEIA Finance Board Meeting Thursday, January

West Virginia Retiree Health Benefit Trust Fund and PEIA Finance Board Meeting Thursday, January

West Virginia Retiree Health Benefit Trust Fund and PEIA Finance Board Meeting Thursday, January 21, 2016, 1:00 p.m. Canaan Valley Room 1041, DEP Building, 601 57th Street, SE Charleston, WV 25304 Agenda Roll Call Call to Order Approval

432 views • 28 slides

More recommend