Tests that (Almost) Write Themselves Hints for Golden Master - PowerPoint PPT Presentation

Tests that (Almost) Write Themselves Hints for Golden Master Testing in Python Stefan Baerisch, stefan@stbaer.com, 2020-07-20 1 image: freepik.com premium, by rawpixel

About Stefan Baerisch stbaer.com (German only, sorry) Software since 2005 Python since 2006 Project mangement / Test Management since 2010 2

Some Background on Golden Master Tests 3

Testing Ducks Good Duck? How would you describe an ‘acceptable’ rubber duck ? Do you know it when you see it? 4 image: freepik.com premium, by rawpixel

Testing Ducks How would you describe an ‘acceptable’ rubber duck ? Do you know it when you see it? 5 image: freepik.com premium, by rawpixel

The Idea behind Golden Master Testing How to implement test if data is unknown or too complex for an assert() ? 6

What is Golden Master Testing ? ? 1a Capture 1 Run Output ? ? ? ? 2 Change 4 Compare ? ? ? ? 3a Capture 3 Run Again Output ? ? 7

How does this help? Scenario Action Ensure that changes are Testing a legacy system visable Testing complex data Identify and review changes Testing complex data with Select or filter data used for some changes tests 8

Golden Master Testing - An Overview Golden Master Tests are also known as: - Characterization Tests Because changes to the - Approval Tests data are ‘approved’ - Snapshot Tests In the Javascript World, especially Jest 9

Implementing Golden Master Tests 10

The Process assert(check(data)) Data No Yes exists? Store Create Review Di ff erences data Di ff able Format Return Compare current Not OK? OK? True data to stored data Di ff erences? Replace No Di ff erences? Change Stored Store current Code Return Data results True Return False Automatic Test Process Manual Approval Process 11

Design Decisions Test Target State Log Output Test Size Few Large Many Small Storage Text Json/XML Custom Formats Diff CLI $… Lib Custom Approach What to Key Order Dates Random Values Ignore 12

Test Target Text / Logfiles Use as is, with Filtering Databases Select and Store as DDL JSON import json custom implementation or… import jsonpickle or… Objects import deepdiff Images pillow Dataframes json Keep it simple. Python makes it quite easy to get a text presentation of data 13

What to Ignore What to ignore? Dates / timestamps Key order for non-ordered dicts Random data (object ids / sequence ids) Non-relevant data (non significant floating point data) Needs some pre-processing, depending on use case 14

Test Size Large Tests Small Tests will need to break up / easy, fast setup select data will notice most changes may miss changes less frequent reviews / frequent reviews shorter reviews likely more work cleaning Less work cleaning up data up data 15

Storage Formats How do you store results from past runs? The format should be: Easy to store in VCSs Di ff able in Python and External Tools Supported by Editors / Views Should be structured to support working with Python If possible, use normalized JSON with linebreaks 16

An Example Implementation 17

Design Goals Keep things simple Compare Python Objects, jsonpickle no extentions difflib.unified_diff Simple comparision Simple storage store files with tests, explicit naming of tests Simple operations check () - store or compare data list () - show stored data and conflicts review (name) - show di ff erences approve (name) - mark the current version as ok 18

Our Sample Class 19

Overview Set storage location Save Success Compare Save Failure Compare Show state of tests / stored data Show di ff of “name2” test Accept the lastest version 20

Overview Results checker.list() checker.review( "name2" ) checker.approve( "name2" ) 21

Implementation - Check 22

Implementation - List Status 23

Implementation - Store 24

Implementation - Diff 25

Other Existing Libraries github.com/syrusakbary/ snapshottest Inspired by Javascript’s Jess. Nice Integration with unittest/nose/pytest github.com/approvals/ ApprovalTests.Python Python implementation of approval testing, 26

Summary Golden Master Tests work by capturing & comparing results of program Why executions. It helps with complex data, especially wen we want to monitor for changes. We store results and compare between run. On How di ff erences, we review & approve the results Python has many modules to help us. Python Some existing implementations exist if we don’t want to implement it ourselves 27

Thank you! Stefan Baerisch, stefan@stbaer.com, 2020-04-07 28