Reinforcement Learning To Write Java Unit Tests Mathew Lodge - - PowerPoint PPT Presentation

▶

Dec 05, 2023 49 likes •341 views

Reinforcement Learning To Write Java Unit Tests Mathew Lodge Diffblue: AI For Code 1 Talk in a nutshell: In the same way AI can search for the best Go move, AI can search for the best unit test Continuous Integration In Theory Write Code

SLIDE 1

SLIDE 2

Reinforcement Learning To Write Java Unit Tests

Mathew Lodge

SLIDE 3

Diffblue: AI For Code

SLIDE 4

Talk in a nutshell: In the same way AI can search for the best Go move, AI can search for the best unit test

SLIDE 5

Continuous Integration In Theory

Write Code Commit Unit Tests Integrate Full Tests Deploy

SLIDE 6

Continuous Integration in Practice

Minutes Hours Hours to Days Time to find & fix errors Write Code Commit Tests? Integrate Full Tests

Uncaught Errors Result in Broken Pipelines and Low Velocity

SLIDE 7

What is AI?

SLIDE 8

Code Transforms Input to Output

Q: 26,464*36,361/4? A: 240,564,376

SLIDE 9

Limitations Of Human Coding

A: ???? Q: What’s this?

SLIDE 10

AI: Computer Iterates on Statistical Model That Relates Input To Output

AI Model

A: 91% Cat 23% Tabby 86% Dog 32% Spaniel 11% Pebble Q: What’s this?

SLIDE 11

AI: Not Artificial, Not Intelligent

Not Artificial:

Solves real problems programmers cannot reach

Not Intelligent: Just

maths

Great when you

can’t brute-force the problem

SLIDE 12

Reinforcement Learning AI in AlphaGo

SLIDE 13

Go Is A Hard Game To Automate

Far more possible moves

than Chess: ~10170

~1092 atoms in the known

universe

Long-term strategy makes a

big difference in Go

Hard to tell if a move is good

until much later

SLIDE 14

Reinforcement Learning In General

Reward State update Action Observation Environment Evaluate Update Model

SLIDE 15

AlphaGo

Reward State update Algorithm Go Moves Play Game AlphaGo Evaluation Updated Neural Networks

SLIDE 16

AlphaGo Is Probabilistic Search Of The Go Gameplay Space

Monte Carlo to generate

potential moves

“Policy” neural network

predicts best next move

“Value” neural network

predicts who will win

SLIDE 17

Monte Carlo With Prediction

SLIDE 18

What Does This Have To Do With Java Testing Or DevOps?

SLIDE 19

Testing As Search Of Program Space

Set of All Possible Test Programs Programs that are valid and run High coverage tests Tests that developers find easy to read

SLIDE 20

Test Writing Is Hard To Automate

Exponential search space
Complex interdependencies

between program and data

Hard to tell if a test is “good”
Practical difficulties: IO,

frameworks, dynamic typing, semantic understanding, idiomatic code

SLIDE 21

Diffblue Reinforcement Learning

JVM Test Writer Evaluator Reward Existing Java code Test mutation Test Candidate Coverage + Results

SLIDE 22

Unsupervised Learning Very Fast Can Run On A Dev Laptop

SLIDE 23

Demo

SLIDE 24

Diffblue AI Writes Unit Test Suites

Run quickly
Run early
Find unit-level errors
Improve coverage
Are easy to understand

SLIDE 25

Typical Software Lifecycle

Running code Engineer writes code change Pull Request Run tests Is the change correct? Engineer updates code PR approved No Yes

SLIDE 26

Lifecycle with Diffblue Cover

Running code Engineer writes code change Pull Request Run tests + Diffblue suite Is the change correct? Engineer updates code PR approved Diffblue AI updates Unit Test Suite No Yes Regression Unit Test Suite AI writes tests from current code

SLIDE 27

Diffblue At Goldman Sachs

2x better test coverage (70%)

10x faster than manual coding

SLIDE 28

Summary

Lack of tests to run early in pipeline is a key DevOps blocker
Diffblue Cover AI writes unit tests for your current Java code
Cover is 10-100x faster than manual test writing

SLIDE 29