Rocket Fuelled Cucumbers Joseph Wilk Dealing with an Outbreak of - - PowerPoint PPT Presentation

▶
rocket fuelled cucumbers
SMART_READER_LITE
LIVE PREVIEW

Rocket Fuelled Cucumbers Joseph Wilk Dealing with an Outbreak of - - PowerPoint PPT Presentation

Dec 11, 2022 340 likes •1.34k views

Rocket Fuelled Cucumbers Joseph Wilk Dealing with an Outbreak of Cucumbers Joseph Wilk No Cucumbers where harmed in the making of this presentation. Just mildly shaken up Scenario : Happy Railsconf attendees Given you have a vague idea what

slide-1
SLIDE 1

Rocket Fuelled Cucumbers

Joseph Wilk

slide-2
SLIDE 2

Dealing with an Outbreak of Cucumbers

Joseph Wilk

slide-3
SLIDE 3

No Cucumbers where harmed in the making of this presentation.

Just mildly shaken up

slide-4
SLIDE 4

Scenario: Happy Railsconf attendees Given you have a vague idea what Cucumber is When Joseph completes his presentation Then you should understand the scaling choices And you should feel better prepared to face them

slide-5
SLIDE 5

Cucumber

  • BDD framework
  • Plaintext
  • Promotes

communication

Given /^I have trained monkeys$/ do @test_pilots << Monkeys.new end Feature: Filling Cucumbers with rocket fuel Scenario: Rocket fuel Given I have trained monkeys When I click the launch button Then the monkeys should not die

slide-6
SLIDE 6

Your code makes it slow Cucumber provides the venue

Shame on you

slide-7
SLIDE 7

What are Acceptance Tests?

  • Customer-facing
  • Cut through the whole stack
  • Examples of how the system is intended

to be used

slide-8
SLIDE 8
slide-9
SLIDE 9

You made me Cuke

slide-10
SLIDE 10
slide-11
SLIDE 11

1 minute ~ build

slide-12
SLIDE 12

1 minute ~ build Local builds

slide-13
SLIDE 13

1 minute ~ build Autotest Local builds

slide-14
SLIDE 14

1 minute ~ build Autotest Local builds Growl

slide-15
SLIDE 15
slide-16
SLIDE 16
slide-17
SLIDE 17

Don’t Panic

slide-18
SLIDE 18

Slow testing problems

Development is slowed down by the time to run focused Cukes Running all the Cukes takes too long!

slide-19
SLIDE 19

Development Test Build Types of Feedback

Speed Confidence

  • +

Speed Confidence +

slide-20
SLIDE 20

Happiness is a 10 min build time

slide-21
SLIDE 21

Slow builds are the enemy of continuous integration

slide-22
SLIDE 22

Development Feedback

Its too slow to run the cukes. Just push the code and run away

slide-23
SLIDE 23

Reuse - Spork

$ spork cucumber $ cucumber --drb

require 'rubygems' require 'spork' Spork.prefork do puts "I'm loading all the heavy stuff..." end Spork.each_run do puts "I'm loading the stuff just for this run..." # Cucumber hooks must go here end

git://github.com/timcharper/spork.git

slide-24
SLIDE 24

Slow services

  • Search
  • Solr
  • Sphinx
  • Databases
  • Mongo
  • Redis
  • Mysql
  • Message Queues
  • RabbitMQ

Before('@solr') do Solr.boot unless Solr.running? end Feature: Searching @solr Scenario: Indexed search Given ... When ... Then ...

slide-25
SLIDE 25

Run Just Enough Tests

# * Test files must be stored in test/ # * Test files names must start with test_ # * Test class names must start with Test # * Implementation files must be stored in lib/ # * Implementation files must match up with a # test file named # test_.*implementation.rb @analyzer = Rcov::CodeCoverageAnalyzer.new

Cucover Autotest

slide-26
SLIDE 26

http://vimeo.com/12500864

slide-27
SLIDE 27

Slicing Features

Tagging Filenames

$ cucumber --tags @media

Subsystems

$ cucumber visitor_* $ cucumber features/admin/*

Profiles

$ cucumber --profile admin

@media, @public Feature: Visitor views artist’s media @feed, @admin Feature: Logged in user views views their activity feed

slide-28
SLIDE 28

Just enough Database

slide-29
SLIDE 29

Just enough Database

city data venue data

SELECT * FROM `cities` WHERE (`cities`.`id` = 105838)

INSERT INTO `cities` (`name`) VALUES ('Test name') INSERT INTO `venues` ...

SELECT * FROM `venues` WHERE ...

Cache Cache

slide-30
SLIDE 30

Test Build Feedback

Commit code Have a 1 hour nap

slide-31
SLIDE 31

Options

  • Hardware
  • Intelligent test selection
  • Tighter focused tests
  • Divide and Conquer
slide-32
SLIDE 32

Testjour

Songkick.com 254 Features, 1257 scenarios, 10807 steps 1 build server ~ 4 hours 5 build servers ~ 2 hours

slide-33
SLIDE 33

120 sad minutes of my life...

slide-34
SLIDE 34

$ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help

slide-35
SLIDE 35

$ git clone git://github.com/brynary/testjour.git $ rake gem install $ testjour --help testjour help:

slide-36
SLIDE 36

Testjour

Master Work Queue Slave

Redis

Slave Slave Worker Worker Worker Worker Worker Worker

slide-37
SLIDE 37

Testjour

Master Work Queue Slave

Redis

Slave Slave

rsync - code rsync - code

Worker Worker Worker Worker Worker Worker

slide-38
SLIDE 38

Testjour

Master Work Queue Slave

Redis

Slave Slave

rsync - code rsync - code

Worker Worker Worker Worker Worker Worker

slide-39
SLIDE 39

Testjour

Master Work Queue Slave

Redis

Slave Slave

rsync - code rsync - code

Worker Worker Worker Worker Worker Worker

slide-40
SLIDE 40

Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure

slide-41
SLIDE 41

Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours

slide-42
SLIDE 42

Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes

slide-43
SLIDE 43

Looking to the Cloud

EC2, Rackspace Cloud, Windows Azure 1 build server ~ 4 hours 20 build servers ~ 11:40 minutes $2068.99 / $3000

slide-44
SLIDE 44

Ec2

AMI slim-sumo

Disc image

slide-45
SLIDE 45

Ec2

AMI

Launch

slim-sumo

Disc image

slide-46
SLIDE 46

Ec2

AMI

Launch

ec2 node ec2 node ec2 node ec2 node slim-sumo

Disc image

slide-47
SLIDE 47

Ec2

AMI

Launch

ec2 node ec2 node ec2 node ec2 node slim-sumo

Build

ENV['EC2_URL'] = ec2_url require 'slim-sumo' Sumo.new.running.map{|ec2_node| ec2_node[:hostname]}

Disc image

slide-48
SLIDE 48

http://vimeo.com/12467100

slide-49
SLIDE 49

require 'rubygems' require 'hydra' require 'hydra/tasks' Hydra::TestTask.new('hydra') do |t| t.add_files 'features/**/*.feature' t.verbose = false t.autosort = false end

slide-50
SLIDE 50
slide-51
SLIDE 51

“Buy lots of hardware”

slide-52
SLIDE 52

“Buy lots of hardware” “more hardware please”

slide-53
SLIDE 53

“Buy lots of hardware” “more hardware please” “Just a little bit more hardware”

slide-54
SLIDE 54

“Buy lots of hardware” “more hardware please” “Just a little bit more hardware”

slide-55
SLIDE 55

Just run the tests that matter...

slide-56
SLIDE 56

Failure probability matters

European Computing manufacturer

slide-57
SLIDE 57

Failure probability matters

European Computing manufacturer Automated test suite ~ 18 hours

slide-58
SLIDE 58

Failure probability matters

European Computing manufacturer Tests that were most ~ 2 hours likely to fail Automated test suite ~ 18 hours

slide-59
SLIDE 59

http://vimeo.com/12467122

slide-60
SLIDE 60
slide-61
SLIDE 61

Tests that never fail

slide-62
SLIDE 62

Tests that never fail

slide-63
SLIDE 63

Tests that never fail Tests which regularly fail

slide-64
SLIDE 64

Daily Build Tests that never fail Tests which regularly fail

slide-65
SLIDE 65

Daily Build Nightly Build Tests that never fail Tests which regularly fail

slide-66
SLIDE 66

Daily Build Nightly Build Tests that never fail Tests which regularly fail

$cucumber --tags ~@nightly $cucumber --tags @nightly

slide-67
SLIDE 67

Flickering Confidence

slide-68
SLIDE 68

Flickering Confidence

FAIL

slide-69
SLIDE 69

Flickering Confidence

FAIL FAIL

slide-70
SLIDE 70

Flickering Confidence

FAIL PASS FAIL

slide-71
SLIDE 71

Flickering Confidence

FAIL PASS FAIL Main Flicker

@flicker

slide-72
SLIDE 72

Divide and conquer

  • Rails - Engines
  • Service Orientated Architecture

1 hour 7 7 7 7 7 7 7 7 7

slide-73
SLIDE 73

Divide and Concuquer

slide-74
SLIDE 74

Divide and Concuquer

slide-75
SLIDE 75

Divide and Concuquer

slide-76
SLIDE 76

Divide and Concuquer

slide-77
SLIDE 77

Divide and Concuquer

slide-78
SLIDE 78

Divide and Concuquer

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

slide-79
SLIDE 79

Divide and conquer

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

slide-80
SLIDE 80

Divide and conquer

Related Artists Service Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

slide-81
SLIDE 81

Divide and conquer

related(artist)

Related Artists Service Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

slide-82
SLIDE 82

Divide and conquer

related(artist)

Related Artists Service

HTML

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

slide-83
SLIDE 83

Divide and conquer

related(artist)

Related Artists Service

HTML

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison”

slide-84
SLIDE 84

Divide and conquer

related(artist)

Related Artists Service

HTML

Rails

Scenario: Related artists Given an artist “SYGC” And “M.Bison” is related to “SYGC” When I visit the artist Then I should see within Similar Artists a link to “M.Bison” unit test unit test

slide-85
SLIDE 85

Don’t use Acceptance tests

http://jamesshore.com/Blog/Alternatives- to-Acceptance-Testing.html

Heresy!

slide-86
SLIDE 86

Pairwise

Scenario: Testing Cucumber against different environments Given I have a rails app <Rails version> And I’m using Ruby <Ruby version> And I am using the <Test Framework> And I am using the driver <Driver> Then Cucumber should install and play nicely Examples: | Rails version | Ruby version | test framework | Driver | | 3.0 | 1.8.7 | Rspec | Webrat | | 3.0 | 1.8.7 | Testunit | Capybara | | 3.0 | 1.8.7 | Rspec | Capybara | | 3.0 | 1.8.7 | Testunit | Webrat | | 3.0 | 1.9 | Rspec | Webrat | | 3.0 | 1.9 | Testunit | Webrat | | 2.3.1 | 1.9 | Rspec | Webrat | ....

slide-87
SLIDE 87

| Rails version | Ruby version | test framework | Driver | | 2.3.1 | 1.8.7 | Rspec | Webrat | | 2.3.1 | 1.8.7 | Rspec | Capybara | | 2.3.1 | 1.8.7 | Testunit | Webrat | | 2.3.1 | 1.8.7 | Testunit | Capybara | | 2.3.1 | 1.9 | Rspec | Capybara | | 2.3.1 | 1.9 | Rspec | Webrat | | 2.3.1 | 1.9 | Testunit | Capybara | | 2.3.1 | 1.9 | Testunit | Webrat | ....

“most faults are caused by interactions of at most two factors”

slide-88
SLIDE 88

Pairwise

http://www.screencast.com/t/NGI0NjVk

slide-89
SLIDE 89

We should add some funky Ajax features

Just when you thought things were going well

slide-90
SLIDE 90

JavaScript

In-memory Browser based

Selenium, Watir Celerity

slide-91
SLIDE 91

Need for speed

Webrat Celerity Watir Selenium 3.75 7.5 11.25 15

12.9 12 12 0.8

start-up time (seconds)

slide-92
SLIDE 92

Capybara

Like Webrat but bigger

envjs

Johnson

spidermonkey

gem install johnson --prerelease gem install envjs sudo gem install capybara

slide-93
SLIDE 93

http://vimeo.com/12467133

slide-94
SLIDE 94

Capybara-envjs

slide-95
SLIDE 95
slide-96
SLIDE 96

Find the right compromise for you

slide-97
SLIDE 97

Find the right compromise for you Slow test builds produce slow release cycles

slide-98
SLIDE 98

Find the right compromise for you Slow test builds produce slow release cycles Your test builds are a goldmine of data

slide-99
SLIDE 99

Thanks

Joseph Wilk http://blog.josephwilk.net @josephwilk

Its been emotional