- Multivariate A/B Testing with Drupal MontenaSoft MD Systems - PowerPoint PPT Presentation

- Multivariate – A/B Testing with Drupal MontenaSoft MD Systems Miro Dietiker Ivica Puljic Zeljko Puljic

Overview • Basic of Drupal A/B Testing: Concepts & History (Zeljko Puljic, MontenaSoft) • Case Study Acredis (Miro Dietiker, MD-Systems) • Hands-On: distribution, usage, enchancement LINK TO DISTRIBUTION DOWNLOAD (Ivica Puljic, MontenaSoft)

Purpose of A/B Testing • Every website has a target audience (segmentation) • But perhaps you want to expand to new markets • No Website is ever ready (optimized & finished) • Even if marketing department has all answers ready, there is no way around reality • so you should test & measure the impact of different settings to find out what really works But … you need to have clear idea about initiate & success criteria

Usual suspects... On diagram

Success, Conversion, Path clear def of initiate paths & success Success examples: (1) user has visited Page XYZ or (2) has bought Product XYZ Specific path example (for 2): Products list overview, product page, basket, checkout, credit card details, order confirmation

Basic Concepts of A/B Testing (1) BASKET NE Menu WS • Point1 • Point2 • Point3 AD1 Product Product P1 List AD2 Description P1 P2 PN AD2 BASKET Product List P1 P2 Product P1 PN Description Menu • Point1 • Point2 NE • Point3 WS AD1 AD2 AD2

Do it with Drupal • Several external tools to integrate (change something on your landing page, for instance) • http://drupal.org/project/multivariate (D6 taken over by MontenaSoft & MD-Systems) and today, for the first time ever ... • official announce of D7 Port

Multivariate Vocabulary & A/B (1) 1:N 1:N 1:N Test Set Variant Mutation A/B Test Set Variant1 - Layout 1 Variant1, Variant2, Variant1, .... - CSS 1 AD2 Variant2 - Layout 2 - CSS 2

Multivariate Vocabulary & A/B (2) 1:N 1:N 1:N Test Set Variant Mutation Multivariate Test Set 2 Set 1 Variant1 Variant1 - CSS 1 - Layout 1 [Set1] x [Set2] AD2 AD2 Variant2 Variant2 (Layout1, CSS1), (Layout1, CSS2), - CSS 2 - Layout 2 (Layout2, CSS1), (Layout2, CSS2),

Multivariate Measurements Assign & Track mutation combinations from different Sets to user Module responsability: specify activation conditions (cache) Measure conversion define success criteria: landing on a page, create user account... measure & present results Plugable Architecture Ctools Plugin

Testjng - Why ● Solves your real business problems ● Can make you rich! ;-)

Leads - Business ● Cost per Lead htup://en.wikipedia.org/wiki/Cost_per_Lead

Leads - Conversion ● Lead marketjng platgorms ● Convert a Visitor ● … to a qualifjed Lead ● Own leads ● Trade leads

Conversion - Improvements ● Small changes ● +10 .. 30% ● Difgerent approaches ● +100 … 1000%

Conversion - Sequences ● 3 Iteratjons of +26% ● +100% Conversion 1.26*1.26*1.26 = 2.00+ ● ● +100% Profjt ;-) ● With virtual goods...

Cases - Landing Page ● Visual difgerence

Cases - Registratjon ● Big signup form ● Small reduced form

Case 1 - Registratjon Page 1/2

Case 1 - Registratjon Page 2/2

Variatjons - A few Ideas ● Images, Characters ● Design, Color, Style, Efgects ● Content, Length / Details, Perspectjve ● Personalizatjon ● +++

Conversion - Aspects 1/2 ● Conversion = Trust ● Expectatjons ● Client needs ● Frame / Emotjons

Conversion - Aspects 2/2 ● Focus / Distractjon ● Service quality ● Value ● Right Price ● Your reliability

Conversion - Unexpected Form ● Ask less ● Reductjon, Simplifjcatjon ● Expected more Conversion ● Ask more ● Provided more Conversion

Unexpected Form - Reasons ● Trust ● Sensitjve informatjon ● Care about everything ● Feel safe

Testjng - It's all about ● Truth ● NO Philosophy ● NO debate ● Decisions ● Let clients decide what's best

Risks - Cautjon! ● Don’t break your site ● Multjple states... ● Know what you’re testjng ● False conclusions possible

Alright! - But... ● Start Testjng NOW!

Current state of Multivariate • What is currently done - Tests are now entity based - Mutations plugins for path mutation, path redirection, variable change, node load, google analytics, internal css, form class and inline css • Success plugins for path and form submit • Basic table and chart reports • Override tests for specific user

Additional features needed for D7 stable release • Bring back simple plugin support - Add theme_change mutation plugin • Should other two tables be entities so we have views support for them also? • Add Views support • More reports - Time based charts • Upgrade path from D6

Future • ctools export / import • Even more plugins out of the box • Provide documentation, howto, video, ... • Set A/B/X rates instead of equal distribution • Permanently switch to variant (disable testing) • Automatically switch to permanent variant on statistical (winner) condition • Improved continuous stats (add date-from, date-to, counters) • Introduce hooks (e.g. success, apply mutation, ...) / rules integration • Push AB test states into Drupal.settings JS so client side JS can build on it • And mooooore! ;) current list on http://drupal.org/node/1136868#comment-5716584

As always... Help needed

Multivariate D7 TODO • Finish D7 port effort • Add simpletests testing • Fix all important problems/improvements • CTools cache problems • ... and the rest of @todo's in code ;-)