A / B testing for marketing AN ALYZIN G MAR K E TIN G C AMPAIG N - - PowerPoint PPT Presentation

▶
a b testing for marketing
SMART_READER_LITE
LIVE PREVIEW

A / B testing for marketing AN ALYZIN G MAR K E TIN G C AMPAIG N - - PowerPoint PPT Presentation

Feb 24, 2024 266 likes •624 views

A / B testing for marketing AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS Jill Rosok Data Scientist What is A / B testing ? Prior to r u nning the test determine : What is the desired o u tcome of the test ? What is o u r h y pothesis

slide-1
SLIDE 1

A/B testing for marketing

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

Jill Rosok

Data Scientist

slide-2
SLIDE 2

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

What is A/B testing?

Prior to running the test determine: What is the desired outcome of the test? What is our hypothesis? What is the metric we are trying to impact (i.e., page views, conversions)? Will we get enough trac to our site to reach statistical signicance and make a decision in a timely manner?

slide-3
SLIDE 3

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Testing allows us to understand marketing impact

slide-4
SLIDE 4

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

How long does a test need to run?

slide-5
SLIDE 5

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Personalized email test

slide-6
SLIDE 6

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Test allocation

email = marketing[marketing['marketing_channel'] == 'Email' allocation = email.groupby(['variant'])\ ['user_id'].nunique() allocation.plot(kind='bar') plt.title('Personalization test allocation') plt.xticks(rotation = 0) plt.ylabel('# participants') plt.show()

slide-7
SLIDE 7

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Allocation plot

slide-8
SLIDE 8

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Setting up our data to evaluate the test

# Group by user_id and variant subscribers = email.groupby(['user_id', 'variant'])['converted'].max() subscribers = pd.DataFrame(subscribers.unstack(level=1))

slide-9
SLIDE 9

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Setting up our data to evaluate the test

# Drop missing values from the control column control = subscribers['control'].dropna() # Drop missing values from the personalization column personalization = subscribers['personalization'].dropna()

slide-10
SLIDE 10

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Conversion rates

print("Control conversion rate:", np.mean(control)) print("Personalization conversion rate:", np.mean(personalization)) Control conversion rate: 0.2814814814814815 Personalization conversion rate: 0.3908450704225352

slide-11
SLIDE 11

Let's get testing!

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

slide-12
SLIDE 12

Calculating lift & significance testing

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

Jill Rosok

Data Scientist

slide-13
SLIDE 13

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Treatment performance compared to the control

Calculating li:

Control conversion rate Treatment conversion rate - Control conversion rate

slide-14
SLIDE 14

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Calculating lift

# Calcuate the mean of a and b a_mean = np.mean(control) b_mean = np.mean(personalization) # Calculate the lift using a_mean and b_mean lift = (b_mean-a_mean)/a_mean print("lift:", str(round(lift*100, 2)) + '%') lift: 194.23%

slide-15
SLIDE 15

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

T-distribution

Identication of Timed Behavior Models for Diagnosis in Production Systems. Scientic Figure on ResearchGate.

1

slide-16
SLIDE 16

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

P-values

T-statistic of 1.96 is typically statistically signicant at the 95% level Depending on the context of the test, you may be comfortable with a lower or higher level of statistical signicance.

slide-17
SLIDE 17

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

T-test in Python

from scipy.stats import ttest_ind t = ttest_ind(control, personalized) print(t) Ttest_indResult(statistic=-2.7343299447505074, pvalue=0.006451487844694175)

slide-18
SLIDE 18

Let's practice!

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

slide-19
SLIDE 19

A/B testing & segmentation

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

Jill Rosok

Data Scientist

slide-20
SLIDE 20

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Don't forget about segmentation!

slide-21
SLIDE 21

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Personalization test segmented by language

for language in np.unique(marketing['language_displayed'].v print(language)

slide-22
SLIDE 22

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Isolate the relevant data

for language in np.unique(marketing['language_displayed'].values): print(language) language_data = marketing[(marketing['marketing_channel'] == 'Email') & (marketing['language_displayed'] == language)]

slide-23
SLIDE 23

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Isolate subscribers

for language in np.unique(marketing['language_displayed'].values): print(language) language_data = marketing[(marketing['marketing_channel'] == 'Email') & (marketing['language_displayed'] == language)] subscribers = language_data.groupby(['user_id', 'variant'])['converted']\ .max()

slide-24
SLIDE 24

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Isolate control and personalization

for language in np.unique(marketing['language_displayed'].values): print(language) language_data = marketing[(marketing['marketing_channel'] == 'Email') & (marketing['language_displayed'] == language)] subscribers = language_data.groupby(['user_id', 'variant'])['converted']\ .max() subscribers = pd.DataFrame(subscribers.unstack(level=1)) control = subscribers['control'].dropna() personalization = subscribers['personalization'].dropna()

slide-25
SLIDE 25

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Full for loop

for language in np.unique(marketing['language_displayed'].values): print(language) language_data = marketing[(marketing['marketing_channel'] == 'Email') & (marketing['language_displayed'] == language)] subscribers = language_data.groupby(['user_id', 'variant'])['converted']\ .max() subscribers = pd.DataFrame(subscribers.unstack(level=1)) control = subscribers['control'].dropna() personalization = subscribers['personalization'].dropna() print('lift:', lift(control, personalization)) print('t-statistic:', stats.ttest_ind(control, personalization), '\n\n')

slide-26
SLIDE 26

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Results

Arabic lift: 50.0% t-statistic: Ttest_indResult(statistic=-0.58, pvalue=0.58) English lift: 39.0% t-statistic: Ttest_indResult(statistic=-2.22, pvalue=0.03) German lift: -1.62% t-statistic: Ttest_indResult(statistic=0.19, pvalue=0.85) Spanish lift: 166.67% t-statistic: Ttest_indResult(statistic=-2.36, pvalue=0.04)

slide-27
SLIDE 27

Let's practice!

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

slide-28
SLIDE 28

Wrap-up

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS

Jill Rosok

Data Scientist

slide-29
SLIDE 29

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Dataset

marketing = pd.read_csv('marketing.csv') print(marketing.head()) user_id date_served channel variant conv \ 0 a100000029 2018-01-01 House Ads personalization True 1 a100000030 2018-01-01 House Ads personalization True 2 a100000031 2018-01-01 House Ads personalization True 3 a100000032 2018-01-01 House Ads personalization True 4 a100000033 2018-01-01 House Ads personalization True language_displayed preferred_language age_group 0 English English 0-18 years 1 English English 19-24 years 2 English English 24-30 years 3 English English 30-36 years i i

slide-30
SLIDE 30

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Preprocessing

Feature engineering Resolving errors in the data

slide-31
SLIDE 31

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Marketing metrics

Conversion rate = Retention rate = Total number of people who we market to Number of people who convert Total number of people who converted Number of people who remain subscribed

slide-32
SLIDE 32

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Customer segmentation

marketing.groupby(['channel', 'age_group'])\ ['user_id'].count()

slide-33
SLIDE 33

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

Dip in conversion rate?

house_ads = marketing[marketing['channel'] == 'House Ads'] language = conversion_rate(house_ads, ['date_served', 'language_displayed'])

slide-34
SLIDE 34

ANALYZING MARKETING CAMPAIGNS WITH PANDAS

You analyzed an A/B test

Li T-tests

slide-35
SLIDE 35

Good luck!

AN ALYZIN G MAR K E TIN G C AMPAIG N S W ITH PAN DAS