Building a culture of data-informed decision making: lessons in one - - PowerPoint PPT Presentation

Building a culture of data-informed decision making: lessons in one year of data analytics at Slack

Sarah Edge Mann, PhD April 24, 2017

Me & Analytics at Slack

University of Arizona

(Aug 2008 - Aug 2013)

• PhD in Applied Mathematics
• Minor in Computer Science
• Dissertation work in informations theory:

○ Error correcting codes ○ Fast decoding algorithms

Facebook: Data Scientist

(Oct 2013 - Feb 2016) Worked on:

• Growth
• Interfaces for low end phones,

slow internet connections

• Facebook Lite

Slack: Growth Analytics Lead

(Feb 2016 - Present)

Slack Data Analytics

• What product areas should we invest in?
• What features should we ship?
• How do people use our product?
• What is the health of our business?
• What drives our growth?
• What can we do to drive our growth?
• How can we spend money to acquire users?
• What are our capacity needs in the next year?

Mission: Help the company make data informed decisions

Slack Data Analytics

Behind the scenes:

• Ensure high quality logging - you can’t know what you don’t log!
• Curate high quality data sets, make it easy to answer common questions
• Build tools to make data accessible to everyone at Slack - data to the people!

With the product and business teams:

• Goal setting
• Goal tracking
• Understand long term trends and usage patterns
• Scope reach and impact of potential projects
• Prioritization
• Support A/B testing, should we launch a feature?

What we do

Slack Data Analytics

• Machine Learning
• Artificial Intelligence
• Build user facing products

Slack’s Search, Learning and Intelligence team does these things.

What we don’t do

When should you build an analytics team?

• You have achieved basic product market fit
• You have a large user base - analysts need large data sets to be useful

You need to have data to analyze it

Data Engineering: data availability Data Analytics: voice of data within the company Build a data engineering team before building an analytics team.

Slack’s Data Team

Data Analytics: 20 people

Product: 6 Growth & Marketing: 7 Business & Sales: 3 Analytics Tools: 3

Data Engineering: 10 people

Infrastructure: 4 Product: 5

Feature development

Two Case Studies

Team communication for the 21st century.

Case Study: Invites in the team creation flow

Case Study: Invites in the team creation flow

Invites added back into team creation flow Christmas holidays Ad campaigns and PR events

Health of new teams Date of team creation

Invites removed from team creation flow

Case Study: Invites in the team creation flow

Lesson learned: You don’t know what you don’t track!

Case Study: Mobile team creation

Health of new teams

Teams created on desktop Teams created on mobile

Date of team creation

Slice metrics to find gaps

Case Study: Mobile team creation

Teams created on desktop Teams created on mobile Product work on mobile

% of new teams that send invites Date of team creation

Dig deeper to identify opportunities

Tools & Process

Keys to a data driven culture

Slack has an Analytics Tools Team who develop:

Data visualization tools Experiments framework

• Democratization of data:

enable PMs & engineers to access data

• Minimize the time between

data questions and answers

Data visualization tools

• Science!
• Measure the impact of

every change you make

Experiments Framework

Example A/B test

Control Test

Get Started

Experiment Framework: Diversions

Get Started

?

Who should see which experience?

Experiment Framework: Exposure Logs

Get Started

Who saw which experience?

Experiment Framework: Metrics, logging

Get Started

What did people do?

Click Click

Experiment Framework: Compute, Visualize Results

Get Started

Which experience performed better?

4% Click Through Rate 5% Click Through Rate Click Through Rate: +25% 95% Confidence Interval: 18% - 32%

That sounds complicated….

So you also need process

We do two things in Growth to help make experiments run smoothly: 1) Weekly experiment reviews 2) Bi-weekly numbers review

• Meet weekly in small groups
• Everyone involved with the feature should attend:

○ Product manager, engineers, design, analyst, QA

• Check in on all experiments:

○ During development ○ Shortly after experiment launch ○ Upon conclusion

• This is a highly participatory meeting!

Experiment Reviews

• Held bi-weekly, includes the entire growth

team

• Product managers present a synthesized

story about each feature ○ What was tested and why ○ How did they test it ○ Experiment outcome ○ Lessons learned, suggestions for future work

• Deck gets shared company wide + a short

synopsis of highlights

Growth Numbers Review

• Data literacy
• Quality
• Speed
• Accountability
• Visibility
• Professional development

Benefits of this process

Case Study: Allow bulk inviting in the team creation flow.

Eligibility Any team with an email domain identified to be using GSuite (via MX record). Hypothesis Giving team creators that use GSuite an

• ption to import address books will

improve invite sends / sender and team activation.

• The number of teams

reaching 3+ users was down 5%

• Users joining teams from a

shared invite link was down dramatically

In V1.0, we saw some concerning declines.

We made two small changes - V1.1

Many users were clicking “send” with no recipients selected.

• So we grayed out the Send Invites button unless it would actually send.

Saw lower adoption and usage of invite links.

• So, we added a link to the UI.

Results

• Invites sent up 7%
• All metrics neutral to positive

Goals

Goals: What they are

• A metric + target value
• A statement about priorities

You should typically have company level goals as well as team level goals.

Goals: Why set them?

• Connects feature work to the company’s

mission and success

• Creates accountability
• Forces eyes on the numbers
• Pushes you to understand your funnel

deeply

Goal Metric Date

Process Around Goals

• Dashboards that track progress

against goals

• Weekly written reports on our goals
• Start our Numbers Review Meeting

with an update on goals

• All growth projects tie back to one
• f our goals

Growth Analytics On Call Rotation

On Call: What we do

One growth analyst is on call each week. They are responsible for:

• Triaging data availability and quality

problems

• Answering ad hoc questions about

growth data

• Writing our weekly reports on growth

metrics and goals

On Call: Benefits

• Shared responsibility for our data
• Highly collaborative team
• Better code reviews
• No one person is a single point of failure
• Everyone on the team understands all of
• ur growth metrics
• Space to focus when you aren’t on call

A better team!

Questions?