Business Backtesting of ML Models: A Case Study in Real Estate QCon - - PowerPoint PPT Presentation

▶
business backtesting of ml models
SMART_READER_LITE
LIVE PREVIEW

Business Backtesting of ML Models: A Case Study in Real Estate QCon - - PowerPoint PPT Presentation

Nov 05, 2022 828 likes •1.43k views

Business Backtesting of ML Models: A Case Study in Real Estate QCon New York June 2017 Nelson Ray Who has run an A/B test before? Did it go off without a hitch? Unguided A/B Testing Focus of this Talk Observational Analysis Simulation-Based

slide-1
SLIDE 1

A Case Study in Real Estate QCon New York June 2017 Nelson Ray

Business Backtesting of ML Models:

slide-2
SLIDE 2

Who has run an A/B test before?

slide-3
SLIDE 3

Did it go off without a hitch?

slide-4
SLIDE 4

Unguided A/B Testing

slide-5
SLIDE 5

Focus of this Talk

A/B Test Quasi-experiments Simulation-Based Inference Observational Analysis Cost Confidence

slide-6
SLIDE 6

Talk Structure

  • Real Estate 101 for Home Buyers and Sellers
  • The Opendoor Way
  • Resale Risk
  • Problems with A/B Testing
  • How Simulation Helped in Real Estate
  • A General Recipe for Simulation
  • Team Info.
slide-7
SLIDE 7

A/B Testing Support Group Hi, I’m Nelson! I’m a recovering A/B testing user from such places as…

  • Facebook
  • Metamarkets
  • Google
  • Opendoor

I’ll cover how to perform a business backtest of your ML models using simulations!

Introduction

slide-8
SLIDE 8

M I S S I O N

Empower everyone with the freedom to move

$25T of assets 63.5% of Americans are homeowners #1 consumer expenditure ($17,798/yr) $1.4T of annual transaction volume $100B in fees

slide-9
SLIDE 9

Research online Receive bids Interview Choose

Decide to move

100+ day process with 14% failure rate

Seller

Improvements Yard work Cleaning Photographs MLS, Zillow, Trulia Open houses Showings Maintenance Offer Counter, Acceptance Inspection Financing period Final walkthrough Offline signatures Title records Offer Inspection Financing

Realtor Sale Ready List Contract Closing Contract

C U R R E N T P R O C E S S

5.5M Americans per year buy and sell through this process

Decides to move

Buyer

Finances Location Timing

Discovery

Research online Receive bids Interview Choose Open houses Showings Viewings

Realtor Search

100+ day process with friction at each step

S E L L E R S B U Y E R S

DAY 0

DAY 121 $1000’s in upfront costs 90+ Days 4-5% in price drops and concessions 14% of deals fall- through 6-7% in fees Months of research and gathering data Months of viewing suboptimal listings Fears of the home condition and financing

DAY 0

slide-10
SLIDE 10

Research online Receive bids Interview Choose

Decide to move

100+ day process with 14% failure rate

Seller

Improvements Yard work Cleaning Photographs MLS, Zillow, Trulia Open houses Showings Maintenance Offer Counter, Acceptance Inspection Financing period Final walkthrough Offline signatures Title records Offer Inspection Financing

Realtor Sale Ready List Contract Closing Contract

C U R R E N T P R O C E S S

5.5M Americans per year buy and sell through this process

Decides to move

Buyer

Finances Location Timing

Discovery

Research online Receive bids Interview Choose Open houses Showings Viewings

Realtor Search

100+ day process with friction at each step

S E L L E R S B U Y E R S

DAY 0

DAY 121 $1000’s in upfront costs 90+ Days 4-5% in price drops and concessions 14% of deals fall- through 6-7% in fees Months of research and gathering data Months of viewing suboptimal listings Fears of the home condition and financing

DAY 0

slide-11
SLIDE 11

to experience an automated, hassle-free sales process. Simply enter your address to ensure we can accurately price your home. Fill out a short home profile with a full report of your home’s value. And receive an offer in minutes Selling your home is as easy as clicking next

S E L L E R S

slide-12
SLIDE 12

Research online Receive bids Interview Choose

Decide to move

100+ day process with 14% failure rate

Seller

Improvements Yard work Cleaning Photographs MLS, Zillow, Trulia Open houses Showings Maintenance Offer Counter, Acceptance Inspection Financing period Final walkthrough Offline signatures Title records Offer Inspection Financing

Realtor Sale Ready List Contract Closing Contract

C U R R E N T P R O C E S S

5.5M Americans per year buy and sell through this process

Decides to move

Buyer

Finances Location Timing

Discovery

Research online Receive bids Interview Choose Open houses Showings Viewings

Realtor Search

100+ day process with friction at each step

S E L L E R S B U Y E R S

DAY 0

DAY 121 $1000’s in upfront costs 90+ Days 4-5% in price drops and concessions 14% of deals fall- through 6-7% in fees <50% have a bachelor’s degree Months of research and gathering data Months of viewing suboptimal listings Fears of the home condition and financing

DAY 0

slide-13
SLIDE 13

B U Y E R S

Thousands of buyers shop with us monthly All homes come with a money-back guarantee and a 2-year warranty Searching and showings are self-service, on-demand Buying a home is as easy as clicking next Our buyers have exclusive access to our inventory

slide-14
SLIDE 14
slide-15
SLIDE 15

What is our risk in reselling a home?

slide-16
SLIDE 16
  • Listed ~$800k
  • 6+ months on market

Home 1

slide-17
SLIDE 17

Home 2

  • Listed ~$300k
  • 1 month on market
slide-18
SLIDE 18
  • Opendoor bears risk in reselling the house
  • Costs vary substantially by house
  • Fair to each seller to charge based on their expected cost

Our Philosophy

slide-19
SLIDE 19

Framing the problem

slide-20
SLIDE 20

House Economics

Conversion Fee Profit Fee

slide-21
SLIDE 21

Formalization

  • Infinite number of pricing models
  • Assuming we even had a candidate f’, how do we test this?
  • A/B testing approach
  • randomize on offers: f vs f’
  • evaluate {# of houses, profit}
slide-22
SLIDE 22

Metric Measurement Lag

  • Time to observe #
  • days
  • Time to observe $
  • months
slide-23
SLIDE 23
slide-24
SLIDE 24

Formalization

  • Infinite number of pricing models
  • Assuming we even had a candidate f’, how do we test this?
  • A/B testing approach
  • randomize on offers: f vs f’
  • evaluate {# of houses, profit}
  • Many months of measurement lag
slide-25
SLIDE 25
slide-26
SLIDE 26

Simulating Offers

  • Historical transaction data
  • House lists on the market
  • Simulate our buying process
  • Estimate our costs
  • Observe actual outcome for house
slide-27
SLIDE 27

Simulating Offers

Actual resale cost: $50k Expected resale costs

  • funder: $10k -> {Paccept: .9, $: -40k}
  • fbase: $55k -> {Paccept: .1, $: 5k}

Actual resale cost: $10k Expected resale costs

  • funder: $5k -> {Paccept: .9, $: -5k}
  • fbase: $8k -> {Paccept: .7, $: -2k}
slide-28
SLIDE 28

Simulating Offers

$ #

fbase funder

slide-29
SLIDE 29

Simulating Offers

$ #

fbase funder f2base

slide-30
SLIDE 30

Simulating Offers

$ #

fbase funder f2base f3base

slide-31
SLIDE 31

Simulating Offers

$ #

fbase funder f2base f3base f4base

slide-32
SLIDE 32

Simulating Offers

$ #

fbase funder f2base f3base f4base

slide-33
SLIDE 33

Simulating Offers

$ #

fbase funder f2base f3base f4base

slide-34
SLIDE 34

Simulating Offers

$ #

fbase funder f2base f3base f4base

$ #

fbase funder

slide-35
SLIDE 35

Simulating Offers

$ #

fbase funder f2base f3base f4base

$ #

fbase funder

slide-36
SLIDE 36

Understanding Current Trade-Offs

$ #

fbase funder f2base f3base f4base

$ #

fbase funder

  • Clarity into trade-offs
  • Identify suitable

candidates

  • Backtesting with business

metrics

  • Seconds vs months
  • Only cost is computational
  • Though quality dependent
  • n simulation models
slide-37
SLIDE 37

Estimating Future Trade-Offs

$ #

fbase funder f2base f3base f4base

slide-38
SLIDE 38

Estimating Future Trade-Offs

$ #

fbase funder f2base f3base f4base Cdesired

slide-39
SLIDE 39

Estimating Future Trade-Offs

$ #

fbase funder f2base f3base f4base Cdesired Coracle

slide-40
SLIDE 40

Oracle Performance

Actual resale cost: $50k Expected resale costs

  • funder: $10k -> {Paccept: .9, $: -40k}
  • fbase: $55k -> {Paccept: .1, $: 5k}
  • foracle: $50k -> {Paccept: .15, $: 0k}

Actual resale cost: $10k Expected resale costs

  • funder: $5k -> {Paccept: .9, $: -5k}
  • fbase: $8k -> {Paccept: .7, $: -2k}
  • foracle: $10k -> {Paccept: .65, $: 0k}
slide-41
SLIDE 41

Estimating Future Trade-Offs

$ #

fbase funder f2base f3base f4base Cdesired Coracle

slide-42
SLIDE 42

Estimating Future Trade-Offs

$ #

fbase funder f2base f3base f4base Cdesired Coracle CBayes

slide-43
SLIDE 43

$ #

fbase funder f2base f3base f4base C20% Coracle CBayes

  • Estimate what is theoretically achievable
  • Set ML improvement goal
  • Translation into business trade-offs
  • “Easy” part is to hit ML target

Estimating Future Trade-Offs

slide-44
SLIDE 44

Simulation Accuracy

slide-45
SLIDE 45

Unguided A/B Testing

slide-46
SLIDE 46

The Guide

slide-47
SLIDE 47

Guided A/B Testing

slide-48
SLIDE 48

Pyramid of Causal Inference

A/B Test Quasi-experiments Simulation-Based Inference Observational Analysis Cost Confidence

slide-49
SLIDE 49

Recipe for guided testing

slide-50
SLIDE 50

Simulating Offers

  • Historical transaction data
  • House lists on the market
  • Simulate our buying process
  • Estimate our costs
  • Observe actual outcome for house

Data generating process User model

slide-51
SLIDE 51

Recipe: Data Generating Process

Simple version: replay historical data

  • Home buying and selling
  • Past housing transactions
  • Ridesharing services
  • Passenger app sessions
  • Search engine ad auctions
  • Stock of potential ads
slide-52
SLIDE 52

Recipe: User Model

  • Home buying and selling
  • P(sell | cost)
  • Ridesharing services
  • P(accept ride | price, ETA)
  • Search engine ad auctions
  • P(click | user features, ad features)
slide-53
SLIDE 53

A/B test responsibly

slide-54
SLIDE 54

Simulate before testing

slide-55
SLIDE 55
  • Founded: March 2014
  • Transactions / month: 500
  • Number of employees: 300
  • 50 data scientists and engineers
  • We’re hiring!
  • E-mail: nelson@opendoor.com

Opendoor by the numbers

slide-56
SLIDE 56

Title Text

Title Text

Acknowledgements

slide-57
SLIDE 57
slide-58
SLIDE 58

Q&A