Real-Time in the Real World: Building a State of the Art Real-Time - - PowerPoint PPT Presentation

Real-Time in the Real World:

Building a State of the Art Real-Time Analytics Platform

INFORMS Analytics Conference April 14th, 2015 Joe DeCosmo Chief Analytics Officer Enova International

k k

We are a growing global online Lender

• Founded and HQ’d in Chicago since 2004
• 1,000+ employees – 500 corporate employees
• Scalable and flexible technology platform
• Proprietary analytics and data
• Publicly traded on NYSE (ENVA) since Nov. 13th, 2014

Over $13 billion in credit extended to 3MM+ customers around the world

• 51 employees, six teams
• 24 new hires in the last 12 months, 8 interns
• Mix of college hires and experienced analysts

Our Analytics Team …

CAO Data Services BI Portfolio Analytics Fraud Marketing Analytics Research and Platforms

Our teams are responsible for protecting and growing the profitability of the company through data-driven decision-making

Technologies we use and key challenges we face every day

SAS R Python Matlab Wolfram/Mathematica Microstrategy SQL Pentaho Hadoop

• Model accuracy
• High volume, low latency

→ 30K to 75K decisions

per hour, < 1 second response

• Multi-national models and

data

• What about growth?

Real-time analytics is critical for a fast and easy online customer experience

5

• 1. Apply
• 2. Underwrite

Easy-to-complete identity, employment, income, payroll date, bank account information Multi-stage Screening to verify identity and prevent fraud Decisioning in 3 to 6 seconds, analytics system pulls data, determines credit worthiness, and presents an offer Advanced Analytics massive parallel processing of 100 algorithms, 1,000 variables, 10 years and 9 TB of customer behavior data

• 3. Accept & Fund
• 4. Service

Accept agreements reviewed and signed online Funding via ACH by next business day in U.S., within 10 minutes to debit card in U.K. Multi-Channel Service U.S. based in-house service center 24/7 for assistance and payment Proprietary Systems tailored CRM system integrated with analytics engine and marketing channels

Implementing models in real-time can be challenging

Modeler builds model offline in R, SAS, Python, etc Modeler translates model for engineering Specifications are routed to software engineer (via a project manager) Software engineer attempts to implement the specifications in the production system in scalable fashion Software engineer and modeler compare results of test records and debug

This process could take weeks or even months for a complex model !!

Instruction: “Take the log of the variable X1” Modelers Intent: Natural log of X1, if X1 < 0 then use a specified default value Software Engineer’s Implementation: Log Base 10 of X1. If X1<0, then language throws an error.

Even simple instructions can lead to errors …

• Home-grown system called MEF (Mathematical Equations Framework)
• Used proprietary domain-specific language to specify models
• Constructed in-house in written in C

Our old system

Variable Name Group Operation Coefficient Default A alpha mult 2.5 1.0 B alpha mult 1.7 4 Intercept alpha mult 4.3 4.3 alpha agg sum 1 1

Result = A * 2.5 + B * 1.7 + 4.3 If A is null, replace with 1.0 If B is null, replace with 4

MEF pros and cons

Pros

• Tightly integrated into production

system

• Developed and maintained in house
• It worked!!!

Cons

• Tightly integrated into production

system

• Limited model types
• Slow model deployment and tedious

reconciliation

• Only a few people really knew

how it worked!!

So what did we do?

10

Beginning in Q3 2013, we set out to build a new real-time platform from the ground up … – Support for a wider variety of models, data sources, and variables – Ability to execute scoring code from multiple platforms – Greatly speed up the time from modeling to deployment – Analysts can deploy models without software engineering – Can be called by any Enova or external service and return a result – Can scale up to 500,000 model evaluations per hour

We built a brand new platform … code name COLOSSUS!

We moved quickly and took a phased approach

• Initial

Requirements

• Vendor List

Q3 2013

• RFP
• Paid Pilot
• Final

Selection

Q4 2013

• Design and

build

Q1 2014

• Complete build
• Implementation
• Training

Q2 2014

• GO

LIVE!!! Q3 2014

• Statistical Computing Software Vendors
• Scientific Computing Software Vendors
• Open Source Software with Paid Consultants

We evaluated a variety of vendors and platforms

We took two finalists into a paid pilot project and made our final decision based on the results of the pilot

What did we build?

14

Application 1 Application 2 Application n

• JSON

response JSON request

COLOSSUS

EnovaModelEvaluate

EnovaModelApply Model is applied to clean-key-value rules DefaultVariableCheckAndFill Variable exception checks are run, clean key value rules are prepared EnovaConditionalModelSelector Model selection based on raw key- value rules

Request translated into readable key-value rules Results including detailed variable handling information is converted back to JSON

• Currently have 27 models in production
• Cut average model deployment time in half
• Processed 2.7 Million model requests in first two months of 2015
• Average model response time of .07 seconds

Where are we now?

We’re rolling out to all of our lines of business now and improving the variable fetching and storage to drive response time even lower

Lessons Learned

16

1. Meet the partners’ implementation teams 2. Invest in a paid pilot 3. Provide live use cases, example data, and unit tests at the very beginning 4. Define and enforce common architecture, vocabulary, and documentation 5. Plan for repairs

We learned a lot of lessons (some the hard way!) …

Joe DeCosmo Chief Analytics Officer jdecosmo@enova.com 200 West Jackson Blvd. Chicago, IL 60606 Tel 312.800.4390