From Whiteboard to Production System Demand Forecasting System for - - PowerPoint PPT Presentation

From Whiteboard to Production System Demand Forecasting System for an Online Grocery Retailer

Robert Pesch & Robin Senge Strata Data Conference NYC, September 2019

Rewe and inovex drive big data and data science initiatives in order to

• ptimize supply chain processes since 2015

Trainings Big Data Platform Big Data Applications Data Science & AI

3

Customers of the Rewe delivery service order grocery for future delivery

1. Go to the online shop or mobile app 2. Fill your basket 3. Select a future delivery slot 4. Shop checks availability 5. Receive your purchase

4

Mastering the supply chain is even more important to the Rewe delivery service than to a regular food retailer

Availability of products play a key role to success of the business case.

An improved demand forecasting system for e-Grocery has huge potential for increasing the availability of articles

20 % Inaccurate predictions Central logistic problems Unexpected spoilage 7 % Unexpected inventory correction 17 % 80 % 100,0% Not available articles Reasons for unavailability

(several possible per case)

Not available articles Availability requests

Setup

Data science projects face additional challenges compared to regular software development projects

Whiteboard Production System

• Complexity of the task

(at least ‘complex’ in terms of Cynefin framework)

• High uncertainty about

usefulness of models

• You cannot plan your

success

• Wrong team setup

8

Business problem definition Deployment Analyse and visualize Modelling and programming Evaluation Data gathering Data preparation

Iterative, agile process model, e.g. Scrum

An iterative approach is even more important to success in data science products than traditional software

9

Create a feeling of joint responsibility – Create a common goal

Supply Chain Process Owner Data Scientist Software Engineer Big Data Platform Engineer

€

On the Whiteboard

We measure the real customer request during checkout

Interactive data exploration and evaluation reports help understanding your data

• Learn more about your data
• Evaluate current solution and refined models
• Generate new ideas

There will not be a single best model for all products

• Demand patterns vary strongly

among products

• Long tail: high and low demand

products

• Following the „No free

lunch“-theorem there will not be a single best model

Evaluate simple models to test your pipelines and to learn more about your data

• Last observed data point
• Moving average
• Constant value, …

Classical time-series methods are well researched and statistical sound methods

16

Autoregressive Model Exponential Smoothing ... (S)ARIMA(X), Prophet, ...

The ability to integrate exogenous variables is limited

17

Regression models enable the utilisation of arbitrary features and algorithms

ID Date Price ... Amount 00001 2017- 03-31 2.69 114 00002 2017- 03-31 0.49 111 .. .. .. 99999 2018- 03-31 1.79 121

Train model Transform and define features

• Known orders
• Calendar

information

• Price
• Promotions
• Demand

averages

• Geographical

location, ...

19

Each model class comes with strengths and weaknesses

Nearest Neighbours

Easy to interpret Limited extrapolation capacity

Linear Regression

Easy to interpret Limited expressiveness (linear dependencies)

Boosted Regression Trees

Already strong out-of-the-box Not much data preparation necessary Limited extrapolation capacity Prone to outliers “blind spots”

Ensembles Artificial Neural Networks

Potentially strong model class Specialized topologies (LSTM) Need lots of computing power High effort in engineering Ensemble effect helps combining the strengths of different models High effort to support many models

• M. Ulrich, H. Jahnke, L. Langrock, R. Pesch, R. Senge, Automated model selection in retail demand forecasting

using supervised learning, under review

Automated model selection enables choosing one model per product

• Finding the best models manually does

not scale

• Automated mapping of best models

using error metrics

• Enable adaption to demand pattern

changes

• Legacy system as benchmark and

fallback model

Towards Production System

Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS, 2015

Machine learning code is only a tiny part of a data science product

The forecasting system comprises many components

Complexity

Feature- Generator Trainer & Evaluator Prediction- Workflow Outlier- Detection Data collector Model- Selector Blacklist & Importer

Features Raw data Predictions Filtered predictions Results Selection Runs once per day Runs once per month Cleaned data

Simulator

KPI simulations Blacklist

24

Infrastructure Data Processing Frameworks Languages Machine Learning Frontend and Monitoring

The technology stack

PySpark enables us to parallelize training and application of scikit-learn models

• Many models
• Parallelize model training and

evaluation

• Prototyped models can be

used directly

Monitoring

Monitoring is an integral part of all productive software systems

• Monitoring your system

components for errors and failures is the standard Add the following:

• Check your input data for validity
• Check your output data for

plausibility

Can I trust my input data?

• Do outlier detection!
• Apply e.g.

– quantile filters – rules – proximity based methods

• Act on them by

– imputation – removal of data points – skip features

28

Automated checks test millions of predictions and lets you focus

• Manual checking of millions of

predictions is not feasible

• Predictions might suffer from:

○ yet hidden programming bugs ○ instable models ○ broken assumptions ○ blind spots

• Focus on predictions that seem

strange

Results

New system reduces non-availability by 50% compared to legacy system

99% 50% 1%

Forecasting the whole demand distribution enables us to manage the trade-off between availability and spoilage

• Most models predict expected

value assuming symmetric costs

• Actually e-grocery exhibits

asymmetrical costs and a non-trivial cost-function

• Specific for each individual product

(esp. spoilage)

target quantile Distributional Regression for Demand Forecasting in e-Grocery - https://ssrn.com/abstract=3312609

NAV SP

Take Home Messages

• Reduce complexity iteratively by an agile process
• Create a feeling of joint responsibility in your team
• There is no free lunch and recall Occam’s razor
• Automate input and output outlier checks
• Keep it simple as long as possible, it gets complex early enough

Vielen Dank

• Dr. Robert Pesch
• Dr. Robin Senge

inovex GmbH Ludwig-Erhard-Allee 6 76131 Karlsruhe GERMANY robert.pesch@inovex.de robin.senge@inovex.de