From Whiteboard to Production System Demand Forecasting System for - PowerPoint PPT Presentation

From Whiteboard to Production System Demand Forecasting System for an Online Grocery Retailer Robert Pesch & Robin Senge Strata Data Conference NYC, September 2019

Rewe and inovex drive big data and data science initiatives in order to optimize supply chain processes since 2015 Trainings Big Data Platform Big Data Applications Data Science & AI

Customers of the Rewe delivery service order grocery for future delivery 1. Go to the online shop or mobile app 2. Fill your basket 3. Select a future delivery slot 4. Shop checks availability 5. Receive your purchase 3

Mastering the supply chain is even more important to the Rewe delivery service than to a regular food retailer Availability of products play a key role to success of the business case. 4

An improved demand forecasting system for e-Grocery has huge potential for increasing the availability of articles Reasons for unavailability (several possible per case) 100,0% 80 % 20 % 17 % 7 % Availability Not available Not available Inaccurate Central Unexpected Unexpected requests articles articles predictions logistic inventory spoilage problems correction

Setup

Data science projects face additional challenges compared to regular software development projects Whiteboard Production System • Complexity of the task (at least ‘complex’ in terms of Cynefin framework) • High uncertainty about usefulness of models • You cannot plan your success • Wrong team setup

An iterative approach is even more important to success in data science products than traditional software Analyse and visualize Modelling and Data programming preparation Iterative, agile process model, e.g. Scrum Business Data problem Deployment Evaluation gathering definition 8

Create a feeling of joint responsibility – Create a common goal € Supply Chain Data Scientist Software Engineer Big Data Process Owner Platform Engineer 9

On the Whiteboard

We measure the real customer request during checkout

Interactive data exploration and evaluation reports help understanding your data Learn more about your data ● Evaluate current solution and refined models ● Generate new ideas ●

There will not be a single best model for all products Demand patterns vary strongly ● among products Long tail: high and low demand ● products Following the „No free ● lunch“-theorem there will not be a single best model

Evaluate simple models to test your pipelines and to learn more about your data Last observed data point ● Moving average ● Constant value, … ●

Classical time-series methods are well researched and statistical sound methods Autoregressive Model Exponential Smoothing (S)ARIMA(X), Prophet, ... ... 16

The ability to integrate exogenous variables is limited 17

Regression models enable the utilisation of arbitrary features and algorithms Transform and ID Date Price ... Amount Train define features model 00001 2017- 2.69 114 03-31 Known orders ● 00002 2017- 0.49 111 Calendar ● 03-31 information Price ● .. .. .. Promotions ● Demand ● 99999 2018- 1.79 121 averages 03-31 Geographical ● location, ...

Each model class comes with strengths and weaknesses Linear Regression Nearest Neighbours Easy to interpret Easy to interpret Limited extrapolation capacity Limited expressiveness (linear dependencies) Boosted Regression Trees Artificial Neural Networks Already strong out-of-the-box Potentially strong model class Not much data preparation necessary Specialized topologies (LSTM) Limited extrapolation capacity Need lots of computing power Prone to outliers “blind spots” High effort in engineering Ensembles Ensemble effect helps combining the strengths of different models High effort to support many models 19

Automated model selection enables choosing one model per product • Finding the best models manually does not scale • Automated mapping of best models using error metrics • Enable adaption to demand pattern changes • Legacy system as benchmark and fallback model M. Ulrich, H. Jahnke, L. Langrock, R. Pesch, R. Senge, Automated model selection in retail demand forecasting using supervised learning, under review

Towards Production System

Machine learning code is only a tiny part of a data science product Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS, 2015

The forecasting system comprises many components Runs once per day Blacklist Data Feature- Prediction- Outlier- Blacklist & Workflow collector Generator Detection Importer Predictions Filtered Features Raw data Cleaned predictions data Runs once per month Complexity Trainer & Model- Evaluator Selector Selection Results Simulator KPI simulations

The technology stack Frontend and Monitoring Machine Learning Languages Data Processing Frameworks Infrastructure 24

PySpark enables us to parallelize training and application of scikit-learn models Many models ● Parallelize model training and ● evaluation Prototyped models can be ● used directly

Monitoring

Monitoring is an integral part of all productive software systems ● Monitoring your system components for errors and failures is the standard Add the following: ● Check your input data for validity ● Check your output data for plausibility

Can I trust my input data? Do outlier detection! ● Apply e.g. ● – quantile filters – rules – proximity based methods Act on them by ● – imputation – removal of data points – skip features 28

Automated checks test millions of predictions and lets you focus Manual checking of millions of ● predictions is not feasible Predictions might suffer from: ● yet hidden programming bugs ○ instable models ○ broken assumptions ○ blind spots ○ Focus on predictions that seem ● strange

Results

New system reduces non-availability by 50% compared to legacy system

Forecasting the whole demand distribution enables us to manage the trade-off between availability and spoilage ● Most models predict expected NAV SP value assuming symmetric costs ● Actually e-grocery exhibits asymmetrical costs and a non-trivial cost-function ● Specific for each individual product (esp. spoilage) 99% 50% 1% target quantile Distributional Regression for Demand Forecasting in e-Grocery - https://ssrn.com/abstract=3312609

Take Home Messages • Reduce complexity iteratively by an agile process • Create a feeling of joint responsibility in your team • There is no free lunch and recall Occam’s razor • Automate input and output outlier checks • Keep it simple as long as possible, it gets complex early enough

Vielen Dank Dr. Robert Pesch Dr. Robin Senge inovex GmbH Ludwig-Erhard-Allee 6 76131 Karlsruhe GERMANY robert.pesch@inovex.de robin.senge@inovex.de