One Year with Sagemaker Our experience of enhancing SaaS products - - PowerPoint PPT Presentation

One Year with Sagemaker

Our experience of enhancing SaaS products based on custom Deep Learning

Dennis Weyland – econda GmbH | September 9, 2019

Community Day 2019 Sponsors

Dennis Weyland

»

Big Data & Artificial Intelligence

»

Realtime applications

»

Cloud Infrastructure

»

Technical Business Developement

»

Since 2018 at econda

Data Engineer Personalization Team econda GmbH

E-Mail: weyland@econda.de

1. Starting Point: Perfect Salesperson

» Who visits the shop? » What do I know about this person? » How to increase revenue?

2. The Deep Learning Model

» The Combined Model » The technology behind it (Demo) » What was the uplift?

AGENDA

3. The Architecture

» Requirements » AWS Service used » Latencies

4. One Year later: What has changed?

» Model » Use Cases » Architecture » Feature Wishlist

WHO WHO

visits my shop/app/portal?

WHAT WHAT

do I know about this person?

HOW HOW

can I generate more revenue?

ONSI ONSITE TE Web tracking OFFSIT OFFSITE Adserver networks OFFLI OFFLINE NE CRM ERP ESP

External ID User ID Email hash Adserver ID 1st party cookie Customer ID (Web shop) Recipient ID

MOBILE Mobile app tracking

Mobile ID

IDENTITY + DEVICE MANAGEMENT

DIGITAL PROFILE DATA

»

Data from digital channels (e.g. Web tracking, apps)

»

Online customer journey with all relevant touchpoints

»

Cross device data CRM PROFILE DATA

»

Data from CRM and ERP systems

»

Personal record of customers

»

Offline customer journey with all relevant touchpoints TARGETING DATA

»

Data for future personalization

»

Bridge to campaign and BI systems

REALTIME PROFILE

Each realtime profile contains data from different channels. The different types of data support different use cases.

DEEP LEARNING MODELL A trained model is able to predict in real time. Those Prediction can be used for next best offers, categories or target groups. Viewed Products Shopping Basket Bought Products Search Terms Target Groups Affinities Gender Age Favorite Brands

Usage of Artificial Intelligence

NEXT BEST OFFERS

The Combined Model

Classic Live Combined

• lder than

today from today Viewed Products Shopping Basket Bought Products Search Terms Target Groups Affinities Gender Age Favorite Brands

Void

The Combined Model

Classic Live Void

» Interactions are from before

today

» Chronological order does

not matter

» Recos are slow to change » Reflects general interest » For unknown users » Use campaign information » Use weekday and hour » Before the first product

interaction

» Interactions are from today » Chronological order is

important

» The more recent, the more

relevant

» One click completely

change Recos

» Current curiosity

Combined

» Input is output of other

models

» Combines the best of all

LIVE DEMO

A/B TESTING

The added value of using new algorithms has to evaluated based on A/B tests with real customers.

VS

A

STATIC CONTENT

»

One size fits all

»

Top sellers, manually selected content STATISTICAL PERSONALIZATION

»

Usage of profile data

»

Collaborative filtering, business rules ARTIFICIAL INTELLIGENCE

»

Usage of profile data

»

Deep Learning STATIC CONTENT

»

One size fits all

»

Top sellers, manually selected content STATISTICAL PERSONALIZATION

»

Usage of profile data

»

Collaborative filtering, business rules ARTIFICIAL INTELLIGENCE

»

Usage of profile data

»

Deep Learning

B

A/B TEST – PERSONALIZED MAILINGS

VS

A

STATIC CONTENT

»

One size fits all

»

Top sellers, manually selected content STATISTICAL PERSONALIZATION

»

Usage of profile data

»

Collaborative filtering, business rules ARTIFICIAL INTELLIGENCE

»

Usage of profile data

»

Deep Learning STATIC CONTENT

»

One size fits all

»

Top sellers, manually selected content STATISTICAL PERSONALIZATION

»

Usage of profile data

»

Collaborative filtering, business rules ARTIFICIAL INTELLIGENCE

»

Usage of profile data

»

Deep Learning

B

Manually selected recommendations NBOs (Realtime profile + Deep Learning)

A/B TEST – PERSONALIZED MAILINGS

VS

Redaktionelle Empfehlungen NBOs (Echtzeitprofile + Deep Learning)

EINSATZ VON KÜNSTLICHER INTELLIGENZ

Ergebnisse aus A/B-Tests in Mailings zeigen das enorme Potential von Deep Learning in den Push-Kanälen.

VS

Mailing to customers (6-digit number of recipients)

A/B-Test

A: Widget with manually selected recommendations B: Widget mit Deep Learning NBOs

22%

Increase of revenue (%, last touch):

???

A/B TEST – PERSONALIZED ENTRY PAGE

VS

A

STATIC CONTENT

»

One size fits all

»

Top sellers, manually selected content STATISTICAL PERSONALIZATION

»

Usage of profile data

»

Collaborative filtering, business rules ARTIFICIAL INTELLIGENCE

»

Usage of profile data

»

Deep Learning STATIC CONTENT

»

One size fits all

»

Top sellers, manually selected content STATISTICAL PERSONALIZATION

»

Usage of profile data

»

Collaborative filtering, business rules ARTIFICIAL INTELLIGENCE

»

Usage of profile data

»

Deep Learning

B

DEEP LEARNING MODELL Trainierte Modelle können in Echtzeit ausgewertet werden um zum Beispiel Next Best Offers, Affinitäten oder Kunden-Scores zu berechnen. Gesehene Produkte Warenkorb Gekaufte Produkte Suchbegriffe

EINSATZ VON KÜNSTLICHER INTELLIGENZ

NEXT BEST OFFERS

Widget on main entry page of Web shop

A/B-Test

A: Widget based on statistical recommendations (e.g. bought together) B: Widget based on Deep Learning NBOs

12%

Increase of revenue (%):

???

> TRAINING SHOULD BE FAST AND COST EFFICIENT

As a SaaS our products should be as cheap as possible for our clients. Despite that, the training should be still fast enough to start in the middle of the night and finish before morning.

> TRAINING SHOULD BE DONE AUTOMATICALLY ON A DAILY BASIS

The training should be triggered on a daily basis and be able to collect the necessary data from our econda API.

> THE SERVICE MUST SCALE UP PER CLIENT

Each of our client need its own training and inference service.

> INFERENCE SERVICE WHICH SHOULD BE ROBUST, FAST AND COST EFFICIENT

Inference Service must be 24h available per day. Like training it should be as cheap as possible while allowing for predictions times <100ms.

The Architecture: What do we want to accomplish?

The architecture is designed to perform an automated training on a daily basis for each of our clients and deploy the trained model as a live inference server.

Which aws services did we chose?

Training Storage Inference

The architecture is designed to perform an automated training on a daily basis for each of our clients and deploy the trained model as a live inference server.

How to integrate the services?

aws

econda client’s webshop webshop customer

95 ms 65 ms 20 ms

Year

The Model has changed

Void

» Not every user is

recognized

» No more Topsellers

New Input

NBO Categories

» Includes classic categories,

brand, color . . .

» Show new products of your

favorite brand

» Show banner depending on

the category of interest

New Output

New use cases: Not only Widgets and Banners

Personalized Search

» Change order of Search results

based on relevance

» Can either be done on products

(rearraning) or categories (boosting)

» Real time scenario

Onsite

Display Marketing

» Target groups » Use NBO categories to

determine Target Group

» Use NBO Products to give

individual Recommendations

» Near Real Time scenario ~10 min

Offsite

> LOOK OUT FOR MANAGED SPOT TRAINING

Managed Spot training seems like a good way to reduce costs if you can wait longer for your training jobs to complete and if your algorithm supports checkpointing.

> SAGEMAKER TRAINING JOBS ARE VERY STABLE

Besides very few “Instance not available” Errors. The Service works without any worries,

> ARCHITECTURE DID NOT CHANGE

Because we didn’t have any reason to change it. Up till now our lambdas did a good job of scaling our clients and the

• nboarding of clients is fast.

> LAMBDA YOUR WAY AROUND

AWS Lambdas work well with Sagemaker. If a desired feature is missing, often you can use Lambdas to implement them yourself.

One Year with Sagemaker: Lessons Learned

> MAKE INFERENCE SECTION WORK WITH ECS/EKS

Although Sagemaker use Container for Inference internally, the feature lacks the variety of ECS/EKS. Why not link them together?

> ALARM WHEN TRAINING JOB FAILED

Although there are alarms for a specific training job, there is no metric for all or a group of training jobs.

> SCHEDULED TRAINING JOBS

Instead of triggering lambdas on a daily basis, scheduled training jobs which automatically start every day would be

• nice. They also could be grouped for better visibility.

> TYPE HINTS FOR BOTO3 (PYTHON3)

With type hints writing lambdas would be much easier. Boto3 has complex Parameters and Return types. Not having to look each one every time up would be a huge plus.

Our Wishlist for the Future

QUESTIONS? QUESTIONS?

Dennis Weyland <weyland@econda.de> www.econda.de