Cloud Instance Selection for Your GPU Fleet Lessons from Developing - - PowerPoint PPT Presentation

▶
cloud instance selection for your gpu fleet
SMART_READER_LITE
LIVE PREVIEW

Cloud Instance Selection for Your GPU Fleet Lessons from Developing - - PowerPoint PPT Presentation

Dec 31, 2023 116 likes •836 views

Cloud Instance Selection for Your GPU Fleet Lessons from Developing Smart Kitchen Technology Hristo Bojinov, CTO Rob Laber, Lead CV Engineer Motivation Motivation Motivation The Thesis The Problem Food People disconnect Not-so-smart

slide-1
SLIDE 1

Cloud Instance Selection for Your GPU Fleet

Lessons from Developing Smart Kitchen Technology

Hristo Bojinov, CTO Rob Laber, Lead CV Engineer

slide-2
SLIDE 2

Motivation

slide-3
SLIDE 3

Motivation

slide-4
SLIDE 4

Motivation

slide-5
SLIDE 5
slide-6
SLIDE 6

The Thesis

slide-7
SLIDE 7

The Problem

Food ↔ People disconnect Not-so-smart “smart kitchen” Food info not available, not actionable

slide-8
SLIDE 8

The Vision

slide-9
SLIDE 9

What We Do

Food, personalization, technology “The Way You Eat”

Icons made by Madebyoliver, Popcorn Arts, Freepik from www.flaticon.com are licensed by CC 3.0 BY

slide-10
SLIDE 10

What We Do

Food, personalization, technology “The Way You Eat” (⇒ Computer Vision is essential)

Icons made by Madebyoliver, Popcorn Arts, Freepik from www.flaticon.com are licensed by CC 3.0 BY

slide-11
SLIDE 11

Pushing the Envelope on CV

Go from here...

slide-12
SLIDE 12

Pushing the Envelope on CV

… to here.

slide-13
SLIDE 13
slide-14
SLIDE 14

Computer Vision at Innit

Helps us understand users

❖ Inventory, behaviors, multi-sensor fusion, market analytics ❖ And, build a delightful user experience

Applications in storage and processing

❖ Recognize and act on food state ❖ Visible light, depth, IR

Multi-site program (HQ, academia)

slide-15
SLIDE 15

CV Research

slide-16
SLIDE 16

Food Recognition Service (AWS)

❖ G2 & G3 instance backend (details coming up) ❖ Frontend orchestrates auto and manual processing ❖ Service API for 3rd party use

slide-17
SLIDE 17

CV Tech: Food Recognition System

Service Layer Computer Vision Backend Offline Processing Data Store Web Portal

slide-18
SLIDE 18

CV Tech: Food Recognition System

slide-19
SLIDE 19

CV Tech: Object Detection Stage

slide-20
SLIDE 20

CV Tech: Object Detection Stage

slide-21
SLIDE 21

CV Tech: Object Detection Stage

slide-22
SLIDE 22

CV Tech: Object Detection Stage

slide-23
SLIDE 23

CV Tech: Classification Stage

slide-24
SLIDE 24

CV Tech: Classification Stage

slide-25
SLIDE 25

CV Tech: Classification Stage

slide-26
SLIDE 26

CV Tech: Classification Stage

Controlled scene layout ⇒ precision In-house data collection and tools CNN based

slide-27
SLIDE 27

CV Tech: Product DB Image Retrieval

slide-28
SLIDE 28

CV Tech: Product DB Image Retrieval

❖ Exact product (or attribute) matching ❖ KAZE descriptors (GPU acceleration WIP, stay tuned) ➢ Current need to balance CPU/GPU ➢ Order-of-magnitude acceleration ❖ Hierarchical analysis in the pipeline ❖ + Precise matching using CNN

slide-29
SLIDE 29

Summary: The Challenge

Complex Scenes

❖ Segmentation / object detection required ❖ Must handle occlusions, uncertainty ❖ Highly variable lighting conditions ❖ Product DB not 100% accurate ❖ Non-image feeds?

slide-30
SLIDE 30

Cloud instances: types, plans, sizes...

slide-31
SLIDE 31

So what is all this?

Service Layer Computer Vision Backend Offline Processing Data Store Web Portal

slide-32
SLIDE 32

Decoupled Service Architecture

Service Layer Computer Vision Backend Offline Processing Data Store Web Portal

slide-33
SLIDE 33

Decoupled Service Architecture

Service Layer Computer Vision Backend

slide-34
SLIDE 34

Decoupled Service Architecture

Service Layer Computer Vision Backend

❖ Single responsibility

slide-35
SLIDE 35

Decoupled Service Architecture

Service Layer Computer Vision Backend

❖ Single responsibility ❖ Stateless GPUs

slide-36
SLIDE 36

Decoupled Service Architecture

Service Layer Computer Vision Backend

❖ Single responsibility ❖ Stateless GPUs ❖

slide-37
SLIDE 37

Decoupled Service Architecture

Service Layer Computer Vision Backend

slide-38
SLIDE 38

Decoupled Service Architecture

Service Layer Computer Vision Backend

❖ Non-vision tasks

slide-39
SLIDE 39

Decoupled Service Architecture

Service Layer Computer Vision Backend

❖ Non-vision tasks ❖ Horizontal scaling

slide-40
SLIDE 40

Decoupled Service Architecture

Service Layer Computer Vision Backend

❖ Non-vision tasks ❖ Horizontal scaling ❖ Reserved instances

slide-41
SLIDE 41

Decoupled Service Architecture

slide-42
SLIDE 42

Decoupled Service Architecture

WHY?

slide-43
SLIDE 43

Decoupled Service Architecture

Service Layer Computer Vision Backend

slide-44
SLIDE 44

Decoupled Service Architecture

Service Layer Computer Vision Backend Service Layer Service Layer

slide-45
SLIDE 45

Decoupled Service Architecture

Service Layer Computer Vision Backend Service Layer Service Layer

Cost-effective buffer

slide-46
SLIDE 46

Decoupled Service Architecture

Service Layer Computer Vision Backend

To summarize...

slide-47
SLIDE 47

Decoupled Service Architecture

Service Layer Computer Vision Backend

To summarize... ❖ Reserved instances

slide-48
SLIDE 48

Decoupled Service Architecture

Service Layer Computer Vision Backend

To summarize... ❖ Reserved instances ❖ Optimize hardware usage

slide-49
SLIDE 49

Decoupled Service Architecture

Service Layer Computer Vision Backend

To summarize... ❖ Reserved instances ❖ Optimize hardware usage

Delegate tasks to save money!

slide-50
SLIDE 50

Isolated Background Jobs

Service Layer Computer Vision Backend Offline Processing Data Store Web Portal

slide-51
SLIDE 51

Isolated Background Jobs

Clustering

slide-52
SLIDE 52

Isolated Background Jobs

Big Data

slide-53
SLIDE 53

Isolated Background Jobs

Neural Net Training

slide-54
SLIDE 54

Isolated Background Jobs

Offline Processing Data Store

slide-55
SLIDE 55

Isolated Background Jobs

Offline Processing Data Store

❖ One-off or periodic

slide-56
SLIDE 56

Isolated Background Jobs

Offline Processing Data Store

❖ One-off or periodic ❖ Resource intensive

slide-57
SLIDE 57

Isolated Background Jobs

Offline Processing Data Store

❖ One-off or periodic ❖ Resource intensive ❖ Need it now?

slide-58
SLIDE 58

Spot Pricing?

slide-59
SLIDE 59

Spot Pricing?

❖ Auctioned resources

slide-60
SLIDE 60

Spot Pricing?

❖ Auctioned resources ❖ Price changes with resource availability

slide-61
SLIDE 61

Spot Pricing?

❖ Auctioned resources ❖ Price changes with resource availability ❖ Interrupted jobs

slide-62
SLIDE 62

Isolated Background Jobs

Offline Processing Data Store

❖ One-off or periodic ❖ Resource intensive ❖ Need it now?

Spot pricing saves money!

slide-63
SLIDE 63

Isolated Background Jobs

slide-64
SLIDE 64

Trade-Offs

slide-65
SLIDE 65

Trade-Offs

❖ Cost vs. Performance

slide-66
SLIDE 66

Trade-Offs

❖ Cost vs. Performance ❖ Local vs. Cloud

slide-67
SLIDE 67

Trade-Offs

❖ Cost vs. Performance ❖ Local vs. Cloud ❖ Cost vs. Complexity

slide-68
SLIDE 68

Trade-Offs

❖ Cost vs. Performance ❖ Local vs. Cloud ❖ Cost vs. Complexity ❖ Now vs. Later

slide-69
SLIDE 69

Takeaways

❖ Complex workload → use differentiated EC2 instances ❖ Profile GPU-dependent tasks, choose carefully ❖ Consider online / offline loads, spot instances

slide-70
SLIDE 70

Join the party! hristo@innit.com

slide-71
SLIDE 71

About Innit

❖ Inform and elevate the interaction between people and food ❖ 5+ years in the making, substantial funding, IP & tech ❖ Partnerships (appliances, FMCG, culinary)

About the Presenters

❖ R&D and Technology at Innit