Cloud Instance Selection for Your GPU Fleet Lessons from Developing - PowerPoint PPT Presentation

Cloud Instance Selection for Your GPU Fleet Lessons from Developing Smart Kitchen Technology Hristo Bojinov, CTO Rob Laber, Lead CV Engineer

Motivation

Motivation

Motivation

The Thesis

The Problem Food ↔ People disconnect Not-so-smart “smart kitchen” Food info not available, not actionable

The Vision

What We Do Food, personalization, technology “The Way You Eat” Icons made by Madebyoliver, Popcorn Arts, Freepik from www.flaticon.com are licensed by CC 3.0 BY

What We Do Food, personalization, technology “The Way You Eat” ( ⇒ Computer Vision is essential) Icons made by Madebyoliver, Popcorn Arts, Freepik from www.flaticon.com are licensed by CC 3.0 BY

Pushing the Envelope on CV Go from here...

Pushing the Envelope on CV … to here.

Computer Vision at Innit Helps us understand users Inventory, behaviors, multi-sensor fusion, market analytics ❖ And, build a delightful user experience ❖ Applications in storage and processing Recognize and act on food state ❖ Visible light, depth, IR ❖ Multi-site program (HQ, academia)

CV Research

Food Recognition Service (AWS) G2 & G3 instance backend (details coming up) ❖ Frontend orchestrates auto and manual processing ❖ Service API for 3rd party use ❖

CV Tech: Food Recognition System Computer Service Vision Layer Backend Offline Web Data Processing Portal Store

CV Tech: Food Recognition System

CV Tech: Object Detection Stage

CV Tech: Object Detection Stage

CV Tech: Object Detection Stage

CV Tech: Object Detection Stage

CV Tech: Classification Stage

CV Tech: Classification Stage

CV Tech: Classification Stage

CV Tech: Classification Stage Controlled scene layout ⇒ precision In-house data collection and tools CNN based

CV Tech: Product DB Image Retrieval

CV Tech: Product DB Image Retrieval ❖ Exact product (or attribute) matching ❖ KAZE descriptors (GPU acceleration WIP, stay tuned) ➢ Current need to balance CPU/GPU ➢ Order-of-magnitude acceleration ❖ Hierarchical analysis in the pipeline ❖ + Precise matching using CNN

Summary: The Challenge Complex Scenes Segmentation / object detection required ❖ Must handle occlusions, uncertainty ❖ Highly variable lighting conditions ❖ ❖ Product DB not 100% accurate Non-image feeds? ❖

Cloud instances: types, plans, sizes...

So what is all this? Computer Service Vision Layer Backend Offline Web Data Processing Portal Store

Decoupled Service Architecture Computer Service Vision Layer Backend Offline Web Data Processing Portal Store

Decoupled Service Architecture Computer Service Vision Layer Backend

Decoupled Service Architecture ❖ Single responsibility Computer Service Vision Layer Backend

Decoupled Service Architecture ❖ Single responsibility Computer Service Vision Layer Backend ❖ Stateless GPUs

Decoupled Service Architecture ❖ Single responsibility Computer Service Vision Layer Backend ❖ Stateless GPUs ❖

Decoupled Service Architecture Computer Service Vision Layer Backend

Decoupled Service Architecture ❖ Non-vision tasks Computer Service Vision Layer Backend

Decoupled Service Architecture ❖ Non-vision tasks Computer Service Vision Layer Backend ❖ Horizontal scaling

Decoupled Service Architecture ❖ Non-vision tasks Computer Service Vision Layer Backend ❖ Horizontal scaling ❖ Reserved instances

Decoupled Service Architecture

Decoupled Service Architecture WHY?

Decoupled Service Architecture Computer Service Vision Layer Backend

Decoupled Service Architecture Service Layer Computer Service Vision Layer Backend Service Layer

Decoupled Service Architecture Service Layer Cost-effective Computer Service Vision buffer Layer Backend Service Layer

Decoupled Service Architecture To summarize... Computer Service Vision Layer Backend

Decoupled Service Architecture To summarize... Computer Service Vision Layer Backend ❖ Reserved instances

Decoupled Service Architecture To summarize... Computer Service Vision Layer Backend ❖ Reserved instances ❖ Optimize hardware usage

Decoupled Service Architecture To summarize... Computer Service Vision Layer Backend ❖ Reserved instances Delegate tasks to ❖ Optimize hardware save money! usage

Isolated Background Jobs Computer Service Vision Layer Backend Offline Web Data Processing Portal Store

Isolated Background Jobs Clustering

Isolated Background Jobs Big Data

Isolated Background Jobs Neural Net Training

Isolated Background Jobs Offline Data Processing Store

Isolated Background Jobs ❖ One-off or periodic Offline Data Processing Store

Isolated Background Jobs ❖ One-off or periodic ❖ Resource intensive Offline Data Processing Store

Isolated Background Jobs ❖ One-off or periodic ❖ Resource intensive ❖ Need it now? Offline Data Processing Store

Spot Pricing?

Spot Pricing? ❖ Auctioned resources

Spot Pricing? ❖ Auctioned resources ❖ Price changes with resource availability

Spot Pricing? ❖ Auctioned resources ❖ Price changes with resource availability ❖ Interrupted jobs

Isolated Background Jobs ❖ One-off or periodic Spot pricing saves money! ❖ Resource intensive ❖ Need it now? Offline Data Processing Store

Isolated Background Jobs

Trade-Offs

Trade-Offs ❖ Cost vs. Performance

Trade-Offs ❖ Cost vs. Performance ❖ Local vs. Cloud

Trade-Offs ❖ Cost vs. Performance ❖ Local vs. Cloud ❖ Cost vs. Complexity

Trade-Offs ❖ Cost vs. Performance ❖ Local vs. Cloud ❖ Cost vs. Complexity ❖ Now vs. Later

Takeaways ❖ Complex workload → use differentiated EC2 instances ❖ Profile GPU-dependent tasks, choose carefully ❖ Consider online / offline loads, spot instances

Join the party! � hristo@innit.com

About Innit Inform and elevate the interaction between people and food ❖ ❖ 5+ years in the making, substantial funding, IP & tech Partnerships (appliances, FMCG, culinary) ❖ About the Presenters R&D and Technology at Innit ❖