Towards Scalable Multimedia Analytics Bjrn r Jnsson data sys group - - PowerPoint PPT Presentation
Towards Scalable Multimedia Analytics Bjrn r Jnsson data sys group Computer Science Department IT University of Copenhagen Todays Media Collections Massive and growing Europeana > 50 million items DeviantArt >
– Europeana > 50 million items – DeviantArt > 250 million items (160K/day) – Facebook > 1,000 billion items (200M/day)
– Novices à enthusiasts à scholars à experts – Current systems aimed at helping experts
2
3
Search Exploration
4 [Zahálka and Worring, 2014]
Multimedia Analytics
Multimedia Analysis Visual Analytics
[Zahálka and Worring, 2014] 5
[Zahálka and Worring, 2014; Keim et al., 2010]
6
7 Generic data and annotation based on objective understanding Predefined, fixed annotation based
the collection Specific context and task-driven subjective understanding Dynamically evolving and interaction-driven understanding of collections
Semantic Gap
[Smeulders et al., 2000]
Pragmatic Gap
[Zahálka and Worring, 2014]
[Zahálka and Worring, 2014]
8
Scalable Multimedia Analytics
Multimedia Analysis Visual Analytics Data Management
[Jónsson et al., 2016] 9
10 Generic data and annotation based on objective understanding Predefined, fixed annotation based
the collection Pre-computed indices and bulk processing of large datasets Specific context and task-driven subjective understanding Dynamically evolving and interaction-driven understanding of collections Serendipitous and highly interactive sessions on small data subsets
Semantic Gap
[Smeulders et al., 2000]
Pragmatic Gap
[Zahálka and Worring, 2014]
Scale Gap
[Jónsson et al., 2016]
[Jónsson et al., MMM 2016]
Ten Research Questions for Scalable Multimedia Analytics
12
13
Service Layer
14 Batch Layer Storage Layer Speed Layer [Marz and Warren, 2015]
Service Layer
15 Batch Layer Storage Layer Speed Layer
16
Gylfi Þór Guðmundsson, Laurent Amsaleg, Björn Þór Jónsson, Michael J. Franklin Towards Engineering a Web-Scale Multimedia Service: A Case Study Using Spark Proceedings of the ACM Multimedia Systems Conference (MMSys) Taipei, Taiwan, June, 2017
17
– Multimedia collections are growing – Computing power is abundant
– Automatically Distributed Computing Frameworks – Designed for high-throughput processing
18
– Transform one RDD to another via operators – Lazy execution – Master and Workers paradigm
19
– Well known application – Two phases: Off-line & “On-line”
Search results Query Image CBIR System
20
Properties:
throughput by batching Why?
CBIR algorithms
implementation facilitates comparison
21
22
– Build the index hierarchy – Cluster the data collection
– Approximate k-NN search – Vote aggregation
Index is in RAM Clusters reside on disk
Searching a single feature
Identify Retrieve Scan
k-NN
Clustered collection
l Relative indexing time on 2-24 virtual
2 4 6 8 10 12 14 16 18 20 22 24 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.00 0.10 0.20 0.30 0.40 0.50 0.60
Relative Scalabilty
Measured wall clock time Optimal trend Corrected trend
Number of cores Time relative to using 2 cores
Real cores HT cores
100K 50K 20K 10K 5K 3K 2K 1K 500 200 100 10 1,000 2,000 3,000 4,000 5,000 6,000
Throughput
8.1B desc. 1.6M clusters, L=4 Index Batches of images from 10 to 100,000
Scan wall time per image Scan CPU time per image
Batch size in images Milliseconds
81ms
– Various feature sets (visual, semantic, …) – 99.2M images and 0.8M videos – Largest dataset publicly available
– Goal is to test at a very large scale – No feature aggregation or compression – Largest feature collection reported!
25
26
R1 Scalability Ability to scale out with additional computing power R2 Computational flexibility Ability to carefully balance system resources as needed R3 Capacity Ability to gracefully handle data that vastly exceeds main memory capacity R4 Updates Ability to gracefully update the data structures for dynamic workloads R5 Flexible pipeline Ability to easily implement variations of the indexing and/or retrieval process R6 Simplicity How efficiently the programmer’s time is spent
27
28
– Scalability limited due to RAM per core – Two-step Map-Reduce pipeline is too rigid
– R1, R2, R3 = partially; R4 = no; R5, R6 = no
– Arbitrarily deep pipelines
– Broadcast variables
– Multiple data sources
29
30
.map .flatmap .groupByKey .reduceByKey .collectAsMap
31
32
Indexing Search
– 800 real cores + 800 virtual cores – 2.8 TB of RAM and 30 TB of SSD space
33
Features (billions) Indexing time (seconds) Scaling (relative) 8.5 3,287 – 17.2 5,030 1.53 26.0 11,943 3.63 34.5 14,192 4.31 42.9 19,749 6.00
34
Hadoop limit
35
R1 Scalability R2 Computational Flexibility R3 Capacity R4 Updates R5 Flexible Pipelines R6 Simplicity
Yes Yes Yes Partial
full re- shuffle
Yes Yes
Partial
RAM per core
Partial Partial No No No
demonstration web-site under development
1. Guðmundsson, Amsaleg, Jónsson, Franklin (2017)
– 42.9 billion SIFTs – 51 servers
2. Moise, Shestakov, Guðmundsson, Amsaleg (2013)
– 30.2 billion SIFTs – 108 servers
3. Lejsek, Jónsson, Amsaleg (2017?)
– 28.5 billion SIFTs – 1 server
4. Lejsek, Jónsson, Amsaleg (2011)
– 2.5 billion SIFTs – 1 server
5. Jégou et al. (2011)
– 2 billion ... – 1 server
6. Sun et al (2011)
– 1.5 billion ... – 10 servers
Service Layer
37 Batch Layer Storage Layer Speed Layer
38
Service Layer
39 Batch Layer Storage Layer Speed Layer
40
Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, Marcel Worring Blackthorn: Large-Scale Interactive Multimodal Learning Accepted to IEEE Transactions on Multimedia Jan Zahálka, Stevan Rudinac, Björn Þór Jónsson, Dennis C. Koelma, Marcel Worring Interactive Multimodal Learning on 100 Million Images Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR) New York, NY, USA, June, 2016
[Zahálka and Worring, 2014; Keim et al., 2010]
41
42
43
10-100K images
100M images
44
Image credit: http://demonocracy.info/infographics/usa/us_debt/us_debt.html
45
– But … 1000D features à 800GB
– But … computing 100M feature scores takes minutes!
46
47
48
49
50
51
52
53
54
55
“We choose to … in this decade and do the other things, not because they are easy, but because they are hard, …” We choose to go to the moon. We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard, because that goal will serve to organize and measure the best of our energies and skills, because that challenge is one that we are willing to accept, one we are unwilling to postpone, and one which we intend to win, and the others, too. JFK, September 12, 1962
56
Industry-Level Collections
– Data – Processing capacity
Small-Minded Reviewers
– “Are there users willing to explore 100M data sets interactively?” – Too much engineering – not enough “science”
Interactive Applications
– Application knowledge – User study “victims”
57