Predictive Video Retrieval A Matter of Trust Bouke Huurnink - - PowerPoint PPT Presentation
Predictive Video Retrieval A Matter of Trust Bouke Huurnink MediaMill The Team Bouke Huurnink Michiel van Liempt Jiyin He Richard van Balen Koen van de Sande FeiYan Ork de Rooij Muhammad Tahir Cees Snoek Krystian Mikolajczyk Maarten
MediaMill
Bouke Huurnink Jiyin He Koen van de Sande Ork de Rooij Cees Snoek Maarten de Rijke Jan van Gemert Jasper Uijlings Xirong Li Ivo Everts Vladimir Nedovic Michiel van Liempt Richard van Balen FeiYan Muhammad Tahir Krystian Mikolajczyk Josef Kittler Jan-Mark Geusebroek Theo Gevers Marcel Worring Arnold Smeulders Dennis Koelma
0.05 0.10 0.15 0.20 0.05 0.10 0.15 0.20
UvA
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Distribute ASR and MT over shot neighbourhood, then retrieval using language modelling approach Pseudo active-learning, with positive examples from topic and 100 random negative examples from collection Content based selection from 57 learned concepts, followed by unweighted score-based fusion
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Named entity? Trust speech results Detector match? Trust detector results Else...trust example results
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Retrieval Channels Speech Search Detector Search Example Search Predict Trusted Channel Reranking Final Results Information Need
Find shots of pieces
typing, or printing, filling more than half
Result Lists Trusted Results
Secondary Results
Secondary Results
Truncate result lists to top 1000 Eliminate all results not in trusted list Combine results with (weighted) Borda fusion
Query class determines retrieval strategy Query features determine retrieval strategy Focus on assigning query- class dependent weights Focus on identifying trusted retrieval channel
0.01 0.02 0.03 0.04 0.05 0.06 0.07
Predictive reranking Detector channel Example channel Speech channel Predictive weighted reranking
mean inferred average precision All runs
0.01 0.02 0.03 0.04 0.05 0.06 0.07
Predictive reranking Detector channel Example channel Speech channel Predictive weighted reranking
Predictive reranking
mean inferred average precision All runs
0.01 0.02 0.03 0.04 0.05 0.06 0.07
Predictive reranking Detector channel Example channel Speech channel Predictive weighted reranking
Predictive reranking
Weighting did not have big influence
mean inferred average precision All runs
person opening door a bridge people with trees and plants face filling over half the frame paper with writing people with a body of water a map vehicle moving away people looking in microscope person watching television people in a kitchen a crowd of people outdoors a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer people in white lab coats ships or boats in the water man talking to camera indoors
inferred average precision
0.1 0.2 0.3 0.4 0.5
Predictive w. reranking Detector channel Example channel Speech channel
person opening door a bridge people with trees and plants face filling over half the frame paper with writing people with a body of water a map vehicle moving away people looking in microscope person watching television people in a kitchen a crowd of people outdoors a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer people in white lab coats ships or boats in the water man talking to camera indoors
A lot of variance between channels
inferred average precision
0.1 0.2 0.3 0.4 0.5
Predictive w. reranking Detector channel Example channel Speech channel
person opening door a bridge people with trees and plants paper with writing a map people looking in microscope people in a kitchen a crowd of people outdoors a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer people in white lab coats man talking to camera indoors
Only trusted channel and reranked performance shown
Predictive w. reranking Detector channel Example channel Speech channel inferred average precision
0.1 0.2 0.3 0.4 0.5
person opening door a bridge people with trees and plants paper with writing a map people looking in microscope people in a kitchen a crowd of people outdoors a classroom scene an airplane exterior a plant that is the main object a street scene at night people at table with computer people in white lab coats man talking to camera indoors
Only trusted channel and reranked performance shown
Predictive reranking often close to or better than trusted channel
Predictive w. reranking Detector channel Example channel Speech channel inferred average precision
0.1 0.2 0.3 0.4 0.5
face filling over half the frame people with a body of water vehicle moving away person watching television ships or boats in the water man talking to camera indoors
Predictive w. reranking Detector channel Example channel Speech channel inferred average precision
0.1 0.2 0.3 0.4 0.5
Only trusted channel and reranked performance shown
face filling over half the frame people with a body of water vehicle moving away person watching television ships or boats in the water man talking to camera indoors
Predictive w. reranking Detector channel Example channel Speech channel
Predictive reranking boosts trusted channel results
inferred average precision
0.1 0.2 0.3 0.4 0.5
Only trusted channel and reranked performance shown
Beeld en Geluid Searches
20 20 uur 20 uur journaal aartsen afghanistan ajax algemene beschouwingen amsterdam
andere tijden avondjournaal balkenende beatrix buitenhof bush close up de wereld draait door debat eenvandaag evn feyenoord gemeenteraadsverkiezingen
goedemorgen nederland hirsi ali holland sport holleeder internationale nieuwsuitwisseling irak
iran jeugdjournaal journaal journaal 20 kassa klokhuis
koefnoen koninginnedag kooten kopspijkers kro kruispunt langs de lijn libanon lijst 0 lingo man bijt hond
max catherine maxima mens milosevic miniatuur moszkowicz nederland kiest netwerk nieuwslicht nioscoop nos nos journaal nova nps arena opsporing verzocht paul de leeuw pauw pauw en
witteman pauw witteman pechtold politie polygoon radar rembrandt rouvoet rutte saddam schepper
co schepper en co schipholbrand sesamstraat sonja spiritus sporen uit het oosten sport sportjournaal
studio sport tegenlicht televisie tros tv show twee vandaag uruzgan vandaag
verdonk verkiezingen voetbal vragenuur vragenuurtje vroege vogels wereld draait door wilders wouter bos
zembla zoekt en gij zult vinden zomergasten
General
Named entity queries