SLIDE 4
- M. Malinowski, M. Rohrbach, M. Fritz. Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Symbolic-based Approaches
- Large Scale Datasets
- Attention-based Approaches
- Hybrid Approaches-
4
- S. Antol et. al. Visual QA. ICCV’15
- L. Yu et. al. al. Visual Madlibs. ICCV’15
- D. Geman et. al. Visual Turing Test. PNAS’15
- M. Ren et. al. Image QA. NIPS15
- H. Gao et. al. Are You Talking to a Machine? NIPS’15
- Y. Zhu et. al. Visual7W. arXiv’15
- L. Zhu et. al. Uncovering Temporal Context. arXiv’15
- M. Malinowski et. al. Multiworld. NIPS’14
- M. Ren et. al. Image QA. NIPS’15
- H. Gao. et. al. Are You Talking to a Machine? NIPS’15
- L. Ma et. al. Learning to Answer Questions From Images. arXiv’15
- Z. Yang. et. al. Stacked Attention Networks. arXiv’15
- Y. Zhu et. al. Visual7W. arXiv’15
- J. Andres et. al. Deep Compositional QA. arXiv’15
- H. Xu et. al. Ask, Attend and Answer. arXiv’15
- K. Chen et. al. ABC-CNN. arXiv’15
- K. J. Shih et. al. Where To Look. arXiv’15
What …? λx.Behind(x,Table)
chair(1, brown, position X, Y, Z) window(1, blue, position X, Y, Z)
window
What is the mustache made of? Person A is …
“How” “many” “books” LSTM ... Softmax One Two ... Red Bird .21 .56 ... .09 .01 Linear Image CNN Word Embedding
What is the doing cat ? <BOA> Sitting
umbrella the
CNN
LSTM Embedding Fusing Sitting
umbrella the <EOA> Shared Shared Intermediate Softmax
Question: What are sitting in the basket on a bicycle? CNN/ LSTM Softmax dogs Answer: CNN
+
Query
+
Attention layer 1 Attention layer 2 feature vectors of different parts of image
A cat. Why is the person holding a knife? To cut the cake with. What kind of animal is in the photo? At the top. Where are the carrots? Three. How many people are there?
cat cake
A B C D where count color ... dog standing ...
LSTM couch
cat CNN
Where is the dog? Layout Parser
Related Work
- H. Noh et al. Dynamic Parameter Prediction. arXiv’15
- J. Andres et al. Deep Compositional QA. arXiv’15
