AMECON: Abstract Meta-Concept Features for Text Illustration Ines - - PowerPoint PPT Presentation

▶
amecon abstract meta concept features for text
SMART_READER_LITE
LIVE PREVIEW

AMECON: Abstract Meta-Concept Features for Text Illustration Ines - - PowerPoint PPT Presentation

Oct 22, 2022 316 likes •766 views

AMECON: Abstract Meta-Concept Features for Text Illustration Ines Chami 1, *, Youssef Tamaazousti 2, * and Herv Le Borgne 2 1: Stanford University, USA 2: CEA LIST, FRANCE * Both authors contributed equally | 1 ICMR 2017 | Tamaazousti

slide-1
SLIDE 1

| 1

Ines Chami1,*, Youssef Tamaazousti2,* and Hervé Le Borgne2 1: Stanford University, USA – 2: CEA LIST, FRANCE

AMECON: Abstract Meta-Concept Features for Text Illustration

* Both authors contributed equally

ICMR 2017 | Tamaazousti Youssef

slide-2
SLIDE 2

| 2 ICMR 2017 | Tamaazousti Youssef

Text-illustration System

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-3
SLIDE 3

| 3 ICMR 2017 | Tamaazousti Youssef

Cross-Modal Retrieval Task

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
  • Cross-Modal Retrieval task
  • Given a document in one modality, find (from

database) the most relevant documents in another modality

  • Text-illustration
  • Query: sentences
  • Collection: images
  • Hard problem: semantic gap
slide-4
SLIDE 4

| 4 ICMR 2017 | Tamaazousti Youssef

Cross-Modal Retrieval Approach 1

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
  • Canonical Correlation Analysis
  • Hardoon et al. Neural Computation 2004
  • Hwang and Grauman, IJCV 2012
  • Costa Pereira et al. TPAMI 2014
  • Tran et al., CVPR 2016
  • etc.
slide-5
SLIDE 5

| 5

  • Neural Network (NN)
  • Karpathy and Fei-Fei, NIPS 2014
  • Yan and Mikolajczyk, CVPR 2015
  • Karpathy and Fei-Fei, CVPR 2015
  • Mao et al., ICLR 2015
  • Kiros et al., TACL 2015
  • Wang et al., CVPR 2016
  • etc.

ICMR 2017 | Tamaazousti Youssef

Cross-Modal Retrieval Approach 2

slide-6
SLIDE 6

| 6

Main Principle of NN Approach

slide-7
SLIDE 7

| 7

Main Principle of NN Approach

slide-8
SLIDE 8

| 8

Main Principle of NN Approach

slide-9
SLIDE 9

| 9

Main Principle of NN Approach

slide-10
SLIDE 10

| 10 ICMR 2017 | Tamaazousti Youssef

This work: New Approach

slide-11
SLIDE 11

| 11 ICMR 2017 | Tamaazousti Youssef

This work: New Approach

slide-12
SLIDE 12

| 12 ICMR 2017 | Tamaazousti Youssef

This work: New Approach

slide-13
SLIDE 13

| 13 ICMR 2017 | Tamaazousti Youssef

This work: New Approach

slide-14
SLIDE 14

| 14

  • AMECON: Abstract Meta-CONcept
  • Abstract-concept + Meta-concept

ICMR 2017 | Tamaazousti Youssef

AMECON principle

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-15
SLIDE 15

| 15

  • AMECON: Abstract Meta-CONcept
  • Abstract-concept + Meta-concept

ICMR 2017 | Tamaazousti Youssef

AMECON principle

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-16
SLIDE 16

| 16

  • AMECON: Abstract Meta-CONcept
  • Abstract-concept + Meta-concept

ICMR 2017 | Tamaazousti Youssef

AMECON principle

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-17
SLIDE 17

| 17

  • AMECON: Abstract Meta-CONcept
  • Abstract-concept + Meta-concept

ICMR 2017 | Tamaazousti Youssef

AMECON principle

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-18
SLIDE 18

| 18

  • AMECON: Abstract Meta-CONcept
  • Abstract-concept + Meta-concept

ICMR 2017 | Tamaazousti Youssef

AMECON principle

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-19
SLIDE 19

| 19 ICMR 2017 | Tamaazousti Youssef

Overview of Our Approach

slide-20
SLIDE 20

| 20

  • Learning Textual Features

a. Select all different words from training-data b. Remove stop-words (``is’’, ``of’’, ``for’’, etc.) c. Compute word2vec features for each word d. Cluster (k-means) the whole set of features

ICMR 2017 | Tamaazousti Youssef

Learning Text2Amecon block

slide-21
SLIDE 21

| 21

  • Learning Textual Features

a. Select all different words from training-data b. Remove stop-words (``is’’, ``of’’, ``for’’, etc.) c. Compute word2vec features for each word d. Cluster (k-means) the whole set of features

ICMR 2017 | Tamaazousti Youssef

Learning Text2Amecon block

AMECONs

slide-22
SLIDE 22

| 22 ICMR 2017 | Tamaazousti Youssef

Computing Textual AMECON Features

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.

Test phase

slide-23
SLIDE 23

| 23 ICMR 2017 | Tamaazousti Youssef

Overview of Our Approach

slide-24
SLIDE 24

| 24 ICMR 2017 | Tamaazousti Youssef

Learning Image2Amecon block

slide-25
SLIDE 25

| 25 ICMR 2017 | Tamaazousti Youssef

Learning Image2Amecon block

slide-26
SLIDE 26

| 26 ICMR 2017 | Tamaazousti Youssef

Learning Image2Amecon block

slide-27
SLIDE 27

| 27 ICMR 2017 | Tamaazousti Youssef

Learning Image2Amecon block

slide-28
SLIDE 28

| 28 ICMR 2017 | Tamaazousti Youssef

Learning Image2Amecon block

slide-29
SLIDE 29

| 29 ICMR 2017 | Tamaazousti Youssef

Learning Image2Amecon block

slide-30
SLIDE 30

| 30 ICMR 2017 | Tamaazousti Youssef

Computing Visual AMECON Features

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.

Test phase

slide-31
SLIDE 31

| 31 ICMR 2017 | Tamaazousti Youssef

Overview of Our Approach

slide-32
SLIDE 32

| 32 ICMR 2017 | Tamaazousti Youssef

Matching Multi-Modal Data in AMECON Space

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
  • Matching texts & images in the same

AMECON Space

  • Text and Images directly comparable
  • Perform ANY multi-modal task
slide-33
SLIDE 33

| 33 ICMR 2017 | Tamaazousti Youssef

Text-Illustration in AMECON Space

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-34
SLIDE 34

| 34 ICMR 2017 | Tamaazousti Youssef

Text-Illustration in AMECON Space

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-35
SLIDE 35

| 35 ICMR 2017 | Tamaazousti Youssef

Text-Illustration in AMECON Space

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-36
SLIDE 36

| 36 ICMR 2017 | Tamaazousti Youssef

Text-Illustration in AMECON Space

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
slide-37
SLIDE 37

| 37 ICMR 2017 | Tamaazousti Youssef

Experimental Protocol

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
  • Training data
  • 6,000/30,000 images in Flickr-8k/Flickr-30k
  • Each image associated to 5 captions
  • Testing data (same for Flickr-8k & 30k)
  • 1000 images and 5000 captions
  • All captions as data-queries
  • All images as data-collection
  • Evaluation metric: Recall@K (K = 1, 5, 10)
slide-38
SLIDE 38

| 38 ICMR 2017 | Tamaazousti Youssef

Text Illustration Results

Neural Network-based Approach CCA-based Approach Our Approach

slide-39
SLIDE 39

| 39 ICMR 2017 | Tamaazousti Youssef

Text Illustration Results

Neural Network-based Approach CCA-based Approach Our Approach

slide-40
SLIDE 40

| 40 ICMR 2017 | Tamaazousti Youssef

Text Illustration Results

Neural Network-based Approach CCA-based Approach Our Approach

slide-41
SLIDE 41

| 41 ICMR 2017 | Tamaazousti Youssef

Text Illustration Results

Neural Network-based Approach CCA-based Approach Our Approach

slide-42
SLIDE 42

| 42 ICMR 2017 | Tamaazousti Youssef

Analysis of Parameters

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
  • Quite robust to the parameters
  • Robust to #selected neighbours
  • Sensitive to #clusters (C) but stable when for a

large range of C values

slide-43
SLIDE 43

| 43 ICMR 2017 | Tamaazousti Youssef

Conclusion

  • SVM one-vs-all
  • Multi-Layer neural network
  • etc.
  • Novelty:
  • Principle of AMECONs
  • Abstract MEta-CONcepts
  • Mixing supervised and unsupervised learning to

build a multi-modal space

  • Results on Text-illustration:
  • +4 points of R@K (avg.) compared to best

methods of the literature

  • Future Work:
  • Image captioning with AMECON-features
slide-44
SLIDE 44

| 44

Commissariat à l’énergie atomique et aux énergies alternatives Institut List | CEA SACLAY NANO-INNOV | BAT. 861 – PC142 91191 Gif-sur-Yvette Cedex - FRANCE www-list.cea.fr Établissement public à caractère industriel et commercial | RCS Paris B 775 685 019

Code will be released at:

http://perso.ecp.fr/~tamaazouy/

Thank you (questions ?)