Smart Lifelog Retrieval System with Habit-based Concepts and Moment - - PowerPoint PPT Presentation

▶
smart lifelog retrieval system with habit based concepts
SMART_READER_LITE
LIVE PREVIEW

Smart Lifelog Retrieval System with Habit-based Concepts and Moment - - PowerPoint PPT Presentation

May 13, 2023 605 likes •759 views

Smart Lifelog Retrieval System with Habit-based Concepts and Moment Visualization QUIK team Tokinori Suzuki and Daisuke Ikeda Kyushu University 12 June 2019 1 Lifelog data Query Return Search Lifelong Semantic Access sub- Task (LSAT)

slide-1
SLIDE 1

Smart Lifelog Retrieval System with Habit-based Concepts and Moment Visualization

QUIK team Tokinori Suzuki and Daisuke Ikeda Kyushu University 12 June 2019

1

slide-2
SLIDE 2

Lifelong Semantic Access sub- Task (LSAT)

Given a query topic, a system retrieves relevant moments in lifeloggers’ daily life

2

Find the moments when a user was eating icecream beside the sea.

Query

Lifelog data

Return Search

slide-3
SLIDE 3

Lifelog data

  • Lifelog data are in multimodal data
  • Three contents types of users’ lifelogging

data are provided in this task

3 Multemedia data Wearable camera images, Music listing activities Biometrics data Heart rate, calorie burn, steps and blood glucose Human activity data Semantic location, physical activities

slide-4
SLIDE 4

Visual concepts of lifelog images

  • Visual concepts are labeled for each image by

auto detecters

  • Three types of visual concepts are available

4

  • 1. Attribute
  • 2. Category
  • 3. Concept

Enclosed area Home office Chair Indoor lighting Office Laptop Studying

  • Comput. room

Keyboard

  • We search moments by querying on the visual

concepts (i.e., as documents in traditional search)

slide-5
SLIDE 5

Difficulty of the task

5 Attribute Category Concept Open area Train st. Person Transportin Subway st. Sunny Railroad … …

Find the moments I was taking a train from the city to home.

Query There is a lexical gap between events/activies of a query and the visual concepts Activities / events Places / objects

slide-6
SLIDE 6

Proposed Method

6

  • 1. Similarity to the moments
  • f query topics

➕

Attribute Category Concept Open area Trench Person

  • Natu. light

Desert Person Sunny Promenade Person … … …

  • 2. Similarity on visual concepts of

images with word embeddings

http://groverflanagan.blogspot.com/ 2008_09_01_archive.html (Under a CC license)

Images

  • n the web
slide-7
SLIDE 7

Similarity to the moments of query topics

7 Moment Classifier

Classification 0.655 … 0.860 Use classification scores as the moment similarity

  • Compute the similarity to query topics by the

moments classification of 24 LSAT topics

Input Topic 1

“eating icecream”

Topic 4

“taking a train”

slide-8
SLIDE 8

Collecting training data

  • Collect images using a web search engine

(Google image search)

8

Find the moments when a user was eating icecream beside the sea.

Q

I am eating icecream beside the sea

Topic Modified topic

http://groverflanagan.blogspot.com/ 2008_09_01_archive.html (Under a CC license)

Images

  • n the web
slide-9
SLIDE 9

Collected images

  • Manually checked whether a picture is about the

moment of the queried topic

  • About 170 images were collected on average

9

100 200 300 400

Topic ID

1 3 5 7 9 11 13 15 17 19 21 23 # of images

slide-10
SLIDE 10

Global similarity

10

sim(Q, I) = α ∑

q∈Q

sim(q, V) + (1 − α) × moment(I)

Q: query, I: Images, V: visual concepts of I

  • 1. Similarity to the moments
  • f query topics

➕

Attribute Category Concept Open area Trench Person

  • Natu. light

Desert Person Sunny Promenade Person … … …

  • 2. Similarity on visual concepts of

images with word embeddings

http://groverflanagan.blogspot.com/ 2008_09_01_archive.html (Under a CC license)

Images

  • n the web
slide-11
SLIDE 11

Experiment

11

User Period # of days # of images User 1 3 May ~ 31 May 2018 29 64,132 User 2 9 May ~ 22 May 2018 14 17,615 Total 43 81,747

  • Data: two lifelogger’s data and 24 topics

Find the moments when a user was eating icecream beside the sea.

Topic

user, eating, icecream, sea

Query terms VBG NN NN NN DT VBD IN DT

  • Query terms : verb & noun words in the titles
  • POS tagger (Toutanova et al., ‘03)
slide-12
SLIDE 12

Experimental setting

  • We submitted two runs:
  • Concept uses only similarity on visual

concepts

  • Concept + Moment uses the both visual

concepts and Moment classification

12

Run Attribute Category Concept Moment Concept ✔ ✔ ✔ Concept + Moment ✔ ✔ ✔ ✔

slide-13
SLIDE 13

Official results

13

Group ID Run ID Approach MAP P@10 RelRet NTU Run1 Interactive 0.063 0.237 293 NTU Run2 Interactive 0.110 0.375 464 NTU Run3 Interactive 0.165 0.683 407 DCU Run1 Interactive 0.072 0.191 556 DCU Run2 Interactive 0.127 0.229 1094 HCMUS Run1 Interactive 0.399 0.791 1444 QUIK Run1 Automatic 0.045 0.195 232 QUIK Run2 Automatic 0.045 0.187 232 Run1: Concept, Run2: Concept+Moment

slide-14
SLIDE 14

Conclusion

  • We proposed an approach based on

moment visualization and visual concepts for NTCIR Lifelog-3 task.

  • Need to make adjustments on the weighting

parameter of similarity computing for improvement retrieval

14