HCMUS at the NTCIR-14 Lifelog-3 Task Nguyen-Khang Le, Dieu-Hien - - PowerPoint PPT Presentation

▶
hcmus at the ntcir 14 lifelog 3 task
SMART_READER_LITE
LIVE PREVIEW

HCMUS at the NTCIR-14 Lifelog-3 Task Nguyen-Khang Le, Dieu-Hien - - PowerPoint PPT Presentation

Aug 15, 2022 563 likes •822 views

HCMUS at the NTCIR-14 Lifelog-3 Task Nguyen-Khang Le, Dieu-Hien Nguyen, Trung-Hieu Hoang, Thanh-An Nguyen, Thanh-Dat Truong, Duy-Tung Dinh, Quoc-An Luong, Viet-Khoa Vo-Ho Vinh-Tiep Nguyen, Minh-Triet Tran University of Science, VNU-HCM, Ho Chi

slide-1
SLIDE 1

HCMUS at the NTCIR-14 Lifelog-3 Task

Nguyen-Khang Le, Dieu-Hien Nguyen, Trung-Hieu Hoang, Thanh-An Nguyen, Thanh-Dat Truong, Duy-Tung Dinh, Quoc-An Luong, Viet-Khoa Vo-Ho Vinh-Tiep Nguyen, Minh-Triet Tran

University of Science, VNU-HCM, Ho Chi Minh City, Vietnam University of Information Technology, VNU-HCM, Vietnam.

1

slide-2
SLIDE 2

Outline

  • 1. Lifelog-3 task
  • 2. Retrieval System Overview

○ Data processing ○ User interaction

  • 3. Experiment
  • 4. Result
  • 5. Conclusion

2

slide-3
SLIDE 3

Lifelog-3 task

  • 1. Advance the research in lifelogging
  • 2. Three sub-tasks:

○ Lifelog Insight Task (LIT) ○ Lifelog Activity Detection Task (LADT) ○ Lifelog Semantic Access Task (LSAT) ■ Interactive manner ■ Automatic manner

  • 3. Dataset:

○

42 days

○

Multimedia, Biometrics, Human Activity, Computer Usage

3

slide-4
SLIDE 4

Retrieval System Overview

  • 1. Offline data processing
  • 2. User interaction

4

slide-5
SLIDE 5

Retrieval System Overview

5

slide-6
SLIDE 6

Scene classification

  • Model: Residual Network (ResNet)
  • Dataset: Places365-Standard dataset

○ 102 scene attributes ○ 365 scene categories

  • Filter attributes, categories

6

slide-7
SLIDE 7

Scene classification

7

slide-8
SLIDE 8

Object detection

  • COCO Object detection

○ 80 concepts, 11 super-categories

  • Habit-based object detection

○ A set of detectors ○ To detect concepts in the lifelogger’s daily activities 8

slide-9
SLIDE 9

Object detection

  • COCO Object detection

○ Faster R-CNN ○ MS COCO Dataset

  • Habit-based object detection

○ Faster R-CNN ○ Extracted from Open Images Dataset V4 9

slide-10
SLIDE 10

Open Images Dataset V4

10

slide-11
SLIDE 11

Habit-based object detection

11

slide-12
SLIDE 12

Habit-based object detection

12

slide-13
SLIDE 13

User interaction

  • A friendly user web interface that allow the user to:

○ Input criteria (scene, concepts, time, .etc) ○ Traverse back and forth from a moment ○ Modify answer 13

slide-14
SLIDE 14

User interaction

14

slide-15
SLIDE 15

User interaction

15

slide-16
SLIDE 16

User interaction

16

slide-17
SLIDE 17

User interaction

17

slide-18
SLIDE 18

Experiment

  • Find the moment when User 1 was eating ice-cream

beside the sea 18

slide-19
SLIDE 19

Experiment

  • Find the moment when User 1 was eating fast food

alone in a restaurant. 19

slide-20
SLIDE 20

Results

  • Highest result in NTCIR-14 LSAT
  • Rank 1 in ImageCLEF 2019 Lifelog - LMRT
  • Top 3 LSC Lifelog Search Challenge (LSC 2019)

20

slide-21
SLIDE 21

Conclusion

  • Retrieval System

○ Data processing, User interaction ○ Use visual information

  • Future work

○ Make use of other metadata ○ Automatic run 21

slide-22
SLIDE 22

THANK YOU

22

slide-23
SLIDE 23

Methods comparison

23

slide-24
SLIDE 24

Lifelog Semantic Access Task (LSAT)

  • Retrieve specific moments in the lifelogger's life
  • Example: Find the moment when User 1 was eating ice-

cream beside the sea. 24