Automated Management of More than One Million LifeLog Images Aiden - - PowerPoint PPT Presentation

▶
automated management of more than one million lifelog
SMART_READER_LITE
LIVE PREVIEW

Automated Management of More than One Million LifeLog Images Aiden - - PowerPoint PPT Presentation

May 06, 2023 226 likes •906 views

D UBLIN C ITY U NIVERSITY A DAPTIVE I NFORMATION C LUSTER C ENTRE FOR D IGITAL V IDEO P ROCESSING Automated Management of More than One Million LifeLog Images Aiden R. Doherty Supervisor: Prof. Alan F. Smeaton Centre for Digital Video

slide-1
SLIDE 1

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 -

Automated Management of More than One Million LifeLog Images

Aiden R. Doherty Supervisor: Prof. Alan F. Smeaton Centre for Digital Video Processiong (CDVP) & Adaptive Information Cluster (AIC), Dublin City University (DCU)

slide-2
SLIDE 2

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-3
SLIDE 3

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 -

Who am I?

  • Graduated from University of Ulster in ’05

BSc (Hons) Computing Science

  • Work placement with DuPont
  • 3rd year PhD student funded by IRCSET
  • Recently received Microsoft Postgraduate

Research Scholarship

slide-4
SLIDE 4

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 -

Centre for Digital Video Processing

  • Headed by Prof. Alan Smeaton
  • 3 faculty members, 14 post-docs, 23 PhD

students, 4 RAs, 3 support staff

  • Focus on multimedia information retrieval
  • Now investigating the area of lifelogging
slide-5
SLIDE 5

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 -

Lifelogging

  • Lifelogging is about recording daily life, digitally
  • Sometimes its for a reason,

– work … e.g. security personnel, medical staff, – personal … e.g. diaries, etc.

  • Sometimes its for posterity, recording vacations,

family gatherings, social occasions

  • Sometimes its because we can, and we’re not

yet sure what we’ll do with lifelogs, e.g. MyLifeBits

slide-6
SLIDE 6

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 -

Lifelogging Devices

  • Tano et. al. University of Electro-Communications,

Tokyo, Japan

slide-7
SLIDE 7

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 7 -

Lifelogging Devices

  • Lin & Hauptmann, Carnegie Mellon, PA,

USA

slide-8
SLIDE 8

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 8 -

SenseCam

  • SenseCam is a Microsoft Research Prototype
  • Multi-sensor device

– Colour camera – 3 accelerometers – Light meter – Passive infrared sensor

  • 1GB flash memory storage
  • Smart image capture ~3 images/min
  • Since April 2006 we’ve had two SenseCams …

recently have received 5 more

slide-9
SLIDE 9

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 9 -

How to Review Images?

  • Make a 2 minute movie of your day!
slide-10
SLIDE 10

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 0 -

Lifelogging Aiding Memory

  • Preliminary Study carried out by Cambridge

Memory Clinic, Addenbrooke’s Hospital

  • 63 year old, well-educated married woman,

with limbic encephalitis (usually has no memory a few days after an event)

  • Each day her husband would ask her what

she would remember from an event, and then talk her through it using SenseCam images afterwards

  • A few days later, the same process would be

repeated for that event

slide-11
SLIDE 11

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 1 -

SenseCam as a Memory Aid

Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf

slide-12
SLIDE 12

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 2 -

SenseCam as a Memory Aid

Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf

slide-13
SLIDE 13

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 3 -

SenseCam as a Memory Aid

Microsoft Research Cambridge presentation: http://research.microsoft.com/~shodges/presentations/UBICOMP_senseCam.pdf

slide-14
SLIDE 14

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 4 -

Require Intelligent Summarisation

  • Playing a movie of one’s day takes too long to

review

slide-15
SLIDE 15

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 5 -

Daily Browser Overview

Event Segmentation Novelty Calculation of Each Event

0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9

Composition of the Browser

2 Sept 06

Interactive Browser

Day -1 Day -2 Day -5 Day -3 Day -4 Day -6

Event-Event Comparison within the Multi-day Event database Event database containing last 7 days’ Events

Landmark Image Selection

SenseCam Images of a day (about 3,000)

slide-16
SLIDE 16

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 6 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-17
SLIDE 17

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 7 -

Event Segmentation Reminder

Event Segmentation Novelty Calculation of Each Event

0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9

Composition of the Browser

2 Sept 06

Interactive Browser

Day -1 Day -2 Day -5 Day -3 Day -4 Day -6

Event-Event Comparison within the Multi-day Event database Event database containing last 7 days’ Events

Landmark Image Selection

SenseCam Images of a day (about 3,000)

slide-18
SLIDE 18

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 8 -

Sample Activities

Breakfast Work Talking to colleague Car Airplane Talking to friend

slide-19
SLIDE 19

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 1 9 -

Event Segmentation

... adjacent images/sensor vals One Day’s Images

80 65 70 15

... adjacent blocks of 10 images/sensor vals

...... ......

Event-segmented images of a day

  • Scalable Colour
  • Colour Structure
  • Colour Layout
  • Edge Histogram

Extract MPEG-7 descriptors...

For each image...

  • Accelerometer X/Y/Z
  • Light
  • Temperature
  • Passive Infra Red

Sensor values...

For each sensor reading...

1. Raw data 2. Similarity matching 3. Normalisation & Data fusion 4. Thresholding 5. Events

120 149 289 …

Shot Boundary Detection OR TextTiling

slide-20
SLIDE 20

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 0 -

Event Segmentation Expts.

  • How well does it work?
  • Work is already published at RIAO’2007 conference (1

user and 25k images)

  • Recently completed extensive experiments with 5

different users wearing SenseCam for 1 month each (total = 270k images)

  • Each user groundtruthed their own data
  • Data divided into training and test sets with over 3,000

different combinations evaluated

slide-21
SLIDE 21

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 1 -

Event Segmentation Expts.

  • From groundtruth we noticed:

– Average of 1,785 images per user per day – Average of 22 events groundtruthed per day

  • 2 Approaches Recommended:

– Most accurate (include image MPEG-7 features) – Quick segmentation (sensor values only)

  • Performance:

– RIAO (f score = 0.40) – Sensor only (f score = 0.60) – Image + Sensor (f score = 0.62)

slide-22
SLIDE 22

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 2 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-23
SLIDE 23

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 3 -

Retrieval Reminder

Event Segmentation Novelty Calculation of Each Event

0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9

Composition of the Browser

2 Sept 06

Interactive Browser Event-Event Comparison within the Multi-day Event database Event database containing last 7 days’ Events

Landmark Image Selection

SenseCam Images of a day (about 3,000)

Day -1 Day -2 Day -5 Day -3 Day -4 Day -6

slide-24
SLIDE 24

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 4 -

Finding similar events

Similar Events - Aiden waiting for bus Mon Tue Wed Sat Thr Fri Sun Mon Tue Wed Thr Fri Sat Sun

  • How to represent/model events that consist of

many images?

  • How to compare events against each other?
  • What sources of information to use? How to

combine them?

slide-25
SLIDE 25

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 5 -

Event Retrieval Expts.

  • How well does it work?
  • Recently completed extensive experiments with 5

different users wearing SenseCam for 1 month each (total = 270k images) … corresponds to 3,286 events

  • 10 queries selected for each user e.g. driving, at work,

eating, talking to friend, etc.

  • 13,399 pooled judgements made on relevance of events

to query events

  • Queries divided into training (60%) and test sets
slide-26
SLIDE 26

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 6 -

Event Retrieval Expts.

  • 1,000 combinations investigated in training

phase

  • Overall accuracy of top 5 returned documents is

63% … (top 10 is 52%)

  • Overall MAP score of 0.3608
  • Query scores ranging from 0.0057 (Hyowon on

public transport) to 0.9415 (Michael at work on his PC)

slide-27
SLIDE 27

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 7 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-28
SLIDE 28

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 8 -

Importance Reminder

Event Segmentation Novelty Calculation of Each Event

0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9

Event-Event Comparison within the Multi-day Event database Event database containing last 7 days’ Events

Landmark Image Selection

SenseCam Images of a day (about 3,000)

Day -1 Day -2 Day -5 Day -3 Day -4 Day -6

Composition of the Browser

2 Sept 06

Interactive Browser

slide-29
SLIDE 29

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 2 9 -

Importance

  • Greater emphasis is

placed on important events

  • Routine/mundane

events can be hidden

slide-30
SLIDE 30

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 0 -

Automatic Face Detection

  • Trained on set of

1,758 SenseCam images

  • SenseCam

images are low quality

  • Accuracy = 63%
slide-31
SLIDE 31

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 1 -

Novelty to Detect Event Importance

Mon Tue Wed Thr Fri Sat Sun Unique Events Mon Similar Events - Aiden waiting for bus Similar Events - Aiden at the office corridor Similar Events - Aiden working on the desk

  • Find the most dissimilar event of today by

taking the previous 6 days into account.

  • Also for any event, we only look at how novel

it is with respect to events around the same time from other days

slide-32
SLIDE 32

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 2 -

Event Importance Expts.

  • How well does it work?
  • Recently completed extensive experiments

with 3 different users wearing SenseCam for 4 weeks each (total = 176k images)

  • 83 days of data collected in total, with 8

different approaches evaluated … giving 664 judgements

slide-33
SLIDE 33

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 3 -

Importance Evaluation App.

slide-34
SLIDE 34

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 4 -

Event Importance Expts.

  • 3 final approaches evaluated:

– Face Detection Only (current state of art) – Novelty Only – Face Detection + Novelty

  • Face Detection + Novelty performs at least as

well as state of art 80% of the time, and 4% better overall

  • Face Detection good at highlighting most

important events

  • Novelty good at detecting routine events
slide-35
SLIDE 35

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 5 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-36
SLIDE 36

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 6 -

System Demo

slide-37
SLIDE 37

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 7 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-38
SLIDE 38

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 8 -

Daily Browser Overview

Event Segmentation Novelty Calculation of Each Event

0.1 0.7 0.1 0.1 0.3 0.4 0.8 0.1 0.9

Composition of the Browser

2 Sept 06

Interactive Browser

Day -1 Day -2 Day -5 Day -3 Day -4 Day -6

Event-Event Comparison within the Multi-day Event database Event database containing last 7 days’ Events

Landmark Image Selection

SenseCam Images of a day (about 3,000)

Event Augmentation

slide-39
SLIDE 39

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 3 9 -

Event Augmentation

  • Augment low-quality SenseCam images with

high quality images from external sources

Number of GeoTagged Photos on Flickr

5,000,000 10,000,000 15,000,000 20,000,000 25,000,000 30,000,000 35,000,000 40,000,000 01/08/2006 01/10/2006 01/12/2006 01/02/2007 01/04/2007 01/06/2007 01/08/2007 01/10/2007 01/12/2007

slide-40
SLIDE 40

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 0 -

Event Augmentation – Croke Park

Here’s an image from a SenseCam after a big match in Croke Park, Dublin. We’d really like to see other people’s pictures of this match. Let’s search by location…

slide-41
SLIDE 41

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 1 -

Event augmentation – Croke Park

  • Receive the following pictures…
  • Then filter out to just those results from the same day
slide-42
SLIDE 42

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 2 -

Event augmentation – Santa Barbara

Here’s a SenseCam picture of a building that I like from the pier in Santa Barbara, CA. Again I search for other pictures in the same location…

slide-43
SLIDE 43

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 3 -

Event augmentation – Santa Barbara

  • I receive the following pictures…
  • Then I filter out to just those results that are visually similar
slide-44
SLIDE 44

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 4 -

Event augmentation - Chalkidiki

Here’s an image from my SenseCam at a beach in Chalkidki in

  • Greece. I’d really like to see other people’s pictures of this

beach Therefore I search by location firstly…

slide-45
SLIDE 45

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 5 -

Event augmentation - Chalkidiki

  • I receive the following pictures…
  • Then I filter out to just those visually similar results
slide-46
SLIDE 46

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 6 -

Event augmentation – New York

Here’s an image from my SenseCam looking towards the Statue of Liberty in New York. I’d really like to see other people’s pictures that are similar Therefore I search by location firstly…

slide-47
SLIDE 47

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 7 -

Event augmentation – New York

  • I receive the following pictures…
  • Then I filter out to just those visually similar results
slide-48
SLIDE 48

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 8 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-49
SLIDE 49

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 4 9 -
  • 1. Columbia University, New York
  • 1 week research visit to LabROSA audio processing

group of Prof. Dan Ellis

  • Gathered SenseCam and audio data over a 10 day

period

  • Columbia segmented data into events based on audio
  • We segmented this data into events based on image and

sensor features and also included Columbia audio output

  • Collaborative event segmentation paper accepted in

RIAO conference, Pittsburgh.

slide-50
SLIDE 50

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 0 -
  • 2. UCLA DietSense Project
  • Overview

– To have users wear N95 phones like SenseCam devices

  • Why?

– To allow participants in dietary studies easily audit their diet – To allow health care professionals easily browse and annotate large sets of images

slide-51
SLIDE 51

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 1 -
  • 2. UCLA DietSense Project
slide-52
SLIDE 52

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 2 -
  • 2. UCLA Collaboration
  • UCLA DietSense

– Problem of too many images been taken per day

  • Our SenseCam project

– Segment many images into more manageable events

slide-53
SLIDE 53

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 3 -
  • 2. UCLA Project Goals
  • CENS in UCLA

– Provide us with “Campaignr” software

  • Automatically takes image every 30 seconds
  • Also GPS, Bluetooth, and motion data
  • CDVP in DCU

– Get people to wear N95 phones like UCLA people – Using image processing, provide:

  • A segmentation solution, ala SenseCam event

processing

  • Browser, for UCLA to use
slide-54
SLIDE 54

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 4 -

8:25 AM 9:42 AM 12:11 PM

AFTERNOON AFTERNOON

9:35 AM 8:21 AM 9:53 AM 1:23 PM 2:02 PM 2:45 PM 3:49 PM

MO MORN RNING ING

17 September 2007

Breakfast Lunch Dinner 11 min

(10 Images)

39 min

(14 Images)

1 hr 23 min

(52 Images)

BREAKFAST

9:42 – 9:53am (11 min)

slide-55
SLIDE 55

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 5 -
  • 3. University of Tampere
  • Department of Information Studies of Prof.

Kalervo Järvelin

  • Tampere PhD student shadowing medical

researchers … goal is to learn their task-based information access

  • We provide SenseCam plus event browser tool
  • Collaboration involves analysing the

effectiveness of SenseCam as an ethnographic tool

slide-56
SLIDE 56

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 6 -
  • 3. University of Tampere
slide-57
SLIDE 57

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 7 -

Overview

  • Introduction to LifeLogging
  • Segmentation of Images into Events
  • Retrieval of Similar Events
  • Determining Important Events
  • System Demo
  • Augmenting LifeLog Images
  • Collaborative Work
  • Other Work
  • Conclusions
slide-58
SLIDE 58

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 8 -

Retrieval using Bluetooth + GPS

  • Growth of Bluetooth devices expected to increase from 140

million in 2005 to 583 million by 2009, Gartner Research

  • Unique H/W address of each Bluetooth device means

people can be associated to a particular device e.g. 00:00:3A:69:89:A8 = Barry

slide-59
SLIDE 59

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 5 9 -

Retrieval using Bluetooth + GPS

slide-60
SLIDE 60

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 0 -

Experimental set-up – data collected

  • 1 user; 28,000 images; 350 events; 28 days
  • Images segmented into events using image and sensor features

as detailed earlier

  • 10 events randomly selected from dataset
  • Judgments made by SenseCam wearer on top 10 returned results

by each system

slide-61
SLIDE 61

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 1 -

Retrieval using Bluetooth + GPS

  • Bluetooth and GPS are very encouraging

information sources to retrieve similar Lifelogging events

  • In future it will be necessary to combine

these sources with other existing sources

  • f information content
slide-62
SLIDE 62

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 2 -

Summer Project - Posture Monitoring

  • When to record posture?
  • Identify optimal (combination
  • f) SenseCam sensors to

classify SVM into sitting/not sitting

  • Optimal sensor was

accelerometer Y-axis.

slide-63
SLIDE 63

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 3 -

To reduce empty space: packing algorithm...

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 10 9 10 11 11 12 13

Summer Project – Manga Interface

slide-64
SLIDE 64

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 4 -

Conclusions

  • Introduction to the concept of

Lifelogging

  • Past research in the field was

predominantly hardware+storage based

  • Only now considering the retrieval

challenges

slide-65
SLIDE 65

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 5 -

Conclusions

  • Extensive work complete in segmenting

images into distinct events

  • Retrieval of similar events and highlighting

important events done

  • Future Work: Augmenting FlickR images

… and so much more (biometrics, bluetooth, etc.)!

slide-66
SLIDE 66

DUBLIN CITY UNIVERSITY CENTRE FOR DIGITAL VIDEO PROCESSING ADAPTIVE INFORMATION CLUSTER

  • 6 6 -

Thank You

further information:

http://www.computing.dcu.ie/~adoherty http://www.cdvp.dcu.ie/SenseCam