Social-media Storytelling Linking Hao Wu Seamus Lawless Gareth - - PowerPoint PPT Presentation

▶
social media storytelling linking
SMART_READER_LITE
LIVE PREVIEW

Social-media Storytelling Linking Hao Wu Seamus Lawless Gareth - - PowerPoint PPT Presentation

Sep 21, 2023 381 likes •653 views

Social-media Storytelling Linking Hao Wu Seamus Lawless Gareth Jones Francois Pitie The ADAPT Centre is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund. Task

slide-1
SLIDE 1

Social-media Storytelling Linking

Hao Wu Seamus Lawless Gareth Jones Francois Pitie

The ADAPT Centre is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.

slide-2
SLIDE 2

www.adaptcentre.ie

  • Task definition
  • Challenges & Solutions
  • Training
  • Searching
  • Result
slide-3
SLIDE 3

www.adaptcentre.ie

slide-4
SLIDE 4

www.adaptcentre.ie

Tour France

slide-5
SLIDE 5

www.adaptcentre.ie

slide-6
SLIDE 6

www.adaptcentre.ie

Challenges & Solutions

slide-7
SLIDE 7

www.adaptcentre.ie

Lack of training data Video can’t be concluded by

  • nly one sentences.

Challenges

slide-8
SLIDE 8

www.adaptcentre.ie

Solutions

Pre-train + Fine tuning Video segmentation + Length normalization

slide-9
SLIDE 9

www.adaptcentre.ie

Data pre-processing

slide-10
SLIDE 10

www.adaptcentre.ie

Images Videos Queries Edinburgh Festival 32k 6.2k 60 Le Tour de France 66k 19k 58

slide-11
SLIDE 11

www.adaptcentre.ie

Shot boundary detection Resnet-152

Video Image Image sets Visual embeddings Text Text representation

Word level + Sentence level (Skip-Thought)

slide-12
SLIDE 12

www.adaptcentre.ie

Model

  • verview
slide-13
SLIDE 13

www.adaptcentre.ie

slide-14
SLIDE 14

www.adaptcentre.ie

Training

slide-15
SLIDE 15

www.adaptcentre.ie

Snow Playful dogs People having meal Deep time Show Museum of Edinburgh Highlights of Chris Froome Pre-training Target information

Examples

slide-16
SLIDE 16

www.adaptcentre.ie

Pre-training

Introducing Flickr30k (High quality “image”- “text” pairs) A boy in a dark shirt is reading a book while sitting on a piano bench

slide-17
SLIDE 17

www.adaptcentre.ie

Target information collecting

Collecting from source domain:

  • Identify keywords from query file.
  • Match keywords with data in the source.

E.g. Keyword: taking selfies.

Collecting from search engine:

  • Collect labels from online image search engine

(Google and Bing) using story segments + event name as query. Model

slide-18
SLIDE 18

www.adaptcentre.ie

Snow Chris Froome pedaling

slide-19
SLIDE 19

www.adaptcentre.ie

Searching

slide-20
SLIDE 20

www.adaptcentre.ie

Search

Trade-off between consistency and accuracy

𝑆𝑢 = 0.2*𝑆t−1 + 0.8 * 𝑁𝑢

(M is the model raw output, R is the modified output)

slide-21
SLIDE 21

www.adaptcentre.ie

Search

λ used in penalizing long videos; L denotes number of segments; Sig() is sigmoid function. There are 5 runs submitted. The main difference is the value of λ:

Conf Run1 Run2 Run3 Run4 Run5 λ 3 5 12 20 50 Source Google+ Bing Google Google Google Google

slide-22
SLIDE 22

www.adaptcentre.ie

Results

slide-23
SLIDE 23

www.adaptcentre.ie

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Run1 Run2 Run3 Run4 Run5

Summary Quality

Edfest Tourfrance

slide-24
SLIDE 24

www.adaptcentre.ie

Conclusion & Future Work

Target specific information are crucial. Improve video representations by applying key frame selection (or building sequence model). Build a classifier to filter crawled images to make this process automatic.

slide-25
SLIDE 25

Thanks for listening.

The ADAPT Centre is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.