DEEP LEARNING applications Julia Rabetti Giannella Research - - PowerPoint PPT Presentation

DEEP LEARNING

applications

Julia Rabetti Giannella Research assistant at VISGRAF Lab PhD in Design and Technology (PPDESDI-UERJ) juliagiannella@gmail.com

APPLICATIONS

• Colorization of Black and White Images
• Adding Sounds To Silent Movies
• Object Classification in Photographs
• Automatic Handwriting Generation
• Character Text Generation.
• Image Caption Generation.
• Automatic Game Playing
• Artistic style transfer

Source: http://machinelearningmastery.com/inspirational-applications-deep-learning/

1) Colorization of Black and White Images

• problem of adding color to black and white photographs
• traditionally, this was done by hand with human effort
• CV task attacked by different approaches
• topic of relative importance in SIGGRAPH and EUROGRAPH
• DL approach involves the use of very large CNN 


and supervised layers that recreate the image with 
 the addition of color

Paper Colorful Image Colorization (ECCV, 2016)

Source: http://richzhang.github.io/colorization/

Network architecture

Source: https://arxiv.org/pdf/1603.08511.pdf

Semantic interpretability of results

Source: http://richzhang.github.io/colorization/

[Algorithmia] Demo

Source: http://demos.algorithmia.com/colorize-photos/

Dana Keller - designer and photo colorizer

Source: https://www.youtube.com/watch?v=bYHnWhZkAIc Source: http://www.danarkeller.com/about/

Comparing

Keller Algorithmia

Comparing

Keller Algorithmia

Comparing

Keller Algorithmia

Comparing

Keller Algorithmia

Comparing

Keller Algorithmia

Comparing

Keller Algorithmia

2) Object Classification in Photographs

• task requires the classification of objects within a

photograph as one of a set of previously known objects

• State-of-the-art results have been achieved on benchmark

examples of this problem using very large CNN

• derives from image classification task
• breakthrough: ImageNet Classification with Deep

Convolutional Neural Networks (Krizhevsky et al., 2012)

• AlexNet won ILSVRC-2012 challenge

Source: http://www.cs.toronto.edu/~fritz/absps/imagenet.pdf

Classification with localization

• more complex variation
• f this task involves

specifically identifying

• ne or more objects

within the scene of the photograph and drawing a box around them

• GoogLeNet won

ILSVRC-2014 challenge in this task

Source: https://research.googleblog.com/2014/09/building-deeper-understanding-of-images.html

2.1) DL and RIO2016

• VISGRAF project (out 2016)
• task: automatically classify and cluster images by subject

features related to the Olympic Games, Olympic Torch

• CNN model and supervised learning
• TensorFlow (open source software library)
• Inception-v3 (Going Deeper with Convolutions, 2015)
• transfer learning (manually labeled 100 examples)

Source: http://lvelho.impa.br/dl_rio2016/index.html Source: https://arxiv.org/abs/1409.4842

Confidence score

Source: http://lvelho.impa.br/dl_rio2016/metodologia.html

A subset of 12 from 2091 images with confidence score over 83% for the Olympic torch category

Torch Mosaic

Source: http://lvelho.impa.br/dl_rio2016/mosaico.html

Torch Mosaic

Source: http://lvelho.impa.br/dl_rio2016/mosaico.html

2.2) Twitter Facial Analysis Reveals Demographics

• f Presidential Campaign Followers
• (Mit Technology Review, march 2016)
• IN: Conference on Web and Social Media
• understand follower demographics of Trump and Clinton by

crossing Twitter metadata and facial features

• a CNN model on followers’ profile images extracts

information on gender, race and age

Source: https://www.technologyreview.com/s/601074/twitter-facial-analysis-reveals-demographics-of-presidential- campaign-followers/?utm_campaign=add_this&utm_source=email&utm_medium=post Source: https://arxiv.org/abs/1603.03097

A Comparison of the Trumpists and Clintonists

Source: https://arxiv.org/abs/1603.03097

C"lintonists"
 in the Twitter Sphere

2.3) NVIDIA DRIVENet Demo - Visualizing 
 a Self-Driving Car

Source: https://www.youtube.com/watch?v=HJ58dbd5g8g

3) Artistic style transfer

• task: separate and recombine content and style of arbitrary

images, providing a neural algorithm for the creation of artistic images

• A Neural Algorithm of Artistic Style (Gatys et al., 2015)

Source: https://arxiv.org/abs/1508.06576

Convolutional Neural Network (CNN)

Source: https://arxiv.org/abs/1508.06576

An example

Source: https://research.googleblog.com/2016/02/exploring-intersection-of-art-and.html

The style transfer algorithm crosses a photo with a painting style; for example Neil deGrasse Tyson in the style of Kadinsky’s Jane Rouge Bleu. Photo by Guillaume Piolle, used with permission.

3.1) DeepDream

Source: http://deepdreamgenerator.com/ Source: https://en.wikipedia.org/wiki/DeepDream

• computer vision program created by Google
• given an input image returns a version with h"allucinogenic"

appearance

• originates in a CNN codenamed Inception after the film of

the same name developed for the ILSVRC-2014

• CNN can also be run in reverse, to do synthesis
• enhance faces and certain animals -> pareidolia results

3.1) DeepDream

Source: http://deepdreamgenerator.com/ Source: https://en.wikipedia.org/wiki/DeepDream

3.2) Prisma App

Source: http://prisma-ai.com/ Source: https://en.wikipedia.org/wiki/Prisma_(app)

• photo-editing application that utilizes a neural network and

to transform the image into an artistic effect

• became popular on July 2016
• created by Alexey Moiseenkov
• reference A Neural Algorithm of Artistic Style (2016)

3.2) Prisma App

3.2) Prisma App

3.3) Artistic style transfer (video)

Source: https://arxiv.org/abs/1604.08610 Source: https://www.youtube.com/watch?v=Khuj4ASldmU

• Artistic style transfer for videos (Ruder et al.,2016)

3.4) Supercharging Style Transfer for video

Source: https://arxiv.org/abs/1610.07629 Source: https://research.googleblog.com/2016/10/supercharging-style-transfer.html

• A Learned Representation For Artistic Style (Dumoulin et al.,

2016)

• CNN that learns multiple styles at the same time
• method enables style interpolation

3.4) Supercharging Style Transfer for video

Source: https://www.youtube.com/watch?v=6ZHiARZmiUI