Fixing the image problem of the web using machine learning Chris - PowerPoint PPT Presentation

Fixing the image problem of the web using machine learning Chris Heilmann @codepo8, GotoCon Copenhagen, October 2016

Of innovation and impatience CHRIS HEILMANN @CODEPO8 Chris Heilmann @codepo8, Future Decoded, London, Nov 2015

We all know this character, right? https://en.wikipedia.org/wiki/Mario#Concept_and_creation

But do you know why it looks like it does? https://en.wikipedia.org/wiki/Mario#Concept_and_creation

Red and Blue offered the best contrast to the skin, boots and the game background. https://en.wikipedia.org/wiki/Mario#Concept_and_creation

The cap meant there was no need to worry about hair style, eyebrows and forehead. (There were also not enough pixels for waving hair when falling down a hole)

The large nose and moustache made it possible to avoid a mouth and facial expressions.

 Design by limitations.

⚙ Everything has its reasons and meaning.

Evolution is happening around us… 🖦 → 💼 → 📲 …and user numbers are shifting.

📲 T echnology advanced

📹 Pixels are a side-product of our interactions on 📺 the web.

So much, that they become a nuisance… Type Size (kB) Images 1426 Scripts 357 Video 174 Fonts 123 Stylesheets 76 HTML 67 Other 4 Total 2232 kB http://www.httparchive.org/interesting.php#bytesperpage

🖍 Inspirational obesity 🐙

• Wrong file formats • Delivering scaled hi-res images to everybody 1.4 MB images… • No automatic conversion and optimisation steps • Hero image instead of text content

🚁 We need to change that to make the web fast again…

The web is much bigger than our little developer world and growth happens outside of it.

Surgical solutions: Proxy Browsers and Cloud Services https://www.flickr.com/photos/89306448@N00/334479803 {Guerrilla Futures | Jason Tester}

 Here are a few things you can do.

• Huge images for everybody • Unoptimised images The problems: • No alternative content • No training or incentive to add content in CMS

• Better browsers with responsive image support • Automated, loss-less image optimisation tools • File level access to images to extract metadata Our arsenal: • Scripting solutions to offer alternative content • Cloud services with machine learning APIs for intelligent resizing • Machine learning for tagging

Browsers with responsive image support…

https://www.christianheilmann.com/ Media queries are greedy! 2012/12/19/conditional-loading-of- resources-with-mediaqueries/

🖽 <picture> element and srcset

Support is great!

More info… https://jakearchibald.com/2015/ anatomy-of-responsive-images/

Live demo… https://dev.windows.com/en-us/microsoft- edge/testdrive/demos/picture/

Automated tools for lossless image optimisation

ImageOptim https://imageoptim.com/ PNGOUT, Zopfli, Pngcrush, AdvPNG, extended OptiPNG, JpegOptim, MozJPEG ( jpegtran & jpegrescan), Gifsicle.

File-level access to embedded information in images

EXIF - hidden gems http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/

EXIF - hidden gems http://code.flickr.net/2012/06/01/parsing- exif-client-side-using-javascript-2/

http://code.flickr.net/2012/06/01/parsing- EXIF - THE HIDDEN GEMS exif-client-side-using-javascript-2/

Removing EXIF data… http://removephotodata.com/

Remove locations… http://removephotodata.com/image-to-map.html

How about providing fallback content? http://www.datapointed.net/2010/01/crayola-crayon-color-chart/

Blur-up technique https://css-tricks.com/the-blur-up-technique- for-loading-background-images/

Counting pixels… https://codepo8.github.io/canvas-images-and-pixels/ http://colorify.rocks/index.html

Counting pixels… https://codepo8.github.io/canvas-images-and-pixels/

Colorify.js http://colorify.rocks

Color thief… http://lokeshdhakar.com/projects/color-thief/ https://github.com/jwagner/smartcrop.js/

SCRIPTING SOLUTIONS FOR Intelligent image resizing ALTERNATIVE CONTENT

Smartcrop.js https://github.com/jwagner/smartcrop.js/

Intelligent resizing http://cloudinary.com/blog/ automatically_art_directed_responsive_images

Intelligent resizing http://cloudinary.com/blog/ automatically_art_directed_responsive_images

High contrast tricks https://blog.imgix.com/2015/10/21/automatic- point-of-interest-cropping-with-imgix.html

🔯 What about information that isn’t in the image?

Machine learning and artificial intelligence to the rescue

Automated alternative text (Facebook) Image may contain: dog, outdoor and nature

How? Image may contain: dog, outdoor and nature https://code.facebook.com/posts/561187904071636

Making photos findable… https://photos.google.com

Learning from good data… http://image-net.org/

Learning from *lots* of data… http://image-net.org/

Learning from *lots* of data… https://github.com/ openimages/dataset

Learning from *lots* of data… https://github.com/ openimages/dataset

Refining the data with language https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html

Compare and enhance…

Detect syntax and mix and match

Add visual information

Adding automated content in context… https://research.googleblog.com/2016/05/aw-so-cute-allo-helps-you-respond-to.html

APIs and demos to use yourself… http://captionbot.ai

 Detecting humans…

Detect Faces… Detection Result: JSON: [ { "faceRectangle": { "width": 109, "height": 109, "left": 62, "top": 62 }, "attributes": { "age": 31, "gender": "male", "headPose": { "roll": "2.9", "yaw": "-1.3", "pitch": "0.0" } "faceLandmarks": { "pupilLeft": { "x": "93.6", "y": "88.2" }, "pupilRight": { "x": "138.4", "y": "91.7" }, ...

Verify Faces Verification Result: JSON: [ { "isIdentical":false, "confidence":0.01 } ]

Cluster automatically…

🦅 Empowering people…

https://www.youtube.com/watch?v=R2mC-NUAmMk

CHRIS HEILMANN THANKS! @CODEPO8 CHRISTIANHEILMANN.COM