language based colorization of scene sketches
play

Language-based Colorization of Scene Sketches Changqing Zou* 1,2 , - PowerPoint PPT Presentation

Mar 17, 2024 •878 likes •1.52k views

Language-based Colorization of Scene Sketches Changqing Zou* 1,2 , Haoran Mo* 1 , Chengying Gao 1 , Ruofei Du 3 , Hongbo Fu 4 Sun Yat-sen University 1 Huawei Noahs Ark Lab 2 Google 3 City University of Hong Kong 4 Nov. 20 th , 2019

  1. Language-based Colorization of Scene Sketches Changqing Zou* 1,2 , Haoran Mo* 1 , Chengying Gao 1 , Ruofei Du 3 , Hongbo Fu 4 Sun Yat-sen University 1 Huawei Noah’s Ark Lab 2 Google 3 City University of Hong Kong 4 Nov. 20 th , 2019 SA2019.SIGGRAPH.ORG CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  2. Motivation: Abstract Data and Human Cognition • Sparse • Highly abstract SA2019.SIGGRAPH.ORG 1 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  3. Motivation: Abstract Data Understanding • Lots of early exploration with computational models [Eitz et. al 2012, Li et. al 2013, Schneider et. al 2014, Li et. al 2015] • Limited ability of understanding object-level sketches [1] M. Eitz, J. Hays, and M. Alexa. How do humans sketch objects? In SIGGRAPH, 2012. [2] Y. Li, Y. Song, and S. Gong. Sketch recognition by ensemble matching of structured features. In BMVC, 2013. [3] R. G. Schneider and T. Tuytelaars. Sketch classification and classification-driven analysis using fisher vectors. In SIGGRAPH Asia, 2014. [4] Y. Li, T. M. Hospedales, Y. Song, and S. Gong. Free-hand sketch recognition by multikernel feature learning. CVIU, 2015. SA2019.SIGGRAPH.ORG 2 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  4. Motivation: Sketch Understanding TU-Berlin / Sketchy / QuickDraw SA2019.SIGGRAPH.ORG 3 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  5. Motivation: Sketch Understanding vs SketchyScene [Zou et. al 2018] Scene-level sketch: • Interaction among multiply objects • More empty region, lack of contextual information [1] Changqing Zou, et. al. SketchyScene: Richly-Annotated Scene Sketches. In ECCV, 2018. SA2019.SIGGRAPH.ORG 4 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  6. Motivation: Scene Sketch Understanding Scene sketch colorization Natural language SA2019.SIGGRAPH.ORG 5 6 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  7. Motivation: Why Language-based? • Natural: easily adopted by novice users • Touchless: friendly for people with upper limb impairments • Effective: support batch-processing colorization “the bus is yellow with blue windows” SA2019.SIGGRAPH.ORG 7 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  8. Motivation: Language-based Sketch Colorization • Toy problem, but not simple …… SA2019.SIGGRAPH.ORG 8 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  9. Challenges A. Understanding scene-level sketch is very hard • Too abstract • Lack of contextual information CMPlaces Photo-Sketching SketchyScene [Castrejon et. al 2016] [Li et. al 2019] [Zou et. al 2018] SA2019.SIGGRAPH.ORG 9 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  10. Challenges B. Multimodal learning between language and scene sketch • Mapping between language and target objects “the dog on the rightmost has orange body” SA2019.SIGGRAPH.ORG 10 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  11. Challenges B. Multimodal learning between language and scene sketch • One or multiple objects with single instruction “the two trees on the left of the house are light green” SA2019.SIGGRAPH.ORG 10 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  12. Challenges B. Multimodal learning between language and scene sketch • Various free expressions of location “the dog in the middle is gray ” / “the dog near the house is gray” SA2019.SIGGRAPH.ORG 10 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  13. Challenges C. Multimodal learning between language and object sketch • Object-part-level colorization • Various free expressions of colors “the bus is dark (navy/…) blue with white windows ” SA2019.SIGGRAPH.ORG 11 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  14. Inspiration: Drawing and Intelligence Development • Sensitive to line drawing and color • Mode of thinking and creation SA2019.SIGGRAPH.ORG 12 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  15. Inspiration: Language and Literacy Development • Embedding voice in traditional drawings supports children’s literacy development [Raffle et. al 2007] SA2019.SIGGRAPH.ORG 13 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  16. Related Work SA2019.SIGGRAPH.ORG CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  17. Related Work A. Language-based Image Segmentation • Fusion of textual and visual information • Only natural images • Only one binary mask for single or multiple target objects “ second vase from right ” “ the bottom two luggage cases being rolled ” [Ye et. al 2019] Our work SA2019.SIGGRAPH.ORG 14 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  18. Related Work B. Language-based Image Colorization • Language-based image editing (LBIE) [Chen et. al 2018] • Require pair-wise scene-level sketch and color image “ The flower has red petals with yellow stigmas in the middle ” Scene sketch [Chen et. al 2018] Our work [1] Jianbo Chen, et. al. Language-Based Image Editing With Recurrent Attentive Models. In CVPR, 2018. SA2019.SIGGRAPH.ORG 15 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  19. Our Work SA2019.SIGGRAPH.ORG CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  20. Main contributions • Language-based colorization system for scene sketches • Language-based instance segmentation network for scene sketches • Three large-scale datasets for language-based scene sketch colorization SA2019.SIGGRAPH.ORG 16 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  21. Proposed approach A. System pipeline • Divide-and-conquer and progressive strategy • Two modes (foreground and background) • Three models (instance matching, foreground colorization, background colorization) System pipeline SA2019.SIGGRAPH.ORG 17 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  22. Proposed approach B.1 Instance Matching Model • Training: two phases for binary mask (b) generation • Inferring: fuse binary mask with instance segmentation results Foreground mode SA2019.SIGGRAPH.ORG 18 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  23. Proposed approach B.2 Foreground Colorization Model • GAN + fusion module • Colorize objects from different categories Foreground mode SA2019.SIGGRAPH.ORG 20 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  24. Proposed approach B.3 Background Colorization Model • cGAN + two-branch decoder • Colorization branch • Explicit segmentation branch (segmentation loss) Background mode SA2019.SIGGRAPH.ORG 21 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  25. Datasets • MATCHING dataset : 38k groups of text-based instance segmentation data. SA2019.SIGGRAPH.ORG 22 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  26. Datasets • FOREGROUND dataset : 4k groups of text-based sketch object colorization data. SA2019.SIGGRAPH.ORG 22 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  27. Datasets • BACKGROUND dataset : 20k groups of text-based background colorization data. SA2019.SIGGRAPH.ORG 22 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  28. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Single object “the bus is yellow with blue windows” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  29. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Single object “the bus is yellow with blue windows” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  30. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Multiple objects “ all the trees are dark green” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

  31. Results: Un-targeted A. Un-targeted colorization experiment • Colorize a sketch with free instructions • Multiple objects “ all the trees are dark green” SA2019.SIGGRAPH.ORG 23 CONFERENCE 17-20 November 2019 - EXHIBITION 18-20 November 2019 - BCEC , Brisbane, AUSTRALIA

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Colorization using Optimization Anat Levin Dani Lischinski Yair Weiss Colorization Colorization

Colorization using Optimization Anat Levin Dani Lischinski Yair Weiss Colorization Colorization

Colorization using Optimization Anat Levin Dani Lischinski Yair Weiss Colorization Colorization : a computer-assisted process of adding color to a monochrome image or ) movie. (Invented by Wilson Markle, 1970 Motivation Colorizing black

366 views • 20 slides
Automatic Colorization Gustav Larsson TTI Chicago / University of Chicago Joint work with

Automatic Colorization Gustav Larsson TTI Chicago / University of Chicago Joint work with

Automatic Colorization Gustav Larsson TTI Chicago / University of Chicago Joint work with Michael Maire and Greg Shakhnarovich NVIDIA @ SIGGRAPH 2016 Colorization Let us first define colorization Colorization Definition 1: The inverse

693 views • 50 slides
Learning Representations for Automatic Colorization Gustav Larsson, Michael Maire, Greg

Learning Representations for Automatic Colorization Gustav Larsson, Michael Maire, Greg

Learning Representations for Automatic Colorization Gustav Larsson, Michael Maire, Greg Shakhnarovich TTI Chicago / University of Chicago ECCV 2016 Colorization Let us first define colorization Colorization Definition 1: The inverse of

778 views • 65 slides
DRAFTING THE UNIVERSAL LANGUAGE SKETCHING Rough Sketches are the most common recording method.

DRAFTING THE UNIVERSAL LANGUAGE SKETCHING Rough Sketches are the most common recording method.

DRAFTING THE UNIVERSAL LANGUAGE SKETCHING Rough Sketches are the most common recording method. The term rough describes the state of the design ideas. It suggests that the designs are incomplete and unrefined. Refined Sketches are

329 views • 12 slides
Scene Graphs Scene Representation How does one describe the objects in a 3D scene? Scene

Scene Graphs Scene Representation How does one describe the objects in a 3D scene? Scene

Scene Graphs Scene Representation How does one describe the objects in a 3D scene? Scene Graphs 1 Scene Modeling Languages / API From the low level to the high level State Machine Model -- OpenGL Graph Based Scene Languages

483 views • 17 slides
COLORIZATION USING KNET Jeffrey Lu and Kevin Liu 6.338/18.337 Fall 2017 MOTIVATION Image

COLORIZATION USING KNET Jeffrey Lu and Kevin Liu 6.338/18.337 Fall 2017 MOTIVATION Image

JULIA IMAGE COLORIZATION USING KNET Jeffrey Lu and Kevin Liu 6.338/18.337 Fall 2017 MOTIVATION Image colorization for 6.869 Computer Vision (Jeffrey) Cool application of deep learning Restoring old black and white photos Abstract

390 views • 10 slides
2004: Poisson Matting 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: The

2004: Poisson Matting 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: The

2004: Poisson Matting 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: Flash/No-Flash 2004: The non-photorealistic camera 2004: Colorization 2004: Colorization 2004: Colorization 2004: Colorization 2004: Colorization 2005: Color2Gray

626 views • 30 slides
Physical Sketches CPSC 581 - Fall 2015 Motivation Experience your sketches in a more physical

Physical Sketches CPSC 581 - Fall 2015 Motivation Experience your sketches in a more physical

Physical Sketches CPSC 581 - Fall 2015 Motivation Experience your sketches in a more physical form Test your interaction design with potential end-users Better insights by exploring in 3D! Architecture scale model Industrial Design

417 views • 27 slides
Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a

Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a

Image-Based Rendering and Modeling l Image-based rendering (IBR): A scene is represented as a collection of images l 3D model-based rendering (MBR): A scene is represented by a 3D model plus texture maps l Differences u Many scene details

979 views • 55 slides
Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai

Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai

Moment-Based Quantile Sketches for Efficient Aggregation Queries Edward Gan, Jialin Ding, Kai Sheng Tai, Vatsal Sharan, Peter Bailis Stanford University 1 Motivation: Monitoring production data streams Billions of events / day of mobile app

245 views • 23 slides
Episode 42: I Made Slides 10 February 2019 The Three-Act, Seven Scene Structure Act I:

Episode 42: I Made Slides 10 February 2019 The Three-Act, Seven Scene Structure Act I:

Episode 42: I Made Slides 10 February 2019 The Three-Act, Seven Scene Structure Act I: Act III: Scene 1: The Inciting Incident Scene 5: Dark Moment Scene 2: Choice to Engage Scene 6: Climax Act II: Scene 7: Resolution Scene 3: The

217 views • 10 slides
COMPSCI 326 Web Programming Week 09: ER Diagram Sketches Agenda 4:00 4:35 ER Diagram

COMPSCI 326 Web Programming Week 09: ER Diagram Sketches Agenda 4:00 4:35 ER Diagram

COMPSCI 326 Web Programming Week 09: ER Diagram Sketches Agenda 4:00 4:35 ER Diagram Sketches 4:35 5:10 Team Lightning Data Talks 2 minute presentation on: 1. 1-2 sentences overviewing your startups application 2. 1-2

632 views • 22 slides
Lo Local Im Image Pri riors for Automatic Im Image Colorization with Simultaneous Cla lassifi

Lo Local Im Image Pri riors for Automatic Im Image Colorization with Simultaneous Cla lassifi

Le Let th there be Color!: Jo Joint End-to to-end Le Learning of f Global and Lo Local Im Image Pri riors for Automatic Im Image Colorization with Simultaneous Cla lassifi fication Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa

525 views • 50 slides
Bloom Filters, Count Sketches and Adaptive Sketches Rice University Anshumali Shrivastava

Bloom Filters, Count Sketches and Adaptive Sketches Rice University Anshumali Shrivastava

Bloom Filters, Count Sketches and Adaptive Sketches Rice University Anshumali Shrivastava anshumali@rice.edu 29th August 2016 Anshumali Shrivastava (COMP 640) Sketching 29th August 2016 1 / 22 Basics: Universal Hashing Basic tool for

542 views • 33 slides
Learning to Infer Program Sketches Maxwell Nye, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama

Learning to Infer Program Sketches Maxwell Nye, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama

Learning to Infer Program Sketches Maxwell Nye, Luke Hewitt, Josh Tenenbaum, Armando Solar-Lezama Goal: We want to automatically write code from the kinds of specifications humans can easily provide, such as examples or natural language

1.13k views • 23 slides
Volumetric Scene Reconstruction Volumetric Scene Reconstruction Goal Goal from Multiple

Volumetric Scene Reconstruction Volumetric Scene Reconstruction Goal Goal from Multiple

Image Image- -Based Scene Reconstruction Based Scene Reconstruction Volumetric Scene Reconstruction Volumetric Scene Reconstruction Goal Goal from Multiple Views from Multiple Views Automatic construction of photo Automatic

378 views • 21 slides
Preprocessing and Dimensionality Reduction J er emy Fix CentraleSup elec

Preprocessing and Dimensionality Reduction J er emy Fix CentraleSup elec

Datasets Preprocessing Dimensionality reduction Preprocessing and Dimensionality Reduction J er emy Fix CentraleSup elec jeremy.fix@centralesupelec.fr 2017 1 / 73 Datasets Preprocessing Dimensionality reduction Where to get data

1.22k views • 73 slides
Priority Queues & Heaps CS16: Introduction to Data Structures & Algorithms Spring 2020

Priority Queues & Heaps CS16: Introduction to Data Structures & Algorithms Spring 2020

Priority Queues & Heaps CS16: Introduction to Data Structures & Algorithms Spring 2020 Outline Priority Queues Motivation ADT Implementation Heaps insert( ) and upheap( ) removeMin( ) and downheap( )

552 views • 34 slides
RMIT at the NTCIR-13 We Want Web Task Luke Gallagher with Joel Mackenzie, Rodger Benham,

RMIT at the NTCIR-13 We Want Web Task Luke Gallagher with Joel Mackenzie, Rodger Benham,

RMIT at the NTCIR-13 We Want Web Task Luke Gallagher with Joel Mackenzie, Rodger Benham, Ruey-Cheng Chen, Falk Scholer, and J. Shane Culpepper School of Science (Computer Science) RMIT University NTCIR 17 (December 8, 2017) Gallagher,

454 views • 10 slides
Future Welcome me Canyon Hills Coyote promo moting class of 2021 Night Canyon Hills

Future Welcome me Canyon Hills Coyote promo moting class of 2021 Night Canyon Hills

Future Welcome me Canyon Hills Coyote promo moting class of 2021 Night Canyon Hills Administrative Team Principal Todd Finkbiner 8 th Assistant Principal Renee Edwards 7 th Assistant Principal Heather Post Intervention

750 views • 43 slides
Commercial Detection in Heterogeneous Video Streams Using Fused Multi-Modal and Temporal Features

Commercial Detection in Heterogeneous Video Streams Using Fused Multi-Modal and Temporal Features

Commercial Detection in Heterogeneous Video Streams Using Fused Multi-Modal and Temporal Features Masami Mizutani Fujitsu Labs. LTD. Shahram Ebadollahi Columbia University Shih-Fu Chang Columbia University IEEE ICASSP 2005 Philadelphia

638 views • 21 slides
Machine learning from a complexity point of view Artemy Kolchinsky SFI CSSS 2019 1

Machine learning from a complexity point of view Artemy Kolchinsky SFI CSSS 2019 1

Machine learning from a complexity point of view Artemy Kolchinsky SFI CSSS 2019 1 PART I: Overview PART II: Deep nets deep dive What is machine learning? Why do deep nets work so well? Learning in deep nets, in the What are

997 views • 77 slides
IoDDoS The Internet of Distributed Denial of Service Attacks A Case Study of the Mirai

IoDDoS The Internet of Distributed Denial of Service Attacks A Case Study of the Mirai

SPAWAR S YSTEMS C ENTER P ACIFIC IoDDoS The Internet of Distributed Denial of Service Attacks A Case Study of the Mirai Malware and IoT-Based Botnets 24-26 April, 2017 Presented at the 2 nd International Conference Presented by Roger

654 views • 20 slides
L A S T E X P L O I T A T I O N cp / zet / f9a About Us Researchers from TeamT5 Core

L A S T E X P L O I T A T I O N cp / zet / f9a About Us Researchers from TeamT5 Core

L A S T E X P L O I T A T I O N cp / zet / f9a About Us Researchers from TeamT5 Core Developer of ThreatSonar for Linux, macOS, Windows We mainly focus on state of the art techniques of threat actors and how to effectively

1.07k views • 80 slides

More recommend