Sketch-Based Deep Learning Papers

A resource repository for sketch based deep learning papers and work.

Table of Contents

• Dataset
• Colorization
• Stylization
• 3D Representation
• Realistic
• Others

Dataset

Blending Texture Features from Multiple Reference Images for Style Transfer (SIGGRAPH Asia Tech Briefs 2016) [Paper] Nico Open data [Link]

The Sketchy Database: Learning to Retrieve Badly Drawn Bunnies (TOG, SIGGRAPH 2016) [Link]

Quick, Draw! Dataset (2017) [Link]

SketchyScene: Richly-Annotated Scene Sketches (ECCV 2018) [Link]

Universal Sketch Perceptual Grouping (ECCV 2018) [Paper]

OpenSketch: A Richly-Annotated Dataset of Product Design Sketches (TOG, SIGGRAPH Asia 2019) [Link]

EmoG: Supporting the Sketching of Emotional Expressions for Storyboarding (CHI on Human Factors in Computing Systems, 2020) [Link]

Building a Manga Dataset "Manga109" with Annotations for Multimedia Applications (IEEE MultiMedia, 2020) [Paper] [Link]

SPARE3D: A Dataset for SPAtial REasoning on Three-View Line Drawings (CVPR, 2020) [Paper]

Colorization

Paintschainer (2017) [Link]

Outline colorization through tandem adversarial networks (2017) [Paper]

Comicolorization: semi-automatic manga colorization (SIGGRAPH Aisa 2017 Tech Briefs) [Paper]

Two-stage sketch colorization (TOG, SIGGRAPH Asia 2018) [Paper]

User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks (ACM Multimedia, 2018) [Paper]

Tag2pix: Line art colorization using text tag with secat and changing loss (ICCV 2019) [Paper]

Automatic Colorization of Anime Sketch Image [Link]

Language-based Colorization of Scene Sketches (TOG, SIGGRAPH Asia 2019) [Link]

Deep Line Art Video Colorization with a Few References (2020) [Paper]

Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence (CVPR, 2020) [Paper]

Stylization

Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup (TOG, SIGGRAPH 2016) [Link]

Deep extraction of manga structural lines (TOG, SIGGRAPH 2017) [Link]

A Neural Representation of Sketch Drawings (ICLR 2018) [Paper]

Mastering sketching: adversarial augmentation for struc- tured prediction (TOG, SIGGRAPH 2018) [Link]

Real-Time Data-Driven Interactive Rough Sketch Inking (TOG, SIGGRAPH 2018) [Link]

Im2pencil: Controllable pencil illustra- tion from photographs (CVPR 2019) [Paper]

Adversarial Colorization Of Icons Based On Structure And Color Conditions (2019) [Paper]

Learning to Shadow Hand-drawn Sketches (CVPR 2020) [Link]

Neural Contours: Learning to Draw Lines from 3D Shapes (CVPR 2020) [Paper]

Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt (CVPR, 2020) [Paper]

Sketch to 3D Representation

Sketch to Normal Maps. Sketch to 3D Model.

Normal Maps

Deep Normal Estimation for Automatic Shading of Hand-Drawn Characters (ECCV Workshop 2018) [Link]

Interactive Sketch-Based Normal Map Generation with Deep Neural Networks (I3D 2018) [Paper]

Novel View

Lifting Freehand Concept Sketches into 3D (SIGGRAPH Asia, 2020) [Link]

3D Model

BendSketch: Modeling Freeform Surfaces Through 2D Sketching (TOG, SIGGRAPH 2017) [Paper]

DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling (TOG, SIGGRAPH 2017) [Paper]

3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks (3DV 2017) [Paper]

3D Sketching using Multi-View Deep Volumetric Prediction (I3D 2018) [Paper]

Contour-based 3D Modeling through Joint Embedding of Shapes and Contours (I3D 2020) [Paper]

Realistic

SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis (CVPR 2018) [Paper]

Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition (2018) [Paper]

Learning to Sketch with Shortcut Cycle Consistency (CVPR 2018) [Paper]

Sketch-a-Classifier: Sketch-based Photo Classifier Generation (CVPR 2018) [Paper]

Learning Deep Sketch Abstraction (CVPR 2018) [Paper]

SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval (CVPR 2018) [Paper]

Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation (ICCV 2019) [Link]

Deep Zero-Shot Learning for Scene Sketch (ICIP 2019) [Paper]

Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval (CVPR 2019) [Paper]

Generalising Fine-Grained Sketch-Based Image Retrieval (CVPR 2019) [Paper]

SketchyCOCO: Image Generation from Freehand Scene Sketches (CVPR, 2020) [Paper]

BézierSketch: A generative model for scalable vector sketches (ECCV, 2020) [Paper]

Pixelor: A Competitive Sketching AI Agent. So you think you can beat me? (SIGGRAPH Asia, 2020) [Link]

Others

(Non DL work, but resourceful)

A Survey on Sketch Based Content Creation: from the Desktop to Virtual and Augmented Reality (Eurographics, 2020) [Paper]

Why Do Line Drawings Work? A Realism Hypothesis (Perception 2020) [Paper]

Aaron Hertzmann's Blog [Link]

• Why Edge Detection Doesn’t Explain Line Drawing [Link]

• Apparent Ridges, Edges, and Perception of Line Drawing [Link]

SketchX Lab [Link]

SYNLA Dataset (Synthetic Line Art Dataset) [Link]

Disclaimer

This is a personal collection. No warranty. Working in progress, pull request welcome. ❤️