Awesome-Sketch-Based-Applications

A collection of sketch based applications.

Feel free to create a PR or an issue.

Outline

• 0. Survey
• 1. Sketch Based Image Synthesis
  • 1.1 Automatic Synthesis
  • 1.2 Style-conditioned
  • 1.3 Text-conditioned
• 2. Sketch Based Image Editing
• 3. Sketch Based Image Retrieval (SBIR)
• 4. Sketch Based 3D Shape Retrieval
• 5. Sketch Based 3D Shape Modeling
• 6. Sketch Based Garment Design
• 7. Sketch Completion
• 8. Sketch Restoration, Retargeting and Inpainting
• 9. Sketch and Shading
• 10. Sketch Enhancement
• 11. Sketch-Guided Object Localization
• 12. Sketch-Guided Video Synthesis
• 13. Sketch Recognition
• 14. Sketch Segmentation and Perceptual Grouping
• 15. Sketch Representation Learning
• 16. Sketch and Visual Correspondence
• 17. Sketch Animation/Inbetweening
• 18. Sketch and AR/VR
• 19. Sketch Based Incremental Learning
• 20. Sketch Quality Measurement
• 21. Cloud Augmentation with Sketches
• 22. Sketch and Re-identification
• 23. Sketch-based Salient Object Detection

---

0. Survey

Paper	Source	Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey	TPAMI 2022	[code]

1. Sketch Based Image Synthesis

1.1 Automatic Synthesis

• Natural Image or Object

Paper	Source	Code/Project Link
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis	CVPR 2018	[code]
Image Generation from Sketch Constraint Using Contextual GAN	ECCV 2018	[code] [project]
Multi-Instance Sketch to Image Synthesis With Progressive Generative Adversarial Networks	IEEE Access 2019
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation	ICCV 2019	[code] [project]
SketchyCOCO: Image Generation from Freehand Scene Sketches	CVPR 2020	[code]
Unsupervised Sketch-to-Photo Synthesis	ECCV 2020	[code] [project]
Sketch Your Own GAN	ICCV 2021	[code] [webpage]
Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis	WACV 2022	[code]
Customizing GAN Using Few-Shot Sketches	ACM MM 2022
DiffSketching: Sketch Control Image Synthesis with Diffusion Models	BMVC 2022	[code]
MaskSketch: Unpaired Structure-guided Masked Image Generation	CVPR 2023	[project] [code]
Picture that Sketch: Photorealistic Image Generation from Abstract Sketches	CVPR 2023	[project] [code]

• Human Face / Portrait

Paper	Source	Code/Project Link
LinesToFacePhoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network	ACM MM 2019
DeepFaceDrawing: Deep Generation of Face Images from Sketches	SIGGRAPH 2020	[project] [code]
DeepFacePencil: Creating Face Images from Freehand Sketches	ACM MM 2020	[project] [code]
Controllable Sketch-to-Image Translation for Robust Face Synthesis	TIP 2021
DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN	TVCG 2022
DeepPortraitDrawing: Generating Human Body Images from Freehand Sketches	C&G 2023
Semantics-Preserving Sketch Embedding for Face Generation	TMM 2023	[project] [code]
Parsing-Conditioned Anime Translation: A New Dataset and Method	TOG 2023	[code]

• 3D image

Paper	Source	Code/Project Link
3D-aware Conditional Image Synthesis	CVPR 2023	[project] [code]

1.2 Style-conditioned

Paper	Source	Code/Project Link
TextureGAN: Controlling Deep Image Synthesis with Texture Patches	CVPR 2018	[code]
Multimodal Unsupervised Image-to-Image Translation	ECCV 2018	[code]
SketchPatch: Sketch Stylization via Seamless Patch-level Synthesis	SIGGRAPH Asia 2020
Self-Supervised Sketch-to-Image Synthesis	AAAI 2021	[code]
CoGS: Controllable Generation and Search from Sketch and Style	ECCV 2022
Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model	WACV 2023	[project] [code]
DemoCaricature: Democratising Caricature Generation with a Rough Sketch	CVPR 2024	[project]

1.3 Text-conditioned

Paper	Source	Code/Project Link
Sketch-Guided Text-to-Image Diffusion Models	SIGGRAPH 2023	[project]
Adding Conditional Control to Text-to-Image Diffusion Models	ICCV 2023	[code]
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models	arxiv 23.02	[code]
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild	NeurIPS 2023	[code] [project]
Stable Doodle (ClipDrop)		[demo]
CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing	arxiv 24.02
Block and Detail: Scaffolding Sketch-to-Image Generation	arxiv 24.02
One-Step Image Translation with Text-to-Image Models	arxiv 24.03	[code]
It's All About Your Sketch: Democratising Sketch Control in Diffusion Models	CVPR 2024	[code]

2. Sketch Based Image Editing

• Arbitrary Image

Paper	Source	Code/Project Link
Free-Form Image Inpainting with Gated Convolution (DeepFill v2)	ICCV 2019	[project] [code]
Fashion Editing with Multi-scale Attention Normalization	1906.00884
DeFLOCNet: Deep Image Editing via Flexible Low-level Controls	CVPR 2021	[code]
DeepSIM: Image Shape Manipulation from a Single Augmented Training Sample	ICCV 2021	[code] [project]
SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches	CVPR 2022	[code] [project]
Draw2Edit: Mask-Free Sketch-Guided Image Manipulation	ACM MM 2023	[code]

• Human Face / Portrait / Hair

Paper	Source	Code/Project Link
FaceShop: Deep Sketch-based Face Image Editing	SIGGRAPH 2018	[project]
CaricatureShop: Personalized and Photorealistic Caricature Sketching	TVCG 2018
Sparse, Smart Contours to Represent and Edit Images	CVPR 2018	[project]
Example-Guided Style-Consistent Image Synthesis from Semantic Labeling	CVPR 2019	[code]
SC-FEGAN: Face Editing Generative Adversarial Network With User's Sketch and Color	ICCV 2019	[code]
Intuitive, Interactive Beard and Hair Synthesis with Generative Models	CVPR 2020
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches	ECCV 2020	[code] [project]
DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control	SIGGRAPH 2021	[code] [project]
SketchHairSalon: Deep Sketch-based Hair Image Synthesis	SIGGRAPH Asia 2021	[project]
Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing	ECCV 2022	[code] [project]
NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields	SIGGRAPH Asia 2022	[project]
SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields	SIGGRAPH 2023
DemoCaricature: Democratising Caricature Generation with a Rough Sketch	arxiv 23.12	[project] [code]

• Anime Editing

Paper	Source	Code/Project Link
AniFaceDrawing: Anime Portrait Exploration during Your Sketching	SIGGRAPH 2023	[project]

• Video Editing

Paper	Source	Code/Project Link
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos	SIGGRAPH 2022	[project]

3. Sketch Based Image Retrieval (SBIR)

• Object-level

Paper	Source	Code/Project Link
Sketch Me That Shoe	CVPR 2016	[code-caffe] [code-tf] [project]
Deep Multi-task Attribute-driven Ranking for Fine-grained Sketch-based Image Retrieval	BMVC 2016
Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval	CVPR 2017	[code]
Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval	ICCV 2017	[project]
Zero-Shot Sketch-Image Hashing	CVPR 2018
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval	CVPR 2018
Generative Domain-Migration Hashing for Sketch-to-Image Retrieval	ECCV 2018	[code]
A Zero-Shot Framework for Sketch Based Image Retrieval	ECCV 2018	[code]
Generalising Fine-Grained Sketch-Based Image Retrieval	CVPR 2019
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval	CVPR 2019	[project] [code]
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search	CVPR 2019
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval	CVPR 2019	[code]
Learning Structural Representations via Dynamic Object Landmarks Discovery for Sketch Recognition and Retrieval	TIP 2019
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval	ICCV 2019
TC-Net for iSBIR: Triplet Classification Network for Instance-level Sketch Based Image Retrieval	ACM MM 2019
Sketch-Based Image Retrieval With Multi-Clustering Re-Ranking	TCSVT 2019
Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval	TCSVT 2019
Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network	AAAI 2020
Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval	CVPR 2020	[code]
Solving Mixed-modal Jigsaw Puzzle for Fine-Grained Sketch-Based Image Retrieval	CVPR 2020
Fine-Grained Instance-Level Sketch-Based Image Retrieval	IJCV 2020
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval	CVPR 2021
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch-Based Image Retrieval	CVPR 2021	[code]
DLA-Net for FG-SBIR: Dynamic Local Aligned Network for Fine-Grained Sketch-Based Image Retrieval	ACM MM 2021
Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax	ACM MM 2021	[code]
Relationship-Preserving Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval	ACM MM 2021
Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval	TPAMI 2021	[project]
TVT: Three-Way Vision Transformer through Multi-Modal Hypersphere Learning for Zero-Shot Sketch-Based Image Retrieval	AAAI 2022
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval	CVPR 2022	[code]
Sketch3T: Test-time Training for Zero-Shot SBIR	CVPR 2022
Augmented Multi-Modality Fusion for Generalized Zero-Shot Sketch-based Visual Retrieval	TIP 2022	[code]
Adaptive Fine-Grained Sketch-Based Image Retrieval	ECCV 2022	[code]
Conditional Stroke Recovery for Fine-Grained Sketch-Based Image Retrieval	ECCV 2022	[code]
A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch	ECCV 2022	[code] [project]
Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval	ACM MM 2022	[code]
Prototype-based Selective Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval	ACM MM 2022
DLI-Net: Dual Local Interaction Network for Fine-Grained Sketch-Based Image Retrieval	TCSVT 2022	[code]
Data-Free Sketch-Based Image Retrieval	CVPR 2023	[code]
Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style	CVPR 2023	[code]
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR	CVPR 2023	[project]
CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not	CVPR 2023	[project]
Cross-Domain Alignment for Zero-Shot Sketch-Based Image Retrieval	TCSVT 2023
Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval	AAAI 2023
Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers	CVPR 2024	[project]
You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval	CVPR 2024	[project]
How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?	CVPR 2024	[project]
Asymmetric Mutual Alignment for Unsupervised Zero-Shot Sketch-Based Image Retrieval	AAAI 2024

• Scene-level

Paper	Source	Code/Project Link
SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches	ECCV 2020
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input	CVPR 2022
SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval using Adaptive GCNs	TIP 2022
FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context	ECCV 2022	[code] [Dataset]
Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision	AAAI 2023

• Video Retrieval

Paper	Source	Code/Project Link
Fine-Grained Instance-Level Sketch-Based Video Retrieval	TCSVT 2020
Fine-Grained Video Retrieval with Scene Sketches	TIP 2023	[project]

4. Sketch Based 3D Shape Retrieval

Paper	Source	Code/Project Link
Sketch-based 3D Shape Retrieval using Convolutional Neural Networks	CVPR 2015
Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval	AAAI 2016
Deep Correlated Metric Learning for Sketch-based 3D Shape Retrieval	AAAI 2017
Learning Barycentric Representations of 3D Shapes for Sketch-based 3D Shape Retrieval	CVPR 2017
Deep Cross-modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-based 3D Shape Retrieval	ECCV 2018
Unsupervised Learning of 3D Model Reconstruction from Hand-Drawn Sketches	ACMMM 2018
Towards 3D VR-Sketch to 3D Shape Retrieval	3DV 2020	[code] [project]
Uncertainty Learning for Noise Resistant Sketch-Based 3D Shape Retrieval	TIP 2021
Towards Fine-Grained Sketch-Based 3D Shape Retrieval	TIP 2021
Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval	AAAI 2022
Retrieval-Specific View Learning for Sketch-to-Shape Retrieval	TMM 2023
Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval	AAAI 2023	[code]
Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting	ICCV 2023

5. Sketch Based 3D Shape Modeling

• Free-hand sketch

Paper	Source	Code/Project Link
3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks	3DV 2017	[code] [project]
Shape Synthesis from Sketches via Procedural Models and Convolutional Networks	TVCG 2017
DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling	SIGGRAPH 2017	[project] [code]
BendSketch: Modeling Freeform Surfaces Through 2D Sketching	SIGGRAPH 2017	[project]
Robust Flow-Guided Neural Prediction for Sketch-Based Freeform Surface Modeling	SIGGRAPH Asia 2018	[code] [project]
DeepSketchHair: Deep Sketch-based 3D Hair Modeling	TVCG 2019
Interactive Liquid Splash Modeling by User Sketches	SIGGRAPH Asia 2020
Monster Mash: A Single-View Approach to Casual 3D Modeling and Animation	SIGGRAPH Asia 2020	[project] [code] [demo]
SAniHead: Sketching Animal-like 3D Character Heads Using a View-surface Collaborative Mesh Generative Network	TVCG 2020
Towards Practical Sketch-Based 3D Shape Generation: The Role of Professional Sketches	TCSVT 2020
Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches	CVPR 2021
Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches	ICCV 2021
Real-time Skeletonization for Sketch-based Modeling	SMI 2021	[code]
Sketch2Pose: Estimating a 3D Character Pose from a Bitmap Sketch	SIGGRAPH 2022	[project] [code]
Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch	TVCG 2022
Structure-aware Editable Morphable Model for 3D Facial Detail Animation and Manipulation	ECCV 2022	[code]
SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling	ECCV 2022	[code]
Deep Reconstruction of 3D Smoke Densities from Artist Sketches	EG 2022	[project] [code]
A Diffusion-ReFinement Model for Sketch-to-Point Modeling	ACCV 2022	[code]
RaBit: Parametric Modeling of 3D Biped Cartoon Characters with a Topological-consistent Dataset	CVPR 2023	[project]
SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling	TVCG 2023	[project] [code]
Reality3DSketch: Rapid 3D Modeling of Objects from Single Freehand Sketches	TMM 2023
Doodle Your 3D: From Abstract Freehand Sketches to Precise 3D Shapes	CVPR 2024	[project]
SENS: Part-Aware Sketch-based Implicit Neural Shape Modeling	EG 2024	[code] [project]

• NeRF

Paper	Source	Code/Project Link
Sketch2NeRF: Multi-view Sketch-guided Text-to-3D Generation	arxiv 24.01
SKED: Sketch-guided Text-based 3D Editing	ICCV 2023	[project]

• CAD sketch

Paper	Source	Code/Project Link
Lifting Freehand Concept Sketches into 3D	SIGGRAPH Asia 2020	[project] [code]
Sketch2CAD: Sequential CAD Modeling by Sketching in Context	SIGGRAPH Asia 2020	[project] [code]
Free2CAD: Parsing Freehand Drawings into CAD Commands	SIGGRAPH 2022	[project] [code]
Symmetry-driven 3D Reconstruction From Concept Sketches	SIGGRAPH 2022	[project]
Piecewise-smooth Surface Fitting Onto Unstructured 3D Sketches	SIGGRAPH 2022	[project]
Reconstruction of Machine-Made Shapes from Bitmap Sketches	SIGGRAPH Asia 2023	[project]
CAD-SIGNet: CAD Language Inference from Point Clouds using Layer-wise Sketch Instance Guided Attention	CVPR 2024

6. Sketch Based Garment Design

Paper	Source	Code/Project Link
Sketching Folds: Developable Surfaces from Non-Planar Silhouettes	TOG 2015	[project]
FoldSketch: Enriching Garments with Physically Reproducible Folds	SIGGRAPH 2018	[project]
Learning a Shared Shape Space for Multimodal Garment Design	SIGGRAPH Asia 2018	[project]
Garment Ideation: Iterative view-aware sketch-based garment modeling	3DV 2022	[code]
Controllable Visual-Tactile Synthesis	ICCV 2023	[project] [code]
Controllable Garment Image Synthesis Integrated with Frequency Domain Features	PG 2023
Toward Intelligent Interactive Design: A Generation Framework Based on Cross-domain Fashion Elements	ACM MM 2023
FashionDiff: A Controllable Diffusion Model Using Pairwise Fashion Elements for Intelligent Design	ACM MM 2023

7. Sketch Completion

• Sketch completion

Paper	Source	Code/Project Link
SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network	CVPR 2019
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches	BMVC 2020	[code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt	CVPR 2020	[code]
Generative Sketch Healing	IJCV 2022

• Sketch gap / connectivity detection

Paper	Source	Code/Project Link
Joint Gap Detection and Inpainting of Line Drawings	CVPR 2017	[project] [code]
Detecting Viewer-Perceived Intended Vector Sketch Connectivity	SIGGRAPH 2022	[project]

8. Sketch Restoration, Retargeting and Inpainting

• Manga

Paper	Source	Code/Project Link
Exploiting Aliasing for Manga Restoration	CVPR 2021	[webpage] [code]
Seamless Manga Inpainting with Semantics Awareness	SIGGRAPH 2021	[webpage] [code]
Screentone-Preserved Manga Retargeting	arxiv 22.03
Manga Rescreening with Interpretable Screentone Representation	arxiv 23.06

• Hand-Drawn Drawings

Paper	Source	Code/Project Link
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator	CVPR 2023

9. Sketch and Shading

Paper	Source	Code/Project Link
Learning to Shadow Hand-drawn Sketches	CVPR 2020	[project] [code]
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings	ICCV 2021	[project]

10. Sketch Enhancement

Paper	Source	Code/Project Link
SketchMan: Learning to Create Professional Sketches	ACM MM 2020	[code]
APISR: Anime Production Inspired Real-World Anime Super-Resolution	CVPR 2024	[code]

11. Sketch-Guided Object Localization

Paper	Source	Code/Project Link
Sketch-Guided Object Localization in Natural Images	ECCV 2020	[project] [code]
Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild	arxiv 21.09	[code]
What Can Human Sketches Do for Object Detection?	CVPR 2023	[project]
Sketch-based Video Object Segmentation: Benchmark and Analysis	BMVC 2023	[code]
Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch	WACV 2024	[project] [code]

12. Sketch-Guided Video Synthesis

Paper	Source	Code/Project Link
Deep Sketch-guided Cartoon Video Inbetweening	TVCG 2021	[code]

13. Sketch Recognition

Paper	Source	Code/Project Link
Sketch-a-Net that Beats Humans	BMVC 2015	[code]
Sketch-a-Net: A Deep Neural Network that Beats Humans	IJCV 2017	[code]
Deep Self-Supervised Representation Learning for Free-Hand Sketch	TCSVT 2020	[code]
S3Net:Graph Representational Network For Sketch Recognition	ICME 2020	[code]
Sketchformer: Transformer-based Representation for Sketched Structure	CVPR 2020	[code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt	CVPR 2020	[code]
Multi-Graph Transformer for Free-Hand Sketch Recognition	TNNLS 2021	[code]
Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition	TVCG 2021	[code]
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting	CVPR 2021	[code]
SketchAA: Abstract Representation for Abstract Sketches	ICCV 2021
SketchLattice: Latticed Representation for Sketch Manipulation	ICCV 2021	[code]
Multi-Stage Spatio-Temporal Networks for Robust Sketch Recognition	TIP 2022

14. Sketch Segmentation and Perceptual Grouping

• Semantic / Instance Segmentation

Type	Paper	Source	Code/Project Link
Pixelwise	SPFusionNet: Sketch Segmentation Using Multi-modal Data Fusion	ICME 2019
SketchyScene: Richly-Annotated Scene Sketches (scene-level)	ECCV 2018	[code]
Exploring Local Detail Perception for Scene Sketch Semantic Segmentation (scene-level)	TIP 2022
Open Vocabulary Semantic Scene Sketch Understanding (scene-level)	CVPR 2024	[code] [project]
Stroke-level	Free Hand-Drawn Sketch Segmentation	ECCV 2012
	Data-driven Segmentation and Labeling of Freehand Sketches	SIGGRAPH Asia 2014	[code] [project] [dataset]
	Example-Based Sketch Segmentation and Labeling Using CRFs	TOG 2016
	SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation	IEEE Access 2019
	Fast Sketch Segmentation and Labeling With Deep Learning	CGA 2019
	Stroke-based sketched symbol reconstruction and segmentation	CGA 2020
	SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks	TOG 2021	[code]
	One Sketch for All: One-Shot Personalized Sketch Segmentation	TIP 2022
	Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches	TIP 2023	[code]
	CreativeSeg: Semantic Segmentation of Creative Sketches	TIP 2024	[code]

• Panoptic Segmentation

Type	Paper	Source	Code/Project Link
Vector	FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting	ICCV 2021	[project]
	GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings	CVPR 2022
	CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings	CVPR 2022	[code]
	VectorFloorSeg: Two-Stream Graph Attention Network for Vectorized Roughcast Floorplan Segmentation	CVPR 2023	[code]

• Perceptual Grouping

Type	Paper	Source	Code/Project Link
Stroke-level	Sketching by perceptual grouping	ICIP 2013
	Making Better Use of Edges via Perceptual Grouping	CVPR 2015
	Universal Sketch Perceptual Grouping / Toward Deep Universal Sketch Perceptual Grouper	ECCV 2018 / TIP 2019	[code]

15. Sketch Representation Learning

• Stroke order importance/saliency, sketch abstraction

Paper	Source	Code/Project Link	Data Manner	Related Applications
Learning Deep Sketch Abstraction	CVPR 2018		Vector/stroke-level	FG-SBIR
Goal-Driven Sequential Data Abstraction	ICCV 2019		Vector/stroke-level	Sketch recognition
Pixelor: a competitive sketching AI agent. So you think you can sketch?	SIGGRAPH Asia 2020	[Project] [Code]	Vector/stroke-level	Sketch synthesis and recognition
SketchAA: Abstract Representation for Abstract Sketches	ICCV 2021		Vector/stroke-level	Sketch recognition, FB-SBIR, sketch healing
CLIPasso: Semantically-Aware Object Sketching	SIGGRAPH 2022	[project] [code]	Vector/stroke-level
Abstracting Sketches through Simple Primitives	ECCV 2022	[code]	Vector/stroke-level	Sketch recognition, FG-SBIR
Learning Geometry-aware Representations by Sketching	CVPR 2023	[code]	Vector/stroke-level	object attribute classification, domain transfer, stroke-based generation, FG-SBIR
SketchXAI: A First Look at Explainability for Human Sketches	CVPR 2023	[project]	Vector/stroke-level	sketch recognition
Prediction with Visual Evidence: Sketch Classification Explanation via Stroke-Level Attributions	TIP 2023		Vector/stroke-level
What Sketch Explainability Really Means for Downstream Tasks	CVPR 2024		Vector/stroke-level

• Conventional Representation Learning

Paper	Source	Code/Project Link	Data Manner	Related Applications
A Neural Representation of Sketch Drawings (Sketch-RNN)	ICLR 2018	[code]	Vector/stroke-level	Reconstruction and interpolation
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches	BMVC 2020	[code]	Vector/stroke-level	Sketch recognition, retrieval, completion and analogy
Sketchformer: Transformer-based Representation for Sketched Structure	CVPR 2020	[code]	Vector/stroke-level	Sketch classification, SBIR, reconstruction and interpolation
CoSE: Compositional Stroke Embeddings	NeurIPS 2020	[Code]	Vector/stroke-level	Auto-completing diagrams
SketchLattice: Latticed Representation for Sketch Manipulation	ICCV 2021	[code]	Lattice graph	Sketch healing and image-to-sketch synthesis
SketchODE: Learning neural sketch representation in continuous time	ICLR 2022	[Project]	Vector/Stroke-level	Continuous-time representation, reconstruction & interpolation
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation	AAAI 2023	[code]	Vector/Stroke-level	Sketch synthesis and sketch healing
SketchKnitter: Vectorized Sketch Generation with Diffusion Models	ICLR 2023	[code]	Vector/Stroke-level	vectorized sketch generation
ChiroDiff: Modelling chirographic data with Diffusion Models	ICLR 2023	[Project]	Vector/Stroke-level	vectorization, de-noising/healing, abstraction
Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing	AAAI 2024	[code]	Vector/Stroke-level	recognition and segmentation
Modelling complex vector drawings with stroke-clouds	ICLR 2024	[code]	Vector/Stroke-level	reconstruction and generation
Sketchformer++: A Hierarchical Transformer Architecture for Vector Sketch Representation	CVM 2024		Vector/Stroke-level	reconstruction, recognition and semantic segmentation

Paper	Source	Code/Project Link	Data Manner	Related Applications
A Learned Representation for Scalable Vector Graphics	ICCV 2019	[code]	SVG	Font design
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation	NeurIPS 2020	[Code] [Project]	SVG	Vector Graphics Animation, reconstruction and interpolation
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers	CVPR 2023		SVG	Reconstruction, classification, interpolation, retrieval
StrokeNUWA: Tokenizing Strokes for Vector Graphic Synthesis	arxiv 24.01		SVG	generation

• Self-supervised or few/zero-shot learning

Paper	Source	Code/Project Link	Data Manner	Related Applications
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation	TCSVT 2019			few-shot classification
On Learning Semantic Representations for Large-Scale Abstract Sketches	TCSVT 2020	[Code]	Bitmap and Vector	Retrieval and recognition
Deep Self-Supervised Representation Learning for Free-Hand Sketch	TCSVT 2020	[Code]	Vector/stroke-level	Retrieval and recognition
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt	CVPR 2020	[Code]	Vector/stroke-level	Sketch recognition, retrieval, and gestalt
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting	CVPR 2021	[Code]	Both Vector and Raster-level	Recognition

• Scene-level

Paper	Source	Code/Project Link
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text	CVPR 2023

• Implicit Neural Representations

Paper	Source	Code/Project Link
SketchINR: A First Look into Sketches as Implicit Neural Representations	CVPR 2024

16. Sketch and Visual Correspondence

• Datasets

Name	Paper	Source	Code/Project Link
CreativeFlow+	Creative Flow+ Dataset	CVPR 2019	[code]
ATD-12K	Deep animation video interpolation in the wild	CVPR 2021	[code]
AnimeRun	AnimeRun: 2D Animation Correspondence from Open Source 3D Movies	NeurIPS 2022	[code]

• Methods

Paper	Source	Code/Project Link
Globally optimal toon tracking	SIGGRAPH 2016	[project]
SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence	TCSVT 2020
SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings	CGF 2021	[project]
The Animation Transformer: Visual Correspondence via Segment Matching	ICCV 2021
Differential Operators on Sketches via Alpha Contours	SIGGRAPH 2023	[code]
Learning Inclusion Matching for Animation Paint Bucket Colorization	CVPR 2024	[code] [project]

17. Sketch Animation/Inbetweening

• Inbetweening

Paper	Source	Representation	Code/Project Link
BetweenIT: An Interactive Tool for Tight Inbetweening	CGF 2010	stroke
Context-Aware Computer Aided Inbetweening	TVCG 2017	stroke
FTP-SC: Fuzzy Topology Preserving Stroke Correspondence	SCA 2018	stroke	[webpage] [video]
Cacani: 2d animation and inbetween software	/	stroke	[software]
Stroke-Based Drawing and Inbetweening with Boundary Strokes	CGF 2021	stroke
Joint Stroke Tracing and Correspondence for 2D Animation	TOG 2024	stroke	[code] [webpage]
Deep Geometrized Cartoon Line Inbetweening	ICCV 2023	vertex/point	[code]
Optical Flow Based Line Drawing Frame Interpolation Using Distance Transform to Support Inbetweenings	ICIP 2019	raster
Bridging the Gap: Fine-to-Coarse Sketch Interpolation Network for High-Quality Animation Sketch Inbetweening	arxiv 23.08	raster

• Animation

Paper	Source	Representation	Code/Project Link
Autocomplete Hand-drawn Animations	SIGGRAPH Asia 2015	stroke	[webpage] [video]
Live Sketch: Video-driven Dynamic Deformation of Static Drawings	CHI 2018	vector	[video]
Animating Portrait Line Drawings from a Single Face Photo and a Speech Signal	SIGGRAPH 2022	image	[code]
A Method for Animating Children’s Drawings of the Human Figure	TOG 2023	image	[code] [project] [demo]
Non-linear Rough 2D Animation using Transient Embeddings	CGF 2023	stroke
Sketch Video Synthesis	EG 2024	stroke	[code] [project]
Breathing Life Into Sketches Using Text-to-Video Priors	CVPR 2024	stroke	[code] [project]
AniClipart: Clipart Animation with Text-to-Video Priors	arxiv 24.04	image	[project]

18. Sketch and AR/VR

Paper	Source	Code/Project Link
SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image	UIST 2017	[video]
Model-Guided 3D Sketching	TVCG 2018	[video]
Mobi3DSketch: 3D Sketching in Mobile AR	CHI 2019	[video]
Interactive Liquid Splash Modeling by User Sketches	SIGGRAPH Asia 2020	[video]
Towards 3D VR-Sketch to 3D Shape Retrieval	3DV 2020	[code] [project]
3D Curve Creation on and around Physical Objects with Mobile AR	TVCG 2021	[video]
HandPainter - 3D Sketching in VR with Hand-based Physical Proxy	CHI 2021	[video]
Fine-Grained VR Sketching: Dataset and Insights	3DV 2021	[code]
Structure-Aware 3D VR Sketch to 3D Shape Retrieval	3DV 2022	[code]
GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand	TVCG 2023
3D VR Sketch Guided 3D Shape Prototyping and Exploration	ICCV 2023	[code]

19. Sketch Based Incremental Learning

Paper	Source	Code/Project Link
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches	CVPR 2022	[code]

20. Sketch Quality Measurement

Paper	Source	Code/Project Link
Annotation-Free Human Sketch Quality Assessment	IJCV 2024	[code]
Finding Badly Drawn Bunnies	CVPR 2022	[code]

21. Cloud Augmentation with Sketches

Paper	Source	Code/Project Link
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches	ACM MM 2022	[project]

22. Sketch and Re-identification

Paper	Source	Code/Project Link
Cross-Compatible Embedding and Semantic Consistent Feature Construction for Sketch Re-identification	ACM MM 2022	[code]
SketchTrans: Disentangled Prototype Learning with Transformer for Sketch-Photo Recognition	TPAMI 2023	[code]

23. Sketch-based Salient Object Detection

Paper	Source	Code/Project Link
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings	CVPR 2023