The recent advances in deep learning techniques provide diverse possibilities for generation of artificial images based on different input parameters. One interesting functionality is deep image-to-image translation, when a new picture is generated on the basis of the provided reference image.This way, it is possible to create, for example, an artificial photograph of a person based on its initial rough hand-made sketch.
Up until now such kind of image generation suffered from different limitations. One of them required the reference image to be quite well-done due to the fact that existing algorithms tended to overfit the resulting synthetic image, leading to unnaturally-looking distortions.
In a recent paper published on arXiv.org, a team of scientists demonstrated an improved platform for deep generation of face images. To solve aforementioned limitation, the researchers implicitly modeled the shape space of potential face images and to use this shape space to approximate the input sketch, thus leading to much higher realism of synthesized face images.
In this paper we have presented a novel deep learning framework for synthesizing realistic face images from rough and/or incomplete freehand sketches. We take a local-to-global approach by first decomposing a sketched face into components, refining its individual components by projecting them to component manifolds defined by the existing component samples in the feature spaces, mapping the refined feature vectors to the feature maps for spatial combination, and finally translating the combined feature maps to realistic images. This approach naturally supports local editing and makes the involved network easy to train from a training dataset of not very large scale. Our approach outperforms existing sketch-to-image synthesis approaches, which often require edge maps or sketches with similar quality as input. Our user study confirmed the usability of our system. We also adapted our system for two applications: face morphing and face copy-paste.
Link to the project site: https://geometrylearning.com/DeepFaceDrawing/