Theme

Human-Computer Interaction

Topic

3D Style Transfer

Background

As the data visualization team of Data Technology & Products Department in Alibaba Groups, one of our missions is to bring innovative real-time 3D animation to the dashboards of media center during our Double Eleven Shopping Festival. Inspired by the recent advances of deep learning in 2D image style transfer tasks, we would like to have a set of scalable algorithms being capable of transferring geographic 3D animation into a variety of styles such as watercolor or oil painting in real-time fashion. The challenge of this proposal is both high-resolution quality and low-latency computing. One related work about video stylization by Adobe is promising however it requires expensive motion processing and multi-keyframe examples given. 

Depth image and model annotation are available for our 3D scene animation since we have complete control of our 3D rendering pipeline. The research work taking advantage of these information in style transfer has not been seen to our best knowledge.

Target

• A set of deep learning algorithms for style transfer of 3D scene/animation in real-time fashion.

Related Research Topics

• Example-based approach to video stylization.
• Generative Adversarial Networks.
• Image analogy framework.
• Temporal coherence.

Suggested Collaboration Method

ARF (Alibaba Research Fellowship), three months ~ one year onsite.