Visual Computing Markerless Motion Pose Face Detection Tracking And 3D Reconstruction Save

Project is developed as an interest in Visual Computing.

Markerless Motion and/or Pose and/or Face detection and/or tracking and it's 3D reconstruction (in real time)

Inspiration:

• I get very interested when camera captured images and videos (or live videos) turn into 3D virtual characters and their motion into its animation. I want to pursue this feeling.
• What is Visual Computing?.
• What is 3D reconstruction?. This link gives a basic foundational idea on 3D reconstruction from images. Using 3D reconstruction one can determine any object’s 3D profile, as well as knowing the 3D coordinate of any point on the profile. from section Motivation and applications from this link is euphoric. Plus the methods for 3D reconstruction and in detail explanation of Binocular stereo vision and how it works is beautiful. There are other sources and external link.
• What is Structure From Motion and its application in 3D reconstruction.
• Real time face reconstruction in 3D.
• Videos collection from ETH Zurich Computer graphics lab.
• ETH Zurich Computer Graphics webiste with links to its research and published papers.
• High-Resolution 3D Reconstruction - Humboldt-Universität zu Berlin.
• CS231A: Computer Vision, From 3D Reconstruction to Recognition Course notes. Online Course for the same - Convolutional Neural Networks for Visual Recognition.
• Here is one such excitement and here is the direct link to the page containing the paper which is also present in tum research lab.
• Here is disney research hub youtube channel and disney research website.
• Here is another such excitement on Avatar Digitization From a Single Image For Real-Time Rendering.
• From University of Bonn on 3D Reconstruction of Human Motion from Video.
• Again from University of Bonn on 3D Pose Estimation from a Single Monocular Image .
• 3D reconstruction from multiple images wiki information and other external links here.
• The Multi-View Environment, MVE, is an implementation of a complete end-to-end pipeline for image-based geometry reconstruction. It features Structure-from-Motion, Multi-View Stereo and Surface Reconstruction. MVE is written in C++ and comes with a set of efficient, cross-platform and easy-to-use libraries. The code runs on Linux, MacOS X and Windows.
• Middlebury Computer Vision Pages is a repository for computer vision evaluations and datasets. See the Middlebury Stereo Vision Page and Middlebury Multi-View Stereo Vision Page for more relevant info for this project.
• Browse here for interesting papers.

Other tutorials and links :

• Interesting tutorial on Head Pose Estimation using OpenCV and Dlib .
• In this tutorial you will learn how to use the reconstruction api for sparse reconstruction.
• Variational Methods in Computer Vision.
• 3D Reconstruction from multiple views.
• Live 3D Reconstruction on Mobile Phones - ETH Zurich.
• Beginners guide to computer vision.
• A curated list of computer vision resources.

Books and online courses on Visual Computing:

1. Introduction to Deep Learning from MIT. The lecture series is also available as youtube playlist.
2. Great lecture on understanding of Computer Vision from very foundation : First Principles of Computer Vision by Shree Nayar, Professor of Computer Science at Columbia Engineering. The Lecture Series is also available as a youtube channel.
3. Applied Machine Learning (Cornell Tech CS 5787, Fall 2020) by Volodymyr Kuleshov.
4. A beautiful book with detail understanding of 3D reconstruction from basic to advanced - Multiple View Geometry in Computer Vision.
5. A MUST HAVE COURSE ON MATHEMATICAL FOUNDATION FOR VISUAL COMPUTING Computational Science and Engineering I, its video lectures collection are here. This course provides a review of linear algebra, including applications to networks, structures, and estimation, Lagrange multipliers. Also covered are: differential equations of equilibrium; Laplace's equation and potential flow; boundary-value problems; minimum principles and calculus of variations; Fourier series; discrete Fourier transform; convolution; and applications.
6. Online Course for Convolutional Neural Networks for Visual Recognition.
7. Computer Vision: Algorithms and Applications has a dedicated chapter on 3D reconstruction and other interesting applications. Book is from this author.
8. Book- Computer Vision: From 3D Reconstruction to Visual Recognition.
9. Here is a course in edx from TUM Munich on Autonomous Navigation for Flying Robots which is adopted from their own university MOOC course. It gives concepts on 3D reconstruction as well.
10. Big bunch of and series of computer vision teachings from Stanford University vision lab. For example:
    • CS131 Computer Vision: Foundations and Applications.
    • CS231A: Computer Vision, From 3D Reconstruction to Recognition.
11. Lecture videos on Computer Vision For Visual Effects by Richard J. Radke.
    • Here is the youtube playlist on the same.
    • Associated is this book on Computer Vision for Visual Effects. The book describes classical computer vision algorithms used on a regular basis in Hollywood (such as blue-screen matting, structure from motion, optical flow, and feature tracking) and exciting recent developments that form the basis for future effects (such as natural image matting, multi-image compositing, image retargeting, and view synthesis).

Books and resources on Mathematics for Visual Computing (AI/ML/CV/CG):

1. Book : Github repository for the book Mathematics for Machine Learning.
2. Learn essence of Linear Algebra from 3Blue1Brown youtube channel.
   • Also learn essence of Calculus.
3. Learn linear algebra and Computational science and engineering from Gilbert strang.
   • Follow his other lectures and books from his website.

On Computer Graphics:

• You sure will have to know much about computer Graphics too along with Computer Vision.
• Also complete with TODO mentioned on this OpenGL playground of yours.

Noteworthy open source libraries for visual computing:

1. CGAL is a software project that provides easy access to efficient and reliable geometric algorithms in the form of a C++ library. CGAL is used in various areas needing geometric computation, such as geographic information systems, computer aided design, molecular biology, medical imaging, computer graphics, and robotics.
2. The Visualization Toolkit (VTK) is an open-source, freely available software system for 3D computer graphics, modeling, image processing, volume rendering, scientific visualization, and 2D plotting. It supports a wide variety of visualization algorithms and advanced modeling techniques, and it takes advantage of both threaded and distributed memory parallel processing for speed and scalability, respectively.
3. The Point Cloud Library (or PCL) is a large scale, open project [1] for 2D/3D image and point cloud processing. The PCL framework contains numerous state-of-the art algorithms including filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation.
4. OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning software library.
5. Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real world problems.
6. Althouh the libraries I mention in this point are far related to project topic, however they are noteworthy libraries under visual computing area :
   • The Geospatial Data Abstraction Library (GDAL) is a computer software library for reading and writing raster and vector geospatial data formats. Although it is used for GIS data I have put it here because it can come under visual computing with raster GIS data.
   • Another, library which uses GDAL is Orfeo Toolbox which also can fall under visual computing because it works on using maching learning and computer vision with GIS raster images.

Get started with:

• OpenCV Face detection C++.
• Mastering opencv with practical projects.
• Get started with this opencv tutorial series on Camera Calibration and 3D Reconstruction.
• Now you can as well do this tutorial on Head Pose Estimation using OpenCV and Dlib.
• Start with this tutorial on 3D scene reconstruction from these set of OpenCV Tutorials called Structure From Motion. Make sure you do the tutorials in sequence and make sure the version of this OpenCV docs matches with the version of OpenCV version you are using.
• This tutorial as well on Real Time pose estimation of a textured object .

Future direction:

• 3D pose estimation
• 3D motion reconstruction
• Mapping human motion to 3D models or characters

Final Goal:

• Reach a level near or far to Real time face reconstruction as good as this. The link to main page of this paper is here.
• Also this exercise can be used to implement VFX like this using OpenCV - Realtime Hand.