Collection of scripts for generating depth maps for videos using machine learning.
Collection of scripts for generating depth maps for videos using machine learning.
You can run the whole process directly from your browser without setting anything up locally, thanks to Google Colab.
June 2021 update
Use the following Colab notebook to use BoostingMonocularDepth instead of MiDaS for even more detailed (and slower) depth maps.
pip3 install torch torchvision opencv-python timm Pillow numpy
depth.py
, average.py
and merge.py
into a folder with the video you wish to depthify. In our case we are using a short, royalty-free video of an elephant.rgb
, depth
, averaged
, and merged
.Your folder structure should look like this:
.
├── averaged
├── average.py
├── depth
├── depth.py
├── Elephant.mp4
├── merged
├── merge.py
└── rgb
ffmpeg -i Elephant.mp4 2>&1 | sed -n "s/.*, \(.*\) fp.*/\1/p"
rgb
folder).ffmpeg -i Elephant.mp4 -qmin 1 -qscale:v 1 ./rgb/%06d.jpg
depth
folder).For faster (but lower quality) depth map generation you can switch to a lighter model by changing line 13 to False
.
python3 depth.py
averaged
folder).See Elephant_depth_averaged_sound.mp4 vs. Elephant_depth_sound.mp4 to see the effect of this step.
python3 average.py
merged
folder).python3 merge.py
25
fps with the number you got from step 1).ffmpeg -framerate 25 -i ./merged/%06d.jpg -vcodec libx264 -pix_fmt yuv420p Elephant_depth.mp4
ffmpeg -i Elephant_depth.mp4 -i ./Elephant.mp4 -c copy -map 0:0 -map 1:1 -shortest Elephant_depth_sound.mp4
MiDaS result:
BoostingMonocularDepth result:
BoostingMonocularDepth does not always provide better results compared to vanilla MiDaS, as you can see on the following example.
MiDaS result:
BoostingMonocularDepth result: