Python examples on how to use GStreamer within OpenCV. Now with GPU support! :fire::fire::fire:
These examples, written in Python, will provide a good starting point for a lot, and the most common, applications of GStreamer and OpenCV. The snippets mainly use OpenCV's VideoWriter and VideoCapture object, and include the following functionalities:
import cv2
# Cam properties
fps = 30.
frame_width = 1920
frame_height = 1080
# Create capture
cap = cv2.VideoCapture(0)
# Set camera properties
cap.set(cv2.CAP_PROP_FRAME_WIDTH, frame_width)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, frame_height)
cap.set(cv2.CAP_PROP_FPS, fps)
# The following string usually works on most webcams
webcam2appsink_YUY2_640_480 = "v4l2src device=/dev/video0 ! video/x-raw, format=YUY2, width=640, height=480, pixel-aspect-ratio=1/1, framerate=30/1 ! videoconvert ! appsink"
gst_str = "appsrc ! videoconvert ! shmsink socket-path=/tmp/foo sync=true wait-for-connection=false shm-size=10000000"
cap = cv2.VideoCapture("shmsrc socket-path=/tmp/foo ! video/x-raw, format=BGR, width=640, height=480, pixel-aspect-ratio=1/1, framerate=30/1 ! decodebin ! videoconvert ! appsink")
gst_str_rtp = "appsrc ! videoconvert ! x264enc noise-reduction=10000 tune=zerolatency byte-stream=true threads=4 " \
" ! h264parse ! mpegtsmux ! rtpmp2tpay ! udpsink host=127.0.0.1 port=5000"
# mfxh264enc does all the HW encoding on the INTEL HD GPU
appsink2file = "appsrc ! videoconvert ! mfxh264enc ! \
video/x-h264, profile=baseline ! \
matroskamux ! filesink location=the_gstreamer_enjoyer.mkv"
Since you are here, you probably know why you want to use GStreamer and OpenCV and I'm not gonna list all the advantages that GStreamer brings to the table. However, if you find this repo helpful or even remotely funny, consider leaving a star. Or not. Your choice.
Be a unix enthusiast. The GPU encoding pipelines in GStreamer are extremly powerful, but are hard to install.
Ubuntu 20.04
Correct drivers for GPU (CUDA + CUDNN if necessary)
Install GStreamer
sudo apt-get install gstreamer1.0* libgstreamer-plugins-bad1.0-0 libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev libgstreamer1.0 libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgtk-3-dev
optional FFMPEG, QT (+ OpenGL)
Intel-CPUs
Supported platforms: Intel Haswell / Broadwell / Skylake with Intel HD / Iris Pro graphics / Apollo Lake) (>= Gen4)
Noticable features:
Installation process (Honestly this is pretty frustrating Intel. I got very mad followed by getting drunk and yelling at the PC, but as far as i remember the process went something like this):
Detailed examples here
Achieved speedup for writing a Full-HD frame (1920x1080px) to NVMe SSD:
NVIDIA GPUs
/usr/lib/x86_64-linux-gnu/gstreamer-1.0/
, or in /usr/local/lib/gstreamer-1.0/
or even in your local path if you don't have sudo
. Truth is that i copipasta'd the files until it matched. There might be a more elegant way to do this.AMD GPUs
sudo apt-get install cmake-qt-gui
); search and click the features you want to have enabled (even after your exec'd a usual cmake -D
flag)-D WITH_GSTREAMER=ON
$ gst-launch-1.0 v4l2src ! xvimagesink
$ gst-inspect-1.0 | grep mfx
mfx: mfxh264dec: MFX H264 decoder
mfx: mfxhevcdec: MFX HEVC decoder
mfx: mfxh264enc: MFX H.264 encoder
mfx: mfxhevcenc: MFX H.265 encoder
$ gst-inspect-1.0 | grep nvenc
nvenc: nvh264enc: NVENC H.264 Video Encoder
VideoCapture(0)
and puts the raw image in a shared memory. Mind here to define your own webcam properties.VideoCapture(0)
,encodes the frame and streams it to rtp://localhost:5000
rtp://localhost:5000
.v4l2src /dev/video0
(usually webcam) to OpenCV format and writes it as an h264 encoded file.v4l2src /dev/video0
(usually webcam) to OpenCV format and writes it as an h264 encoded rtp stream. Use the provided *.sdp
files for VLC viewer.rtp://localhost:5000
.Inspired by https://github.com/tik0/mat2gstreamer