Connecting openFrameworks to Google MediaPipe Machine Learning Framework over UDP
MediaPipe is a cross-platform framework for building multimodal applied machine learning pipelines. I want to be able to use it in external applications.
This tutorial walks through how to stream MediaPipe data out over UDP, so any external app and receive and use the data.
I show how to modify the mediapipe example mediapipe/examples/desktop/hand_tracking to add in a new node that recieves hand tracking data as input, broadcasts that data over UDP on port 8080, and then passes the tracking data on to the rest of the graph as output.
Tested on macOS Mojave (10.14.6) and openFrameworks 0.10.1
Beigin by installing MediaPipe on your system using google's instructions.
Then install and setup Google Protobufs for openFrameworks using my previous tutorial.
If you've never used Bazel before, the build system and organization of Mediapipe can be really confusing. I try to go through step-by-step below, but you can find more information in the MediaPipe Docs.
Modify mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
# Add New Node
node {
calculator: "MyPassThroughCalculator"
input_stream: "LANDMARKS:hand_landmarks"
input_stream: "NORM_RECT:hand_rect"
input_stream: "DETECTIONS:palm_detections"
output_stream: "LANDMARKS:hand_landmarks_out"
output_stream: "NORM_RECT:hand_rect_out"
output_stream: "DETECTIONS:palm_detections_out"
}
# Modify input_stream names of next node
# Subgraph that renders annotations and overlays them on top of the input
# images (see renderer_cpu.pbtxt).
node {
calculator: "RendererSubgraph"
input_stream: "IMAGE:input_video"
input_stream: "LANDMARKS:hand_landmarks_out"
input_stream: "NORM_RECT:hand_rect_out"
input_stream: "DETECTIONS:palm_detections_out"
output_stream: "IMAGE:output_video"
}
NOTE
You can visualize the graph to test that inputs and outpus match up at https://viz.mediapipe.dev/
When you add in the custom MyPassThroughCalculator your graph should look like this:
Adding UDP, Detections, Landmarks, and Hand Rectangles to the PassThrough Calculator
Copy the file src/mediapipe/my_pass_though_calculator.cc to your Mediapipe calculators directory mediapipe/calculators/core.
The main differences between my_pass_though_calculator.cc
and the original pass_though_calculator.cc
are that it adds UDP streaming, and uses Landmark, Rect, and Detection protobufs in the ::mediapipe::Status Process()
function. Next we need to modify the graph file to declare the Tag of the calculators input and stream.
my_pass_through_calculator
dependencies:cc_library(
name = "my_pass_through_calculator",
srcs = ["my_pass_through_calculator.cc"],
visibility = [
"//visibility:public",
],
deps = [
"//mediapipe/framework:calculator_framework",
"//mediapipe/framework/port:status",
"//mediapipe/framework/formats:landmark_cc_proto",
"//mediapipe/framework/formats:rect_cc_proto",
"//mediapipe/framework/formats:detection_cc_proto",
"//mediapipe/framework/formats:wrapper_hand_tracking_cc_proto",
],
alwayslink = 1,
)
cc_library(
name = "desktop_offline_calculators",
deps = [
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/core:gate_calculator",
"//mediapipe/calculators/core:immediate_mux_calculator",
"//mediapipe/calculators/core:packet_inner_join_calculator",
"//mediapipe/calculators/core:previous_loopback_calculator",
"//mediapipe/calculators/video:opencv_video_decoder_calculator",
"//mediapipe/calculators/video:opencv_video_encoder_calculator",
"//mediapipe/calculators/core:my_pass_through_calculator"
],
)
Copy the wrapper_hand_tracker.proto
in this repo's /mediapipe
directory into mediapipe/framework/formats
directory.
Add it and its dependencies to mediapipe/framework/formats/BUILD
:
mediapipe_proto_library(
name = "wrapper_hand_tracking_proto",
srcs = ["wrapper_hand_tracking.proto"],
visibility = ["//visibility:public"],
deps = [
"//mediapipe/framework/formats:landmark_proto",
"//mediapipe/framework/formats:detection_proto",
"//mediapipe/framework/formats:rect_proto",
],
)
You should be able to build and run the mediapipe hand_tracking_desktop_live example now without any errors. In your mediapipe root directory, run:
bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/hand_tracking:hand_tracking_cpu
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
There's no good way to send multiple messages (like a LandmarkList, Rect, and DetectionList) in proto2. Therefore, you have to wrap those messages into a new protobuf.
I called mine wrapper_hand_tracking.proto
and added it and its dependencies to mediapipe/framework/formats/BUILD
.
Here's a roadblock I hit:
protobuf-3.11.4
and is built with Bazel, but I'm using protobuf-3.6.1
for openFrameworks, and it builds with CMake.wrapper_hand_tracking.proto
generated with Bazel to the openFrameworks project, there are compatibility issues with the protobuf static library I've already built and linked.So the hacky workaround I found was to build wrapper_hand_tracking.proto
with the MediaPipe build system, and then rebuild it with my system-wide protobuf installation. Here's what I did ... in the top-level /mediapipe
directory:
wrapper_hand_tracking.proto
file is commented out, just leaving the import calls to link to dependent protos.protoc
.
.pb.h
and .pb.cc
files from wrapper_hand_tracking.proto
using this library.wrapper_hand_tracking.proto
, comment out the import calls at the top of the file, and uncomment the body of the proto (protoc
doesn't link dependencies, so I just copied all the necessary protobufs into this one file).mediapipes\framework\formats
directory and run protoc --cpp_out=. wrapper_hand_tracking.proto
wrapper_hand_tracking.pb.h
and wrapper_hand_tracking.pb.cc
files over to your openFrameworks src folder.#include "wrapper_hand_tracking.pb.h"
to ofApp.h
When you run the MediaPipe example hand_tracking_desktop_live, it broadcasts any hand landmarks and rectangles on port localhost:8080
. The openFrameworks example example-protobuf-udp is listening for those protobufs on port 8080.
GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/hand_tracking/hand_tracking_cpu --calculator_graph_config_file=mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
You should see a video feed of yourself, with hand landmarks and bounding rectangle overlaid.
Build example-protobuf-udp in openFramework's Project Generator
Drag and drop the /libs
directory into the Xcode Project Pane and select Add to Target.
Drag and drop your newly generated wrapper_hand_tracking.pb.cc
and wrapper_hand_tracking.pb.h
files into the /src
directory in the Project Pane and select Add to Target.
Link the libprotobuf.a
static library from /libs/protobuf/lib/osx
in Project Settings > General > Linked Frameworks and Libraries.
In Project Settings > Build Settings, add the following to Header Search Paths:
$(PROJECT_DIR)/libs/protobuf/include
$(PROJECT_DIR)/libs/protobuf/include/google
$(PROJECT_DIR)/libs/protobuf/include/google/compiler
$(PROJECT_DIR)/libs/protobuf/include/google/compiler/cpp
$(PROJECT_DIR)/libs/protobuf/include/google/io
$(PROJECT_DIR)/libs/protobuf/include/google/stubs
$(PROJECT_DIR)/libs/protobuf/include/google/util
You should see the numbered landmarks and bounding rectangle on a white screen.
Press 'SPACE' to use your hand to swat around some particles.
Note: The example runs on the cpu, so it's a little slow. But the framerate improves a bit once a hand is detected.