Which object a person is pointing at? Detect it by using YOLO, Openpose and depth image (under customized scene).
Detect which object a person's right arm is pointing to.
Contents:
The workflow of this project is:
You may download the video demo here or watch the youtube here. The annotations of the demo are illustrated below:
Environment:
Ubuntu 18.04, ROS melodic, python2.
First, install some common python libraries:
pip2 install --user open3d
The package is here https://github.com/felixchenfy/ros_openpose_rgbd. It uses Openpose and depth image to visualize the 3D humans joints in rviz. Please download it:
cd ~/catkin_ws/src # You catkin workspace. The dir name "catkin_ws" doesn't matter.
git clone https://github.com/felixchenfy/ros_openpose_rgbd
Please read its README, and make sure you can run its example code:
rosrun ros_openpose_rgbd detect_and_draw_joints.py
roslaunch ros_openpose_rgbd run_rviz.launch
I'm using the official CMU's Openpose. You should have installed it when testing ros_openpose_rgbd
Its github is https://github.com/leggedrobotics/darknet_ros. Please read its tutorial and install it :
cd catkin_ws/src
git clone --recursive [email protected]:leggedrobotics/darknet_ros.git
cd ..
catkin_make -DCMAKE_BUILD_TYPE=Release
I'm using its YOLOv3 model, so download the weights file:
cd catkin_ws/src/darknet_ros/darknet_ros/yolo_network_config/weights/
wget http://pjreddie.com/media/files/yolov3.weights
cd ~/catkin_ws/src
git clone https://github.com/felixchenfy/ros_pub_and_sub_rgbd_and_cloud
cd ros_pub_and_sub_rgbd_and_cloud
chmod a+x pub_rgbd_and_cloud.py
# Step 1. Publish data and start rviz.
roslaunch ros_3d_pointing_detection publish_test_rgbd_data.launch
# Step 2. Start object detection.
roslaunch ros_3d_pointing_detection darknet_ros.launch
# Step 3. Start Openpose and pointing detection.
rosrun ros_3d_pointing_detection detect_and_draw_joints.py \
--data_source rostopic \
--detect_object true \
--ros_topic_color camera/color/image_raw \
--ros_topic_depth camera/aligned_depth_to_color/image_raw \
--ros_topic_camera_info camera/color/camera_info
The data publish rate is set as 1 image/second. Please read launch/publish_test_rgbd_data.launch, and change publish settings in config/publish_test_rgbd_data_config.yaml.
Run on Realsense If you want to read data from D435, you may replace step 1 with this command:
roslaunch ros_3d_pointing_detection run_realsense.launch
Speed this project runs pretty slow -- Not real time, only 6 frames/second on my RTX2070. If the code is rewrote in c++ instead of python, it might be able to reach 10 fps.