RELLIS-3D: A Multi-modal Dataset for Off-Road Robotics

    

Peng Jiang¹, Philip Osteen², Maggie Wigness² and Srikanth Saripalli¹
1. Texas A&M University;  2. CCDC Army Research Laboratory
[Website] [Paper] [Github]

Updates

• 11/26/2020 v1.0 release
• 02/25/2021 improve camera and lidar calibration parameter
• 03/04/2021 update ROS bag with new tf (v1.1 release)
• 06/14/2021 fix missing labels of point cloud and fix wrong poses
• 01/24/2022 add Velodyne point clouds in kitti format and labels transfered from Ouster

Overview

Semantic scene understanding is crucial for robust and safe autonomous navigation, particularly so in off-road environments. Recent deep learning advances for 3D semantic segmentation rely heavily on large sets of training data; however, existing autonomy datasets represent urban environments or lack multimodal off-road data. We fill this gap with RELLIS-3D, a multimodal dataset collected in an off-road environment containing annotations for 13,556 LiDAR scans and 6,235 images. The data was collected on the Rellis Campus of Texas A&M University and presents challenges to existing algorithms related to class imbalance and environmental topography. Additionally, we evaluate the current state of the art deep learning semantic segmentation models on this dataset. Experimental results show that RELLIS-3D presents challenges for algorithms designed for segmentation in urban environments. Except for the annotated data, the dataset also provides full-stack sensor data in ROS bag format, including RGB camera images, LiDAR point clouds, a pair of stereo images, high-precision GPS measurement, and IMU data. This novel dataset provides the resources needed by researchers to develop more advanced algorithms and investigate new research directions to enhance autonomous navigation in off-road environments.

Recording Platform

• Clearpath Robobtics Warthog

Sensor Setup

• 64 channels Lidar: Ouster OS1
• 32 Channels Lidar: Velodyne Ultra Puck
• 3D Stereo Camera: Nerian Karmin2 + Nerian SceneScan (Sensor Configuration)
• RGB Camera: Basler acA1920-50gc + Edmund Optics 16mm/F1.8 86-571
• Inertial Navigation System (GPS/IMU): Vectornav VN-300 Dual Antenna GNSS/INS

Folder structure

Rellis-3D
├── pt_test.lst
├── pt_val.lst
├── pt_train.lst
├── pt_test.lst
├── pt_train.lst
├── pt_val.lst
├── 00000
      ├── os1_cloud_node_kitti_bin/             -- directory containing ".bin" files with Ouster 64-Channels point clouds.   
      ├── os1_cloud_node_semantickitti_label_id/     -- containing, ".label" files for Ouster Lidar point cloud with  manually labelled semantics label
      ├── vel_cloud_node_kitti_bin/             -- directory containing ".bin" files with Velodyne 32-Channels point clouds.   
      ├── vel_cloud_node_semantickitti_label_id/     -- containing, ".label" files for Velodyne Lidar point cloud transfered from Ouster point cloud.
      ├── pylon_camera_node/    -- directory containing ".png" files from the color   camera.  
      ├── pylon_camera_node_label_color -- color image lable
      ├── pylon_camera_node_label_id -- id image lable
      ├── calib.txt             -- calibration of velodyne vs. camera. needed for projection of point cloud into camera.  
      └── poses.txt             -- file containing the poses of every scan.

Download Link on BaiDu Pan:

链接: https://pan.baidu.com/s/1akqSm7mpIMyUJhn_qwg3-w?pwd=4gk3 提取码: 4gk3 复制这段内容后打开百度网盘手机App，操作更方便哦

Annotated Data:

Ontology:

With the goal of providing multi-modal data to enhance autonomous off-road navigation, we defined an ontology of object and terrain classes, which largely derives from the RUGD dataset but also includes unique terrain and object classes not present in RUGD. Specifically, sequences from this dataset includes classes such as mud, man-made barriers, and rubble piles. Additionally, this dataset provides a finer-grained class structure for water sources, i.e., puddle and deep water, as these two classes present different traversability scenarios for most robotic platforms. Overall, 20 classes (including void class) are present in the data.

Ontology Definition (Download 18KB)

Images Statics:

Note: Due to the limitation of Google Drive, the downloads might be constrained. Please wait for 24h and try again. If you still can't access the file, please email [email protected] with the title "RELLIS-3D Access Request"..

Image Download:

Image with Annotation Examples (Download 3MB)

Full Images (Download 11GB)

Full Image Annotations Color Format (Download 119MB)

Full Image Annotations ID Format (Download 94MB)

Image Split File (44KB)

LiDAR Scans Statics:

LiDAR Download:

Ouster LiDAR with Annotation Examples (Download 24MB)

Ouster LiDAR with Color Annotation PLY Format (Download 26GB)

The header of the PLY file is described as followed:

element vertex
property float x
property float y
property float z
property float intensity
property uint t
property ushort reflectivity
property uchar ring
property ushort noise
property uint range
property uchar label
property uchar red
property uchar green
property uchar blue

To visualize the color of the ply file, please use CloudCompare or Open3D. Meshlab has problem to visualize the color.

Ouster LiDAR SemanticKITTI Format (Download 14GB)

To visualize the datasets using the SemanticKITTI tools, please use this fork: https://github.com/unmannedlab/point_labeler

Ouster LiDAR Annotation SemanticKITTI Format (Download 174MB)

Ouster LiDAR Scan Poses files (Download 174MB)

Ouster LiDAR Split File (75KB)

Velodyne LiDAR SemanticKITTI Format (Download 5.58GB)

Velodyne LiDAR Annotation SemanticKITTI Format (Download 143.6MB)

Calibration Download:

Camera Instrinsic (Download 2KB)

Basler Camera to Ouster LiDAR (Download 3KB)

Velodyne LiDAR to Ouster LiDAR (Download 3KB)

Stereo Calibration (Download 3KB)

Calibration Raw Data (Download 774MB)

Benchmarks

Image Semantic Segmenation

models	sky	grass	tr ee	bush	concrete	mud	person	puddle	rubble	barrier	log	fence	vehicle	object	pole	water	asphalt	building	mean
HRNet+OCR	96.94	90.20	80.53	76.76	84.22	43.29	89.48	73.94	62.03	54.86	0.00	39.52	41.54	46.44	9.51	0.72	33.25	4.60	48.83
GSCNN	97.02	84.95	78.52	70.33	83.82	45.52	90.31	71.49	66.03	55.12	2.92	41.86	46.51	54.64	6.90	0.94	44.18	11.47	50.13

LiDAR Semantic Segmenation

models	sky	grass	tr ee	bush	concrete	mud	person	puddle	rubble	barrier	log	fence	vehicle	object	pole	water	asphalt	building	mean
SalsaNext	-	64.74	79.04	72.90	75.27	9.58	83.17	23.20	5.01	75.89	18.76	16.13	23.12	-	56.26	0.00	-	-	40.20
KPConv	-	56.41	49.25	58.45	33.91	0.00	81.20	0.00	0.00	0.00	0.00	0.40	0.00	-	0.00	0.00	-	-	18.64

Benchmark Reproduction

To reproduce the results, please refer to here

ROS Bag Raw Data

Data included in raw ROS bagfiles:

Topic Name	Message Tpye	Message Descriptison
/img_node/intensity_image	sensor_msgs/Image	Intensity image generated by ouster Lidar
/img_node/noise_image	sensor_msgs/Image	Noise image generated by ouster Lidar
/img_node/range_image	sensor_msgs/Image	Range image generated by ouster Lidar
/imu/data	sensor_msgs/Imu	Filtered imu data from embeded imu of Warthog
/imu/data_raw	sensor_msgs/Imu	Raw imu data from embeded imu of Warthog
/imu/mag	sensor_msgs/MagneticField	Raw magnetic field data from embeded imu of Warthog
/left_drive/status/battery_current	std_msgs/Float64
/left_drive/status/battery_voltage	std_msgs/Float64
/mcu/status	warthog_msgs/Status
/nerian/left/camera_info	sensor_msgs/CameraInfo
/nerian/left/image_raw	sensor_msgs/Image	Left image from Nerian Karmin2
/nerian/right/camera_info	sensor_msgs/CameraInfo
/nerian/right/image_raw	sensor_msgs/Image	Right image from Nerian Karmin2
/odometry/filtered	nav_msgs/Odometry	A filtered local-ization estimate based on wheel odometry (en-coders) and integrated IMU from Warthog
/os1_cloud_node/imu	sensor_msgs/Imu	Raw imu data from embeded imu of Ouster Lidar
/os1_cloud_node/points	sensor_msgs/PointCloud2	Point cloud data from Ouster Lidar
/os1_node/imu_packets	ouster_ros/PacketMsg	Raw imu data from Ouster Lidar
/os1_node/lidar_packets	ouster_ros/PacketMsg	Raw lidar data from Ouster Lidar
/pylon_camera_node/camera_info	sensor_msgs/CameraInfo
/pylon_camera_node/image_raw	sensor_msgs/Image
/right_drive/status/battery_current	std_msgs/Float64
/right_drive/status/battery_voltage	std_msgs/Float64
/tf	tf2_msgs/TFMessage
/tf_static	tf2_msgs/TFMessage
/vectornav/GPS	sensor_msgs/NavSatFix	INS data from VectorNav-VN300
/vectornav/IMU	sensor_msgs/Imu	Imu data from VectorNav-VN300
/vectornav/Mag	sensor_msgs/MagneticField	Raw magnetic field data from VectorNav-VN300
/vectornav/Odom	nav_msgs/Odometry	Odometry from VectorNav-VN300
/vectornav/Pres	sensor_msgs/FluidPressure
/vectornav/Temp	sensor_msgs/Temperature
/velodyne_points	sensor_msgs/PointCloud2	PointCloud produced by the Velodyne Lidar
/warthog_velocity_controller/cmd_vel	geometry_msgs/Twist
/warthog_velocity_controller/odom	nav_msgs/Odometry

ROS Bag Download

The following are the links for the ROS Bag files.

• Synced data (60 seconds example 2 GB): includes synced /os1_cloud_node/points, /pylon_camera_node/camera_info and /pylon_camera_node/image_raw
• Full-stack Merged data:(60 seconds example 4.2 GB): includes all data in above table and extrinsic calibration info data embedded in the tf tree.
• Full-stack Split Raw data:(60 seconds example 4.3 GB): is orignal data recorded by rosbag record command.

Sequence 00000: Synced data: (12GB) Full-stack Merged data: (23GB) Full-stack Split Raw data: (29GB)

Sequence 00001: Synced data: (8GB) Full-stack Merged data: (16GB) Full-stack Split Raw data: (22GB)

Sequence 00002: Synced data: (14GB) Full-stack Merged data: (28GB) Full-stack Split Raw data: (37GB)

Sequence 00003:Synced data: (8GB) Full-stack Merged data: (15GB) Full-stack Split Raw data: (19GB)

Sequence 00004:Synced data: (7GB) Full-stack Merged data: (14GB) Full-stack Split Raw data: (17GB)

ROS Environment Installment

The ROS workspace includes a plaftform description package which can provide rough tf tree for running the rosbag.

To run cartographer on RELLIS-3D please refer to here

Full Data Download:

Access Link

Citation

@misc{jiang2020rellis3d,
      title={RELLIS-3D Dataset: Data, Benchmarks and Analysis}, 
      author={Peng Jiang and Philip Osteen and Maggie Wigness and Srikanth Saripalli},
      year={2020},
      eprint={2011.12954},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Collaborator

License

All datasets and code on this page are copyright by us and published under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.

Related Work

SemanticUSL: A Dataset for Semantic Segmentation Domain Adatpation

LiDARNet: A Boundary-Aware Domain Adaptation Model for Lidar Point Cloud Semantic Segmentation

A RUGD Dataset for Autonomous Navigation and Visual Perception inUnstructured Outdoor Environments