Intelligent Vehicle Perception Based on Inertial Sensing and Artificial Intelligence

This project aims to develop solutions for vehicular perception through inertial sensor signals and Artificial Intelligence models. Vehicular perception comprises exteroception and proprioception. Exteroception aims to understand the environment outside the vehicle, recognizing the road features on which it travels. These features include transient events in the form of anomalies and obstacles, such as potholes, cracks, speed bumps, etc.; and persistent events, such as surface type, conservation condition, and the road surface quality. On the other hand, proprioception aims to understand vehicular movements to identify their own behavior. These identifications can also be transient in the form of driving events, such as lane change, braking, skidding, aquaplaning, turning right or left; and persistent, as a safe or dangerous driving behavior profile. This situational information (perceptions) has wide applicability in Intelligent Transport Systems (ITS) such as Advanced Driver Assistance Systems (ADAS) and autonomous vehicles.

For the development of this project, we collect nine datasets using GPS, camera, inertial sensors (accelerometers and gyroscopes), magnetometer, and temperature sensor. These data were produced with contextual variations including three different vehicles, driven by three different drivers, traveling through three different environments. To recognize and classify the vehicular perception patterns, we have developed several models based on Artificial Intelligence, among Classical Machine Learning and Deep Learning approaches. Below we describe the datasets produced, models developed and the results obtained, together with published scientific papers and source-codes.

Table of Contents

• Vehicular Perception Research
• Passive Vehicular Sensors Dataset
• Data Classes
• Best Models
• Aplications
• How To Cite
• License

Vehicular Perception Research

The project is active and we are currently developing new models for new perception pattern recognition. Below are described the research progress, in chronological order of research development. At the Research Gate you can also find the published scientific papers and request a full text for free.

Research in English

Vehicular Perception Based on Inertial Sensing: a Systematic Review

In this paper, we describe the state-of-the-art vehicle perception produced through inertial sensors and Artificial Intelligence techniques. Through a literature review, we compiled the data extracted from the selected studies and described each paper in detail and chronological order of publication.
Access here

Vehicular Perception Based on Inertial Sensing: A Structured Mapping of Approaches and Methods

In this paper, we present a structured literature mapping of the state-of-the-art vehicle perception produced through inertial sensors and Artificial Intelligence techniques. We describe a structured, approach, and technologies-oriented panorama of this field.
Access here

Road Surface Type Classification Based on Inertial Sensors and Machine Learning: A Comparison Between Classical and Deep Machine Learning Approaches For Multi-Contextual Real-world Scenarios

In this research, we developed models for the road surface type classification, classifying between segments of dirt, cobblestone, and asphalt roads. We applied classical techniques of K-Means Clustering (KMC), Support Vector Machine (SVM), and K-Nearest Neighbors (KNN); and Deep Learning techniques based on Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM), and hybrid LSTM-CNN. We analyzed in the time domain the data collected near and below the suspension, in addition to the impact of the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here

Multi-Contextual and Multi-Aspect Analysis for Road Surface Type Classification Through Inertial Sensors and Deep Learning

In this research, we developed models for the road surface type classification, classifying between segments of dirt, cobblestone, and asphalt roads. We applied Deep Learning techniques based on Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), and Long Short-Term Memory Network (LSTM). We analyzed various aspects, such as the influence of the vehicle data collection placement, the analysis domain, the model input features, and the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here

Speed Bump Detection Through Inertial Sensors and Deep Learning in a Multi-Contextual Analysis

In this research, we developed models for the speed bump detection in segments of cobblestone and asphalt pavements. We applied Deep Learning techniques based on Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), Long Short-Term Memory Network (LSTM), Convolutional Neural Network Long Short-Term Memory (CNN-LSTM), and Convolutional Long Short-Term Memory (ConvLSTM). We analyzed some aspects, such as the influence of the vehicle data collection placement and the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here

Road Roughness Condition Through Inertial Sensors and Deep Learning

In this research, we developed models to classify the road roughness condition in three quality levels: bad, regular and good road. We applied Deep Learning techniques based on Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), Long Short-Term Memory Network (LSTM), Convolutional Neural Network Long Short-Term Memory (CNN-LSTM), and Convolutional Long Short-Term Memory (ConvLSTM). We analyzed some aspects, such as the influence of the vehicle data collection placement and the data window size. We also evaluate the generability of model learning for unknown contexts, such as an unknown car, driver, or environment.
Access here

Research in Brazilian Portuguese

Percepção Veicular Adaptativa Baseada em Sensoriamento Inercial e Deep Learning

Master's thesis. Access here

Classificacão de Qualidade de Superfície de Pista Baseado em Sensoriamento Inercial e Logica Fuzzy

Conference paper. Access here

ADAPD - Sensor Adaptativo de Tempo Real para Buracos em Rodovias Baseado em Vibração

Conference paper. Access here

Passive Vehicular Sensors Dataset (PVS)

The nine datasets collected are available for download at Kaggle and Arquivos UFSC. To collect and preprocess the raw data that resulted in the PVS datasets, the following projects were used:

• MPU-9250 Sensors Data Collect

  • Source-code used to manipulate the sensors in the data collection.

• MPU-9250 and GPS Raw Data Pre-Processing

  • Source-code used for initial adjustments to inertial sensor raw data, such as combining with GPS data and creating data class labels.

• MPU-9250 Data Plot Video Creator

  • Source-code to create videos with data plot of speed and signals from inertial sensors.

For data collection, we use several passive approach sensors, detailed in the following table:

All the hardware equipment was attached to the vehicle as shown in the next figure. The camera was placed on the outside car roof (1), and the GPS receiver was placed internally on the dashboard (2). Two networks with MPU-9250 modules were distributed in the vehicle to get data coming from different points. Thus, each end of the front axle (right and left side) received one of the sensor networks, where a module was attached to the control arm (4), located below and near to the vehicle’s suspension system; another module was placed above and near the suspension system, attached to the body immediately above the tire (3); and a third module was attached to the vehicle’s dashboard (2), inside the cabin.

The data were produced in three different vehicles, with three different drivers, in three different environments in which there are three different surface types, in addition to variations in conservation state and presence of obstacles and anomalies, such as speed bumps and potholes. The following table details the data collection contexts.

Each dataset consists of the following files:

You can find the playlist of videos side by side on Youtube.

Data Classes

The data classes are available in the dataset_labels.csv file were built in one-hot-encoded form. The available labs are detailed below. In the folder Best Models is the Jupyter Notebook PVS - Data Exploration.ipynb in which you can explore the data through maps and tables with quantitative samples and distribution of data classes. Access here.

Road Surface Type Labels

Road Surface Condition

Road Roughness Condition

Speed Bump Types

Best Models

For those who want to use the best models produced in this research in a simplified way, they are available in the Best Models folder, access here. We provide the following best models:

• PVS - Data Exploration
• Road Surface Type Classification
• Road Roughness Condition Classification
• Speed Bump Classification

Aplications

The recognized and classified patterns of vehicular perception have wide applicability. Below are some applications that can use this information:

• Autonomous Vehicles
• Advanced Vehicle Control Systems
• Advanced Driver Assistance Systems
• Advanced Public Transport Management Systems
• Advanced Traffic Management Systems
• Advanced Traveler Information Systems
• Vehicle Black Box
• Evaluation Software for Control and Quality Assurance in Civil Engineering
• Mobile Crowdsensing Applications

How To Cite

To cite PVS Datasets, use the reference below:

J. Menegazzo and A. von Wangenheim, "Multi-Contextual and Multi-Aspect Analysis for Road Surface Type Classification Through Inertial Sensors and Deep Learning," 2020 X Brazilian Symposium on Computing Systems Engineering (SBESC), Florianopolis, 2020, pp. 1-8, doi: 10.1109/SBESC51047.2020.9277846.

@INPROCEEDINGS{9277846,
  author={J. {Menegazzo} and A. {von Wangenheim}},
  booktitle={2020 X Brazilian Symposium on Computing Systems Engineering (SBESC)}, 
  title={Multi-Contextual and Multi-Aspect Analysis for Road Surface Type Classification Through Inertial Sensors and Deep Learning}, 
  year={2020},
  volume={},
  number={},
  pages={1-8},
  doi={10.1109/SBESC51047.2020.9277846}
}

License

This project is under Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0). Please see License File for more information.