Autonomous RC-Car racing competition in HKUST
Autorace provides hardware and example codes to achieve vision-based autonomous racing on RC-Cars. It is developed by RAM-LAB to support the 1st autonomous RC-Car racing competition in Hong Kong University of Science and Technology (HKUST). The competition data is April 1, 2021.
Event Collaborators: School of Engineering, Robotics Institue (RI), Robotics and Multiperception Lab (RAM-LAB), Intelligent Autonomous Driving Center, Entrepreneurship Center
Keywords: autonomous racing, visual navigation, artifical intelligence, deep learning.
Maintaners: Peide Cai [email protected] Supervisor: Prof. Ming Liu [email protected]
If you meet with any problems, feel free to create an issue to let me and other participants know, then we can solve it together.
If you like the project, give it a star ⭐. It means a lot to the people maintaining it 🧙
📰 Successful end to the first HKUST-Kaisa Autonomous RC Car Racing Competition
Coding language: Python3
Deep learning framework: PyTorch 1.6
On-board computer: Jetson Nano B1
Complete pipeline of data collection, model training and testing
Easy to use and DIY
What can you achieve 🤔?
The RC-Car is named JetRacer, a high speed AI racing robot powered by Jetson Nano.
Some parts of the user guides on software in the above wiki (provided by the vendor) are out-of-date (you may meet different errors during tests). Please follow this repository for system and software installation in the following sections, and just take the above wiki as a reference.
Slides and Assemble Video: Link
This is for data collection and model deployment, covered in our 2nd workshop with slides and videos.
NVIDIA Jetson Nano lets you bring incredible new capabilities to millions of small, power-efficient AI systems. It is also the perfect tool to start learning about AI and robotics in real-world settings, with ready-to-try projects and the support of an active and passionate developer community.
To use this mainboard, we need to install a software development kit on it, which is named JetPack 👇
Based on the introduction in https://developer.nvidia.com/jetpack-sdk-44-archive
JetPack SDK is the most comprehensive solution for building AI applications. It includes the latest Linux Driver Package (L4T) with Linux operating system named Ubuntu (version: 18.04) and CUDA-X accelerated libraries and APIs for Deep Learning, Computer Vision, Accelerated Computing and Multimedia. It also includes samples, documentation, and developer tools for both host computer and developer kit.
In this project we use JetPack 4.4.
JetPack installation is quite simple: Flash the image to a microSD card -> Connect Nano to a display and boot the system (Ubuntu 18.04) -> Finish initialization. Step-by-step instructions are from Page 4 - Page 23 in this slides from workshop#2. Then you will enter the Ubuntu 18.04 system, which looks like the following:
Remember do not upgrade the system after installation, even the system reminds you of that. Because some of our library dependencies rely on the current version of jetpack, which is 4.4.
After installing the Ubuntu 18.04 system with JetPack4.4, we provide a system configuration named JetCard to make it easy to get started with AI on Jetson Nano. Simply execute the provided script and all external library dependencies will be installed automatically.
JetCard comes pre-loaded with:
After configuring your system using JetCard, you can get started prototyping AI projects from your web browser in Python3.
After you install JetPack 4.4 on the SD card, boot the system (Ubuntu 18.04) and open a terminal by pressing Ctrl+Alt+T
, and then do the followings to use JetCard to quickly configure your system:
$ cd ~
$ git clone https://github.com/caipeide/jetcard
$ cd jetcard
$ sh ./install.sh <password>
The whole installation will cost about 40 min. After that the script will ask you to reboot. Take a look at the ip address shown on the display (10.79.157.13 in this case).
Now you can disconnect the HDMI port, keyboard and mouse on the jetson nano and start remote development: Open a browser on your own laptop and enter 10.79.157.13:8888
for remote connection to and development on the car. (the <password>
you set earlier will be asked to enter the jupyterlab)
Finally, we clone this project to the RC-Car:
$ cd ~/projects/donkeycar
$ pip install -e .
$ cd ~
$ git clone https://github.com/caipeide/autorace
To ensure that the Jetson Nano doesn't draw more current than the battery pack can supply, place the Jetson Nano in 5W mode by calling the following command.
$ sudo nvpmodel -m1
$ sudo nvpmodel -q
m1: 5W power mode, m2: 10W power mode (MAXN)
A host PC is needed for model training
For participants who do not have a NVIDIA graphics card (GPU) in their computer, they can apply for using our server to train their models, and skip the following section 2.2. In this way, the laptop will be only used for remote connection to the RC-Car, which is doable on any system including Windows, MacOS and Ubuntu.
The following instructions refer to steps from 2 Ways to Install Nvidia Driver on Ubuntu 18.04 (GUI & Command Line). Here we simply choose to use graphical user interface (GUI) for installing the Nvidia driver.
First, go to system settings
> details
and check what graphics card your computer is using. By default, your integrated graphics card (Intel HD Graphics) is being used.
Then open softare & updates
program from you application menu. Click the additional drivers
tab. You can see what driver is being used for Nvidia card. If you can not see nvidia-driver-xxx
in the list, open a terminal and do sudo apt update
, then re-open this window, and available drivers can be updated.
As you can see many driver versions are available for the GeForce RTX 2080 Ti card on our server. Here we use the nvidia-driver-440
and it works fine for our model training. There might be some other drivers for your particular Nvidia card. Click Apply Changes
button to install the driver.
After it’s installed, reboot your computer for the change to take effect. After that, go to system settings
> details
, you will see Ubuntu is using Nvidia graphics card.
You can open a terminal and do nvidia-smi
to check the running information of your GPU:
We provide a script install_host.sh
for you to quickly configure your host PC (or server account) with all necessary dependencies for model training.
By executing the script, the following packages will be automatically installed:
autorace
will be automatically createdAfter configuring your system, you can get started transmitting data between your host PC (or server account) and your RC-Car (via ssh
), and using the collected dataset to train your own self-driving car (will be introduced in the next section Train a Self-driving Car)
$ sudo apt update
$ sudo apt install openssh-server
$ sudo apt install git
If you are using the server, connect to it with
ssh
first in a new terminal on your PC (or on the RC-Car with jupyterlab), for example, If you are using Ubuntu, open a new terminal (Ctrl+Alt+T
) andssh -p <port_number> <server_account_name>@<server_ip_address>
. Then do the followings in the terminal for host configuration.
$ cd ~
$ git clone https://github.com/caipeide/autorace
$ cd autorace
$ sh ./install_host.sh
$ source ~/.bashrc
$ conda create -n autorace python=3.6 -y
$ conda activate autorace
$ sh ./install_host_continue.sh
All of the car's settings are in the config.py
and myconfig.py
scripts. config.py
stores the default values for all parameterss, and you can adjust these settings in myconfig.py
by uncommenting related lines and changing their values. When the main program manage.py
starts, it will first read config.py
and then override the variables with values you set in myconfig.py
.
You can edit this file using two methods:
nano ~/autorace/myconfig.py
Ctrl+S
to save.Here uncomment means deleting the #
symbol at the beginning of lines. In most editors, you can toggle between the comment and uncomment status by pressing Ctrl+/
.
Make sure your car is off the ground to prevent a runaway situation.
donkey calibrate --channel 1 --bus=1
370
when prompted for a PWM value. Then you should hear your ESC beep indicating that it's calibrated. If not, adjust this value a little, or else the motor won't work.Reverse on RC cars is a little tricky because the ESC must receive a reverse pulse, zero pulse, reverse pulse to start to go backwards. To calibrate a reverse PWM setting...
330
, then the zero throttle value you find above, then the reverse value again.Enter these values in myconfig.py
script as THROTTLE_FORWARD_PWM
, THROTTLE_STOPPED_PWM
, and THROTTLE_REVERSE_PWM
.
donkey calibrate --channel 0 --bus=1
myconfig.py
script as STEERING_RIGHT_PWM
and STEERING_LEFT_PWM
. Note that the default vaules are STEERING_LEFT_PWM = 460
,
STEERING_RIGHT_PWM = 290
, saved in config.py
.Note: You need to make sure that the value (STEERING_LEFT_PWM + STEERING_RIGHT_PWM)*0.5
can make the front wheels face straight ahead, then the car can drive in a straight line with steering = 0
.
Now that you have your car roughly calibrated you can try driving it to verify that it drives as expected. Here's how to fine tune your car's calibration.
python manage.py drive
in this folder.<car_ip_address>:8887
in a browser (Tested on Chrome). The following is what you will see: camera video streams, adjustable control values and driving modes. Now you can press J
or L
on keyboard to adjust the car steering to left or right, and use I
and K
to adjust the throttle values.I
a few times to get the car to go forward. If it goes straight, that is good and you can move on to the next section; if not, adjust your values of STEERING_LEFT_PWM
and STEERING_RIGHT_PWM
until a straight driving trajectory can be achieved.Note: Too small throttle values, e.g., 0.1, may not be enough to drive the car. You need to increase it a bit.
Only for basic testing. If you want to collect data in a more flexible way, use the joystick introduced in the next section Driving with Physical Joystick Controller
This controller provides a UI window accessible at <car_ip_address>:8887
. After you run python manage.py drive
this module will be loaded.
Pilot mode - Choose this if the pilot should control the angle and/or throttle. It has three options:
local_angle
will only let the pilot model control the angle of the car.local_pilot
will let the pilot model control both the angle and throttle.user_mode
where you have full control over the car using keyboard shortcuts or mouse.Notes
manage.py
, which will be introduced in Model Testing.pilot
modes to user_mode
. You can use this function to quickly stop your self-driving car.Recording - Press record data to start recording images, steering angels and throttle values. By default the data will be automatically recorded if throttle is not zero in user_mode
. If the car is in pilot mode, you can use this function to manually record some self-driving data. Note do not use the data from a pilot mode to train you netowrk. Training details will be covered in Model Training.
Throttle mode - Option to set the throttle as constant. This is used in races if you have a pilot that will steer but doesn't control throttle.
Max throttle - Select the maximum throttle for user_mode
.
R
: toggle recordingI
: increase throttle by 0.05K
: decrease throttle by 0.05J
: turn leftL
: turn rightNote: Throttle range and steering range are both [-1.0, 1.0]. Throttle values that are less than 0 indicate reversing.
USE_FPV = True
in myconfig.py
to monitor the camera video streams. The published FPV images are accessible in <car_ip_address>:8890
$ cd ~/autorace
$ python manage.py drive --js
Optionally, if you want joystick use to be sticky and don't want to add the
--js
each time, modify yourmyconfig.py
so thatUSE_JOYSTICK_AS_DEFAULT = True
+------------------+--------------------------+
| control | action |
+------------------+--------------------------+
| a_button | toggle_mode |
| b_button | toggle_manual_recording |
| x_button | erase_last_N_records |
| y_button | emergency_stop |
| right_shoulder | increase_max_throttle |
| left_shoulder | decrease_max_throttle |
| options | toggle_constant_throttle |
| left_stick_horz | set_steering |
| right_stick_vert | set_throttle |
| right_trigger | constant_rage_mode |
| left_trigger | constant_gentle_mode |
+------------------+--------------------------+
left_stick_horz
: Left analog stick - Left and right to adjust steeringright_stick_vert
: Right analog stick - Forward to increase forward throttle, and Backward to increase reverse throttle. No operation for zero throttle.toggle_mode
: Switches modes - "User, Local Angle, Local(angle and throttle)"toggle_manual_recording
: Toggle recording of all data even if your car stops with zero throttle. This is disabled by default because a more suitable method auto record on throttle is enabled by default, which means that whenever the throttle is not zero, driving data will be recorded - as long as you are in user mode. The data will be saved in folder data/tub_xx_xx_xx/
erase_last_N_records
: You don't want to use bad data to train you network, such as collisions with walls. This function can erase the data in the last 100 frames to keep your dataset clean.increase_max_throttle
: the max_throttle for joystick control is set to 0.5 by default (when right analog stick is pushed to the front). Press right shoulder to increase the max_throttle by PER_THROTTLE_STEP
if you need more speed. PER_THROTTLE_STEP
is 0.05 by default, you can change this in myconfig.py
.decrease_max_throttle
: Simmilar to the above. Press left shoulder to decrease the max_throttle by PER_THROTTLE_STEP
.toggle_constant_throttle
: Toggle constant throttle. Sets to max throttle.constant_rage_mode
: Start a constant throttle mode with very a large throttle RAGE_THROTTLE = 0.75
. This can be useful at long and straight tracks, where you can quickly accelerate your car. The throttle value can be adjusted in myconfig.py
constant_gentle_mode
: Start a constant throttle mode with a low throttle GENTLE_THROTTLE = 0.45
. This can be useful when RC-Car enters a sharp corner at high speeds, where you need to quickly slow the speed down but not stop. This throttle can also be adjusted in myconfig.py
AUTO_RECORD_ON_THROTTLE = False
in myconfig.py
to disable recording data automatically.mange.py
process to create a new tub session. Set AUTO_RECORD_ON_THROTTLE = True
. The joystick will auto record with any non-zero throttle.Ctrl+C
in the terminal session for your car.data/
folder in the most recent tub folder.After you finish training a car can that can drive solely on the track based on the following procedure, you should return to this part and cooperate with another team to collect data on wheel-to-wheel driving (two cars running on the track). These data will be crucial for your model to learn how to behave in the presence of another car, such as overtaking and slowing down to avoid collisions. Otherwise, it is likely to perform poorly in the main race.
DRIVE_LOOP_HZ
set in myconfig.py
is the max frequency that the drive loop should run. The actual frequency may be less than this if there are many blocking parts, e.g., your designed AI model is too complex to run quickly.user_mode
. If the value CONTROL_NOISE
is set to True
in myconfig.py
(default value is False
), random action noises on steering angle and throttle will be added during your tele-operation. This can help you to collect more divrese data that the car recovers from off-center and off-orientation mistakes. Based on these your trained agent can be more "intelligent". We also provide two scalars THROTTLE_NOISE
and ANGLE_NOISE
to adjust the level of noise. Note with this module acctivated, the data collection process will be difficult.Training a deep neural network on Jetson Nano can be painful (quite slow 🐌). Therefore, we will use more powerful PCs for faster training. The first step is to copy the collected data from your RC-Car to the host PC.
rsync
to copy your cars data/
folder.$ cd ~/autorace
$ rsync -rv --progress --partial <car_account_name>@<car_ip_address>:~/autorace/data ./
$ cd ~/autorace
$ rsync -rv -e 'ssh -p <port_number>' --progress --partial ./data <server_account_name>@<server_ip_address>:~/autorace/
Now you can check the new data in the folder ~/autorace/data/
on your host PC:
$ ssh -p <port_number> <server_account_name>@<server_ip_address> # if you use the server, connect to it via ssh first on your RC-Car
$ ls ~/autorace/data/
The copied data folders will be printed. The following is an example.
In the same terminal you can now run the training script on the latest tub by passing the path to that tub as an argument. For example,
$ cd ~/autorace
$ python manage.py train --model models/resnet18.pth --type resnet18 --tub data/tub_1_20-12-12/,data/tub_2_20-12-12
The trained model will be saved in ~/autorace/models/resnet18.pth
. Optionally you can pass no arguments for the tub, and then all tubs will be used in the default data/
folder.
$ python manage.py train --model models/resnet18.pth --type resnet18
We provide three basic model types for training: linear, rnn and resnet18. Click these hyperlinks for more details. The following image shows the model architecture from Nvidia Self-Driving Car in 2016, which uses a series of convolutional layers and fully connected layers to learn driving behaviors.
You can change these model architectures in ai_drive_models.py
, where linear -> LinearModel
, rnn -> RNNModel
and resnet18 -> LinearResModel
. Note the rnn model runs slowly on the RC-Car, thus it may not be suitable for racing.
Other training parameters such as batchsize, learning rate, etc., can be configured in myconfig.py
. Their default values should work well in most cases.
During training you can run nvidia-smi
in a terminal to check GPU power consumption and memory usage. The following shows a training demo.
After the training is finished, you can view how the training loss and validation loss decrease in models/loss_plot_resnet18.png
.
In previous step we managed to get a model trained on the data. Now is time to move the model back to RC-Car, so we can use it for testing it if it will drive itself.
$ cd ~/autorace
$ rsync -rv --progress --partial ./models/resnet18.pth <car_account_name>@<car_ip_address>:~/autorace/models/
$ cd ~/autorace
$ rsync -rv -e 'ssh -p <port_number>' --progress --partial <server_account_name>@<server_ip_address>:~/autorace/models/resnet18.pth ./models/
The resnet18
model trained above runs about 150 ms/frame (6.7 Hz) on the car, which can cause problems if your car drives fast (not quickly enough to make a decision to take turns -> collision). Therefore, we will accelerate the model as follows on the RC-Car:
$ cd ~/autorace
$ python accel_model.py --model models/resnet18.pth --type resnet18
The process takes about 1 ~ 2 min, and the accelerated model will be saved in models/resnet18_trt.pth
. The inference speed of this model is about 50 ms/frame (20 Hz).
Notes:
torch2trt
library we use. Unfortunately, I have no solution on that.--half
to the above command if you want to save memory usage of the neural network. Then the accelerated model will be using FP16 rather than FP32, allowing deployment of larger networks.Ensure to place the car on the track and power on its motor so that it is ready to drive.
$ python manage.py drive --model models/resnet18_trt.pth --trt --type resnet18
When all modules are ready (1 ~ 2 min to warm up), the program will notice you to press ENTER
to start. Then the car should start to drive on it's own. Congratulations!
Notes:
--half
to the above command if you use that during model acceleration.data/
folder, DO NOT try to open the tub folder and view images through JupyerLab, because it will cost much time to load the files and the GUI may freeze. You can transfer the data to your laptop to view.sudo shutdown
first to shutdown the OS system, then turn off the power.ENTER
to start the car immediately when using autopilot modego to defination
and code suggestions
, you can choose VSCode for development.Visual Studio Code has a high productivity code editor which, when combined with programming language services, gives you the power of an IDE and the speed of a text editor. In this topic, we'll first describe VS Code's language intelligence features (suggestions, parameter hints, smart code navigation) and then show the power of the core text editor.
After installation, you can install the Remote - SSH
extension within VSCode to let your editor connected to your RC-Car. The following is a video demo showing how.