DEEPWAY V2
Autonomous navigation for blind people.
This project is version 2 of deepWay. You can have a look at this video
Before proceeding look at demo of version2 at
DeepWay v.2
A question you may have in mind
If I already had a repository, why make another ?
Since V1 was based on keras, and I don't like tensorflow much, so for more controll I have shifted to pytorch.
It is complete redesign.
How is it better than others:
- Cost effective: I made the entire project in less than RS 10000 which less than $200.
- Blind people generally develop other senses like hearing very well. Taking away one of their senses by using earphones would not have been nice so I am providing information to the blind person using haptic feedback.
- Everything runs on a edge device--> Nvidiai Jetson Nano.
Hardware requirements
- Nvidia Jetson Nano.
- Arduino nano.
- 2 servo motors.
- USB audio adapter(as jetson nano does not have a audio jack)
- Ethernet cable
- Webcamera
- Power adapter for nvidia jetson nano
- 3D printer.(Not necessary)
- A latop(Nvidia GPU preferred) or any cloud service provider.
Software requirements(If running on Laptop)
- Ubuntu machine(16.04 preferred).
- Install anaconda.
- Install the required dependencies. Some libraries like pytorch, opencv would require a little extra attention.
conda env create -f deepWay.yml
- You can not clone the repository.
- Change the COM number in the arduno.py file according to your system.
- Connect the Ardunio nano and USB audio adapter to your PC.
- Change CAM to video path instead of 0 for running the system on video.
- Compile and run arduino Nano code in the arduino nano.
- Run blindrunner.py
Software Requirements(Jetson nano)
- Follow these instructions for starting up with Jetson nano.
- For connecting headless with jetson nano(using ethernet cable).
ifconfig
Check inet addresss
nmap -sn inet_address/24 --> will return live ip address.
ssh machine_name@ip
Enter password
Now you can connect switch on desktop sharing
Now connect to jetson using Reminna.
- Now install all the required dependicies(it is a time comsuming task, don't loose hope).
1. Collecting dataSet and Generating image masks.
I made videos of roads and converted those videos to jpg's. This way I collected a dataSet of approximately 10000 images.I collected images from left, right and center view(So automatically labelled). e.g:
For Unet, I had to create binary masks for the input data, I used LabelBox for generating binary masks. (This took a looooooooot of time). A sample is as follows->
For downloading the labelled data from Labelbox, I have made a small utility named "downloader.py"
2. Model training
I trained a lane detection model which would predict the lane(left,center,right) I am walking in.
The loss vs iteration curve is as follows:
I trained a U-Net based model for road segmentation on Azure.
The loss(pink:traning, green:validation) vs iterations curve is as follows.
though the loss is less the model does not perform well
I trained a model in keras with a different architecture performs really well
Loss vs iterations curve is:
3. 3D modelling and printing
My friend Sangam Kumar Padhi helped me with CAD model. You can look at it here
4. Electronics on the spectacles
The electronics on the spectacles are very easy. It is just two servo motors connected with a ardunio nano. The arduino nano receives signal from the jetson(using pyserial library), and Arduino Nano controls the servo motors.
5. Pedestrian detection using Mobilenet V1 SSD
I am using Hao repository for pedestrian detection .
It runs at approx 10 FPS(individulaly) on the jetson nano and the accuracy is also pretty good.
Results
- Model for lane detection works really well, it runs at approx 25 fps on the jetson nano. I think it is the really good for an 30 FPS camera.
- The road segmentation model does not work as good as the lane detection one. Though the loss decreases very much but still the output is not as expected.@ptrblck suggests to use focal loss or weighted loss.
- I trained another model using a different unet architecture in keras and it performs really well.
- I am doing naive approach for path planning right now. Assumption: Only people will be on the streets.
- For pedestrian detection, I am using Mobilenet V1 SSD. Thanks to Hao. It runs at 5FPS. I tried to run object detection models in jetson-inference. If runs at approx 15 FPS, but I was not able to capture frames using opencv while gstreamer was also capturing frames.
- To cope up with the slow frame-rate of Mobilenet, I combined it with object tracking. Object detection ran once in 3 seconds to re seed the object-tracker.
- Overall the system runs at 3 FPS. I am running my nano at 5W with a usb type B power supply of 5V 2 Amp. Running the jetson in 10W mode using a 5V 4A supply would further improve performance.
The project is complete from my side, but there is other functionality I desire to add in future
TODO
People to Thank
-
Army Institute of Technology (My college).
-
Prof. Avinash Patil,Sangam Kumar Padhi, Sahil and Priyanshu for 3D modelling and printing.
-
Shivam sharma and Arpit for data labelling.
-
Nvidia for providing a free jetson kit.
-
LabelBox: For providing me with the free license of their Amazing Prodcut.
References
-
Pytorch
-
PyimageSearch
-
Pytorch Community--> special mention @ptrblck
-
AWS
-
U-Net
-
U-Net implementation(usuyama)
-
U-Net implementation(Heet Sankesara)
-
Hao pytorch-ssd
-
Jetson-hacks
-
Tensorflow
-
Keras
-
Advanced lane detection-Eddie Forson
Citations
Labelbox, "Labelbox," Online, 2019. [Online]. Available: https://labelbox.com
Liked it
Tell me if you liked it by giving a star. Also check out my other repositories, I always make cool stuff. I even have a youtube channel "reactor science" where I post all my work.
Read about v1 at:
-
Geospatial Magazine
-
Hackster
-
Anyline
-
Arduino blog