Software Engineering for Self-Driving Cars Guide :car:

Welcome to the self-driving cars guide for computer science students, software engineers, and those looking to develop skills in autonomous vehicle software.

An autonomous car is a vehicle capable of sensing its environment and operating without human involvement. A human passenger is not required to take control of the vehicle at any time, nor is a human passenger required to be present in the vehicle at all. An autonomous car can go anywhere a traditional car goes and do everything that an experienced human driver does. source

Table of Contents

Introduction

• How to Contribute/Guidelines
• Getting Started
• Prerequisite Resources
• Autonomous Vehicles Motivation

Guide

• Introduction
• Software and Hardware
• Developing for Safety
• Deep Learning for Self-Driving Cars
• State Estimation and Localization
• Perception
• Motion Planning
• Testing

Research

• Adversarial Attacks
• Adversarial Defenses
• Vehicle-to-Everything (V2X)

Resources/FAQ

• Resources
• AV Companies
• FAQ

How to Contribute

:white_check_mark: This is an open source guide, so contributions are very much welcome. To contribute:

Fork this repository and clone the repository

git clone https://github.com/YOUR-USERNAME/av-guide

Change into the repository's directory

cd av-guide/

Make the changes on your forked version, and then submit a pull request which will be reviewed shortly after.

Contribution Guidelines

View the complete contribution guidelines, here.

Getting Started w/ the Guide :tada:

Introduction

You can use this guide however you like. If you prefer to follow along material by material, go ahead. If you want to skip ahead, come back, skip ahead again, do as you wish. To get started with basic navigation and resources, briefly skim through the below tips.

Navigating

Navigating this project should be fairly straightforward and simple. But, to help you along any struggles along the way, here is how to navigate to some specific things:

README Document

This README document is the central documenation for this guide. At the top, is the Table of Contents which you can use as a main directory.

Contributing Document

The contributing document provides details regarding contributing guidelines, issues, feature requests, and the repository vision. Before submitting an issue or pull request, please briefly review it.

Sections

The table of contents of this guide is here in this README document. Feel free to jump around from section to section, or skim through a section's resource. These sections will be continuously updating and improving with new material and resources.

Community :speech_balloon:

As this community grows, @tabaddor will look into creating a community Slack and/or Discord. For the time being, all communication will be exclusively on GitHub.

Pre-Requisites (and resources)

Some basic knowledge may be assumed, but below are important topics to review.

• Linear Algebra by MIT OpenCourseware
• C++ and/or Python programming is expected
• Data Structures and Algorithms (Cracking the Coding Interview (book) and/or Stanford's Algorithm Course)
• Mathematics for Machine Learning
• Basic Git & Linux Command Knowledge (Optional)
• Basic Software Development Knowledge (Optional)
• Familiarize yourself with Embedded Development and Safety Standards (Optional)

Safety is Critical :vertical_traffic_light:

Self-driving cars are safety critical. A safety critical system can be defined as something that:

comprises everything (hardware, software, and human aspects) needed to perform one or more safety functions, in which failure would cause a significant increase in the safety risk for the people or environment involved.

It is easy to acknowledge that autonomous vehicles need to be safe, but as a developer or someone working on self-driving systems, this is incredibly important to keep in mind. Working with safety critical systems differs greatly from working on a majority of the other systems.

IOS 26262 Safety Standard

To mitigate risk for developers, teams and companies developing safety critical systems typically have to follow safety standards, such as ISO 26262. ISO 26262 is one of the more well-known safety standards that you should be aware of regarding autonomous vehicles.

I recommend briefly skimming through ISO 26262 to get an idea of the standard, but to summarize some key points:

• The goal is to ensure safety throughout the lifecycle of automotive system development
• Code you write as a developer must be in line with the safety standard (companies and teams specify this for you)
• There are specific steps of ISO 26262, and other standards, that are required in each phase of the software development lifecycle

Standards aid in making sure the code you write as a developer is carefully evaluated, inspected by others, and tested during the development process to mitigate risk. This is an essential component of software engineering for any safety critical system, as a lot of emphasis is placed on "careful coding."

Example of a Safety Critical System for Self-Driving Cars

• Advanced Driver Assistance Systems (ADAS)

Examples of ADAS include adaptive cruise control and pedestrian crash prevention.

The gist? Always remember that safety is critical. If you are interested in learning more about software engineering processes for safety critical systems, I highly recommend you take a look at Phil Koopman's page (linked in the Resources section).

High Impact Work :arrow_up_small:

Software engineers working on self driving cars will:

• Tackle the hardest challenge of our generation
• Work with cutting edge technology that will impact the lives and safety of people everywhere
• Collaborate with the generation's brightest minds across disciplines

1. Introduction

This introductory section will provide high level overviews about the history of autonomous vehicles, basic autonomous vehicle taxonomy/terminology, and the current state of self-driving cars. This section is not overly technical, but interesting and important nonetheless.

Below is a typical software stack for a self-driving car. It should be pretty self-explanatory, however, don't get bogged down in the details, it will all fit together at the end (taken from Hyundai-Aptiv Webinar).

History of Autonomous Vehicles :scroll:

• A Brief History of Autonomous Vehicle Technology
• The Story of Autonomous Vehicles

Taxonomy

• How do self-driving cars work?
• Taxonomy of driving
• Driving Decisions and Actions

Levels of Autonomy :oncoming_automobile:

Summary:

• Self-Driving Car
• Public Health, Ethics, and Autonomous Vehicles
• The Future of Autonomous Vehicles
• State of the Art: Self-Driving Cars w/ MIT's Lex Fridman

Section Terminoloy:

• Sensors: A device or machine whose purpose is to detect events or changes in its environment and record or respond to the data.
• Computer Vision (Perception): A field of computer science that trains computers to interpret and understand the visual world.
• Controls: Manipulation of forces by way of steering, braking, etc. Needed to execute guidance commands to maintain vehicle stability.
• Planning: Planning a path, mission, or behavior of the vehicle and predictions of its surrounding.
• Mobility as a Service (MaaS): A shift from personally-owned modes of transportation and towards mobility provided as a servie.
• OTA Software Update: Distributing new software over an encrypted network. Tesla does this with their vehicle fleets.
• ADAS: Advanced Driver Assistance Systems. Electronic systems that help the vehicle driver while driving or during parking.
• Connected Vehicle: A car that can communicate with other systems outside of the car. Most cars are connected these days in some shape or form.
• Datasets: Collection of data to use in the development and testing of autonomous sytems (example: traffic data, weather data, pedestrian data, etc) A complete gloassary of Automated Vehicle Terms

2. Software and Hardware

As a software engineer working on self driving car technology, it is important you have an understanding of not only the software, but the hardware involved. You don't need to be an expert in computer architecture or circuits, but a general understanding and appreciation of hardware is important. With that said, let's dive into the software and hardware architecture of self-driving cars.

Sensors and Hardware Architecture :battery:

• Sensors and Computing Hardware
• Three Sensor Types of Autonomous Vehicles

Hardware in the Loop (HiL)

Hardware in the Loop is a technique that is used in the development and testing of real-time embedded systems. Working as a software or computer engineer at a self-driving car compnay, you may work on the Hardware in the loop team.

• Hardware in the Loop
• Hardware in the Loop simulation for autonomous vehicles
• HIL Simulator for Developing Automated Driving Algorithms

Below is an image of the various sensors in Google's self driving car. Sensors are devices which mesaure some sort of physical property and records or responds to the data.

Sensors work together to provide the car with visuals of its surroundings to ultimately help detect pedestrians, other cars, etc and feeds the data into learning algorithms that aid with motion planning and path prediction. You'll work with the actual computer vision algorithms that deal with the sensor data and images in the Perception section, but for now, just understand how this piece of hardware ties into the self-driving vehicle as a whole. Read HERE for more information about how a self-driving car uses sensors to see.

Software Architecture

The high level idea behind self-driving car software architecture and stacks is separating the main vehicle functionalities into modules (perception, planning, and controls), and allowing them to work together/communciate.

• Software Architecture for Self-Driving Cars
• IEEE Accelerating Autonomous Vehicles
• Stanford Autonomouos Vehicle Technical Stack Video (great summary)

Section Tutorial:

• ROS (Robot Operating System) Tutorials

Section Terminology:

• LiDar vs. Radar: LiDar is a method for measuring distances by illuminating the target with laser light and measure the reflection with a sensor. Radar uses radio waves to achieve the same task. LiDar vs Radar for Applied Autonomy
• Sensors: A device or machine whose purpose is to detect events or changes in its environment and record or respond to the data.
• Hardware in the Loop: A simulation technique used in the development and testing of real-time embedded systems.
• Software Stack: A set of software subsystems and components needded to develop and complete a platform. Most self-driving car software stacks follow the same idea.
• Computing Hardware:
• CPU/GPU: A central processing unit and a graphical processing unit that work together, sitting atop the compute hardware to carry out instructions and concurrent data.
• Sensor Fusion: The ability to bring data from multiple inputs (cameras, LiDar, Radar, etc) to form a single model of the environment around the vehicle.
• Mapping: Using sensor data to form "maps" of the environment around the vehicle.

3. Developing for Safety :traffic_light:

Like stressed earlier in the motivation section, SAFETY IS CRITICAL. As a software engineer or developer working on autonomous, safety critical systems, it's especially important your code applies to safety standards, is tested rigorously (manually and automatically), and more which will be covered in this section.

Safety Assurance for Self-Driving Cars

• Safety Assurance for Autonomous Vehicles
• Safety Frameworks for Driving
• How Many Miles Would it Take to Demonstrate AV Reliability?

Unfortunately, sometimes things go wrong. That is the challenege.

• Uber Crash Preliminary Report

Software Engineering Activites:computer:

As a software engineer, only around 10% to 30% of your time will be spent actually writing new code. The remaining time will be spent reading, understanding, debugging, testing, and communicating code. That is why, especially for a safety critical system like a self-driving car, you have a solid understanding of software engineering activities. We will focus on the big three that are: Testing, Code Quality and Metrics, and Productivity.

Testing

Testing is expensive. But it is also crucial for self-driving cars. Testing the vehicle modules as a whole may take place in simulations, but you will still need the basics of software development testing down.

• Software Engineering at Goolge
• Code Coverage
• Software Development Lifecyle in 9 minites
• Autonomous Vehicle Virtual Testing and Validation

You'll hear the terms validation and verification, and be aware that though the sound similar, they are widely different terms. Validation involves making sure your system meets requirements. Verification invovles making sure your software system is correct. Validation and verification can go hand in hand.

Static Analysis- Debugging that is done by examining the code without executing the program

• Static Program Analysis
• A Decade of Software Model Checking with SLAM

Dynamic Analysis- Testing and evaluation by executing the data in real-time

• Dynamic Program Analysis

Code Quality/Metrics

What is defined as quality code and how quality code is measured will differ from organization to organization, and maybe even from team to team. Is your goal bug-free code? Is your goal minimal debugging time? Is your goal less lines of code? These are all things managers and teams consider, and important for you as a developer to keep in mind when working on safety critical systems like self-driving cars.

• Quality Assurance
• State of Mutation Testing at Google
• 145 Questions for Data Scientist in Software Engineering

Productivity/Teamwork

For the most part, you won't be working on a system alone. You'll be on a team, and a lot of times, coding with another developer in what is called pair programming. Pair programming involves two developers working on a feature, issue, etc, however, they work on their own parts, and then convene later on to discuss and communicate.

• Cost and Benefits of Pair Programming

Dr. Phil Koopman- a gold mine of autonomous vehicle safety research and information:

His YouTube channel can be found in the Resources section. Here are some noteworthy vehicles I recommend:

• Software Maintenance Best Practices
• 10 Risks of Poor Embedded Software Quality
• Software Testing Best Practices
• Best Practices for Unit Testing
• Embedded Software Security, Safety, and Quality

Section Terminology:

• Safety: Condition of being protected, unlikely to cause or be at risk of danger, injury, or death.
• Testing: Activity involved placing your code under rigurous edge cases that allows you to assess the code quality.
• Metrics: Measurement that allows you to place objective judgement on pieces of code.
• Code Maintainability: Ensuring that code is readable by other developers, low in defects, and low coupling (how related code is).
• Code Readability: How easy code is to follow logically by developers other than the original author.
• Code Reliability: Reliable code is secure and handles exceptions for a segment of code accordingly.
• Test Suite/Cases: A test suite is a collection of test cases intended to test a segment or feature of code. A test case is comprised specific units of execution to test the correctness of a piece of code.
• Software Quality: How well the code conforms to a design and the requirements/specifications. What is defined as "quality code" tends to differ from company to company as most use different metrics to measure code quality.
• Unit Testing: Testing individual units of the source code at a time.
• Integration Testing: Testing where individual software modules are combined for testing together.
• Mutation Testing: Involves modifying a test case and seeing if that "mutant" passes or fails when run against the source code.

4. Deep Learning for Self-Driving Cars 🧠

The high level idea behind Deep Learning is building machine learning algorithms and functions that mimic the human brain in processing data for tasks including object detection, speech recognition, and decision making. If you haven't already, I highly recommend you check out Stanford's Machine Learning Course for free.

ML, DL, RL Review

• [] Mathematics for Machine Learning
• [] Deep Learning
• [] Reinforcement Learning

Machine Learning

• What is Machine Learning?
• Supervised Learning
• Unsupervised Learning
• Stanford CS229 Machine Learning
• University of Michigan EECS 445 Machine Learning
• University of Michigan EECS 445 Machine Learning Resources

A specifc class of Neural Networks you should familiarize yourself with are Convolutional Neural Networks, which are involved in tasks including object detection, image processing, classification, and image segmentation.

Convolutional Neural Networks

• An Intuitive Explanation of Convolutional Neural Networks
• One Layer of a Convolutional Neural Network
• Why Convolutions?
• Convolutional Neural Networks for Visual Recognition
• Deep Learning for Computer Vision

Section Terminology

• Machine Learning: A subset of artificial intelligence that provides a computer, or system, the ability to improve its performance on a task by learning from experience.
• Supervised Learning: A type of machine learning task that is provided labeled training data (both x and y are known).
• Unsupervised Learning: A type of machine learning task that is provided unlabeled training data (only x is known).
• Classification Problems: A machine learning task that involves discrete valued output ("boy" or "girl"; "dog" or "cat").
• Regression Problems: A machine learning task that involves continuous/numerical valued output ("find the expected housing price given this data")
• Gradient Descent: An optimization algorithm for finding the local minimum of a differentiable function. When we say machine learning algorithms involve slow altering of a function (hypothesis) to converge to the optimal output, we are referring to this idea of gradient descent.
• Deep Learning: A subset of machine learning that bases its methods off of artificial neural networks, loosely inspired by the human brain.
• Neural Network: Non-linear algorithm based off the biological neural networks, designed to mimic the way the human brain operates. Neural networks are meant to adapt to input and generate the best possible result without needing to redesign the machine learning model/algorithm.
• Convolutional Neural Network: A class of neural networks applied to visual imagery such as object detection that utilizes the convolution operation, pooling, and other tunable layers to respect the spatial structure of image data.

5. State Estimation and Localization

State estimation is the problem of determining the current state of a system, such as a vehicle. Localization is the implementation of algorithms for state estimation with an error of less than a small, non-zero value.

Least Squares

State Estimation and Kalman Filters

Pose Estimation

Section Terminology

• State Estimation and Localization: State estimation involves determining the state of an object, vehicle, or self. Localization is essentially how a vehicle figures out its location, or where it is, whch involves using its on-board cameras and sensors to estimate positions.
• Least Squares: Minimize the squared difference between an observed and expected values.
• Pose Estimation: Is a subset of computer vision that involves locating and tracking the position of an object or specific parts of an object.
• Kalman Filter: A KF provides estimates of some unknown signal overtime given previously determined measurements for that signal, or correlated signals.
• LiDar Sensing: Illuminate a target object with a laser light, and then measure the reflection.
• Sensor Fusion: Ability to bring together multiple inputs from multiple different sensors to construct one world model and understanding of an environment.
• Point Clouds: A set of data points in space that represent a 3D shape or object.

6. Perception 👀

For the purposes of this guide, perception and computer vision are synonymous. Perception deals with training our vehicle to perceive its environment in order to act on what it perceives.

Computer Vision

• Computer Vision for Autonomous Vehicles

Today, there are two main ways of performing vision: Classical vision and machine learning. This guide will provide resources for understanding and working with both methods.

Classical Vision

• OpenCV Tutorials for Classical Vision

Vision with Deep Learning

• Deep Learning for Computer Vision: University of Michigan All Lectures

Object Detection

• Object Detection Guide
• Gentle Introduction to Object Detection with Machine Learning
• YOLO Original Paper

Semantic Segmentation

Section Terminology

• Computer Vision: How can computers understand a situation based on images? CV is all about extracting and making use of interesting and important features of an image, which is really just made up on pixels which can be represented as vectors/matrices, to ultimately interpret an environment.
• Sensors: Devices that perceive the environment and provide information to actuators or algorithms which make sense of the data received via the sensors. For computer vision, sensors include cameras, radars, and LiDar.
• Object Detection: Related to computer vision and image processing that deals with detecting instances of objects in a given frame or image.
• Image Segmentation: Identify every pixel that belongs to a specific object in a frame or image.
• Classical Vision: Classical techniques for computer vision involve leveraging and extracting features of an image or frame without the use of ML or DL.
• Convolutional Neural Networks: A type of Deep Learning algorithm suited for images and computer vision tasks. More info on CNN's here.

7. Motion Planning

The Problem

Motion planning, also known as path planning, is the the problem of finding a sequence of valid motions, configurations, steps, etc that allow an agent to go from its source location to goal destination. Motional planning has several key applications in robotics and autonomous vehicles, rightfully so. In this section, we will break down some key aspects of this function.

Planning

• Robotics: Computational Motion Planning
• EECS 367 Autonomous Robotics Course at the University of Michigan Homepage
• Path Planning Algorithms in C++

Mapping

• Simultaneous Localization and Mapping (SLAM) Paper: Essential Algorithms
• Visual SLAM Resources

Dynamic Objects

Section Tutorials/Projects

Section Terminology

8. Testing

Importance of Testing

Unit Testing

Integration Testing

Static and Dynamic Analysis

Simulation

Resources

• Carla- Open Source Simulator for AVs
• ROS- Robot Operating System
• Embedded Software Development for Safety Critical Systems by Chris Hobbs. (this is not an affiliate link)
• Phil Koopman
• Deep Learning with Lex Fridman
• openpilot: Open Source Driver Assistance Software

AV Companies :briefcase:

A list of companies working on autonomous vehicle technologies. The purpose of this section is to include any relevant information about the companies and any open internship and full-time positions. Don't see your company, position, or a link is broken? Submit an issue or pull request!

Waymo

Waymo is an autonomous driving technology company that is a subsiduary of Alphabet, Inc.

Open Roles

Cruise

Cruise is a self-driving car company owned by General Motors (GM).

Open Roles

Ford

Ford Motor Company is an American automaker now building hybrid and electric vehicles, AVs, SUVs, and Crossovers.

Open Roles

Zoox

Zoox is an American Autonomous Vehicle company now a subsiduary of Amazon.

Open Roles

Tesla

Tesla is an electric car and solar/clean energy company.

Open Roles

Aptiv

Aptiv is an autoparts company shaping the future of mobility through green, connected ADAS and AV technology.

Open Roles

Rivian

Rivian is an electric vehicle manufacturer also working on AV technology.

Open Roles

HARMAN International

HARMAN International is a Samsung subsiduary focusing on car lifesyle, audio, and connected products, including ADAS.

Open Roles

FAQ :speech_balloon:

Q: Who can use this guide?

A: This guide is mainly geared towards computer science students and software engineers looking to learn more about the self-driving car industry. With that said, anyone is welcome to use on the guide, just make sure you have an understanding of the pre-requisites.

Q: How do I get started with this guide and roadmap?

A: Get started here! Official guide content starts here.

Q: Can I download this guide?

A: You may clone and work with this project on your local machine.

Q: I have a suggestion, how can I contribute to this guide?

A: Contribitions and suggestions are always welcome. First, read through this contributing document, then submit an issue and/or pull request.

Q: I don't know much about computer science/software, how can I still use this?

A: Some resources in this guide assumes a certain level of knwoledge of computer science and software engineering concepts.