11-485/785 Introduction to Deep Learning

Fall 2018

“Deep Learning” systems, typified by deep neural networks, are increasingly taking over all AI tasks, ranging from language understanding, and speech and image recognition, to machine translation, planning, and even game playing and autonomous driving. As a result, expertise in deep learning is fast changing from an esoteric desirable to a mandatory prerequisite in many advanced academic settings, and a large advantage in the industrial job market.

In this course we will learn about the basics of deep neural networks, and their applications to various AI tasks. By the end of the course, it is expected that students will have significant familiarity with the subject, and be able to apply Deep Learning to a variety of tasks. They will also be positioned to understand much of the current literature on the topic and extend their knowledge through further study.

Course description from student point of view

The course is well rounded in terms of concepts. It helps us understand the fundamentals of Deep Learning. The course starts off gradually with MLPs and it progresses into the more complicated concepts such as attention and sequence-to-sequence models. We get a complete hands on with PyTorch which is very important to implement Deep Learning models. As a student, you will learn the tools required for building Deep Learning models. The homeworks usually have 2 components which is Autolab and Kaggle. The Kaggle components allow us to explore multiple architectures and understand how to fine-tune and continuously improve models. The task for all the homeworks were similar and it was interesting to learn how the same task can be solved using multiple Deep Learning approaches. Overall, at the end of this course you will be confident enough to build and tune Deep Learning models.

What students say about the previous edition of the course

Instructor: Bhiksha Raj

• [email protected]

TAs:

• Ryan Brigden ([email protected])
• Raphael Franck Olivier ([email protected])
• Sai Nihar Tadichetty ([email protected])
• Shubham Tripathi ([email protected])
• Soham Ghosh ([email protected])
• Madhura Das ([email protected])
• Ipsita Prakash ([email protected])
• Dhruv Shah ([email protected])
• Shaden Shaar ([email protected])
• David Zhang ([email protected])
• Anushree Prasanna Kumar ([email protected])
• Ahmed Shah ([email protected])
• Jiawei Yang ([email protected])
• Omar Khattab for Doha ([email protected])
• Nebiyou Yismaw for Kigali ([email protected])

Lecture: Monday and Wednesday, 9.00am-10.20am

Location: Gates-Hillman Complex GHC 4102

Recitation: Friday, 9.00AM-10.20AM

Office hours:

• TBD

Course Work

Grading

Grading will be based on weekly quizzes, homework assignments and a final project.

There will be five assignments in all. They will also be due on the same date.

Books

The course will not follow a specific book, but will draw from a number of sources. We list relevant books at the end of this page. We will also put up links to relevant reading material for each class. Students are expected to familiarize themselves with the material before the class. The readings will sometimes be arcane and difficult to understand; if so, do not worry, we will present simpler explanations in class.

Discussion board: Piazza

We will use Piazza for discussions. Here is the link. Please sign up.

You can also find a nice catalog of models that are current in the literature here. We expect that you will be in a position to interpret, if not fully understand many of the architectures on the wiki and the catalog by the end of the course.

Kaggle

Kaggle is a popular data science platform where visitors compete to produce the best model for learning or analyzing a data set.

For assignments you will be submitting your evaluation results to a Kaggle leaderboard.

Academic Integrity

You are expected to comply with the University Policy on Academic Integrity and Plagiarism.

• You are allowed to talk with / work with other students on homework assignments
• You can share ideas but not code, you should submit your own code Your course instructor reserves the right to determine an appropriate penalty based on the violation of academic dishonesty that occurs. Violations of the university policy can result in severe penalties including failing this course and possible expulsion from Carnegie Mellon University. If you have any questions about this policy and any work you are doing in the course, please feel free to contact your instructor for help.

Some ideas for projects

Piazza TA Schedule

Monday Dhruv, Soham, Shubham, David

Tuesday Nihar, Soham, Ryan, Ipsita

Wednesday Dhruv, Nebiyou, Ahmed, Raphael

Thursday Madhura, Nebiyou, Shaden, Jiwaei, Anushree

Friday Madhura, Omar, Jiwaei, Nihar

Saturday Omar, Ipsita, Shubham, Raphael, Anushree

Sunday Ryan, David, Ahmed, Shaden

Documentation and Tools

Textbooks

• Deep Learning --- By Ian Goodfellow, Yoshua Bengio, Aaron Courville --- Online book, 2017
• Neural Networks and Deep Learning --- By Michael Nielsen --- Online book, 2016
• Deep Learning with Python --- By J. Brownlee
• Deep Learning Step by Step with Python: A Very Gentle Introduction to Deep Neural Networks for Practical Data Science --- By N. D. Lewis
• Parallel Distributed Processing --- By Rumelhart and McClelland --- Out of print, 1986