Learning Deep Learning

Motivation

If you are like me, an engineer who has heard a lot about deep learning and at times even applied it to specific problems, but always had a feeling that your understanding of the subject is somewhat superficial, I hope you find this post interesting.

Since my academic program did not include courses covering the topic, I decided to improve my understanding by taking several online courses. This post aims at comparing between the courses’ teaching styles and content, summarize my personal impressions form the learning experience and hopefully help in deciding in which order should you take the courses if you choose so.

Prior to continuing, I wish to clarify that what follows are my personal thoughts only, and I don’t claim to be an expert neither in deep learning nor in teaching. There are many resources for online learning and the ones mentioned here are my personal choice.

Additionally, the price of the courses is not mentioned intentionally as a parameter due to the fact that it can change with time and depends also on your progress speed.

Courses chosen

After examining many courses, I chose to focus on two groups of courses based on their syllabus:

1. Deep Learning Specialization courses provided by deeplearning.ai. The Specialization is taught by Andrew Ng on Coursera and includes the following courses:

• Neural Networks and Deep Learning
• Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
• Convolutional Neural Networks
• Structuring Machine Learning Projects
• Sequential Models

2. Courses provided by the “Lazy Programmer” on Udemy which include:

• Data Science: Deep Learning in Python
• Modern Deep Learning in Python
• Deep Learning: Convolutional Neural Networks in Python
• Unsupervised Deep Learning in Python
• Deep Learning: GANs and Variational Autoencoders

The comparison is based on the first three courses in each group which mostly cover the same topics. The remaining two courses cover different topics and are not mentioned in a comparable context.

Main comparison points

Teaching style

In the deep learning specialization provided on Coursera, you are taught the theory by professor Andrew Ng, who is the Co-Founder of Coursera and has headed the Google Brain Project and Baidu AI group in the past. Professor Ng teaches in a very relaxed and patient tone and the explanations are clear and well formulated. One of the major upsides I liked is that the notation used is carefully chosen and very clear. Professor Ng makes sure to reference the most important scientific papers that contributed to each idea, which is great if you want to dive a little more into details. To progress in the course, at the end of each major chapter you will have to submit a multiple-choice quiz and one or two programming assignments in python. The programming assignments require you to complete a 3/4 finished code, and the focus is on understanding the concepts rather than actual programming.

In the courses taught by the Lazy Programmer, the material is introduced in the following way: you first learn the theory and then implement it (from scratch) using python. The theory is very well explained, however the notation is sometimes inconsistent (not too much of a trouble, but requires attention). There are no mandatory assignments to submit in order to complete the course. The programming exercises are in a “code along” style with provided references for the source code if needed. The exercise focus both on the theory and also on correct implementation of all the concepts in a python environment.

Depth of explanation

Comparing to the deep learning specialization, the courses taught by Lazy Programmer cover a lot more of the math behind the main concepts. You are taught the “why” and not only the “how”. The material is taught step by step using ideas from calculus and linear algebra supported by real life examples. What I enjoyed most in this style of teaching is that a lot of the ideas were developed by asking questions and discussing the possible answers, which in my opinion, is a more effective way to learn.

In the deep learning specialization, in many cases you are given the “how” without presenting the mathematical basis behind. The courses are given from an implementation-based practical point of view. This is fine if you have sufficient background knowledge or if you just want to quickly implement deep learning solutions, however in my case, I often found myself opening the referenced papers to understand more since the phrase “it turns out to be” is used a little too often during the course.

The only difference to the mentioned above is the Convolutional Neural Networks course, which I think was better explained by Andrew Ng.

Contribution to practical application (can I setup a deep learning system after taking the course?)

Here, I find the courses given by Lazy Programmer to be much more beneficial because you first implement everything from scratch, and only later use advanced libraries such as tensorFlow, Theano and Keras. This way, you get much more tools in your arsenal. In the Coursera specialization, however, a lot of the code is already implemented and your are presented with “magical” functions that just work.

It is also worth mentioning that after completing the deep learning specialization on Coursera, if you cancel your subscription, some of the source code used in the programming exercises is not longer available to you.

Volume of content and pace (how long does it take?)

Here I try to evaluate “how much do you learn?” and “how fast do you learn it?”. As a side note, I don’t think completing a course or even several courses is enough to really learn a topic. Implementation of the learned material in your own project is crucial. Having said that, I feel that the courses offered by Lazy Programmer cover less topics per each course and each topic is covered in a deeper manner. The deep learning specialization courses, however can give a wider grasp of the field in shorter time. Investing ~ 6 hours a day, it took me about a week to complete each Lazy Programmer course, and about a week and a half to complete all 5 deep learning specialization courses.

To summarize, I would recommend doing both if you have the time.

Special mention: Structuring Machine Learning Projects by deeplearning.ai

A special mention is needed for the “Structuring Machine Learning Projects” course by deeplearning.ai. This, in my opinion, is the most important course in the specialization! It teaches you how to plan your machine learning project, which errors and challenges can rise during implementation and how can you deal with them. Personally, I feel it helped me a lot as I currently try to plan my machine learning project as part of my thesis.

Suggested order of taking courses

First, I highly recommend to make sure you feel comfortable with the concepts of Logistic Regression, since this is the base for all that follows. I would even suggest taking a course specifically focusing on the subject if you have the time. Assuming sufficient understanding of logistic regression, I recommend the following order of taking the courses:

1. Data Science: Deep Learning in Python (Lazy Programmer)
2. (Optional) Neural Networks and Deep Learning (deeplearning.ai)
3. Modern Deep Learning in Python (Lazy Programmer)
4. (Optional) Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization (deeplearning.ai)
5. Convolutional Neural Networks in Python(deeplearning.ai)
6. (Optional) Deep Learning: Convolutional Neural Networks in Python (Lazy Programmer)
7. Structuring Machine Learning Projects (deeplearning.ai)
8. Sequential Models (deeplearning.ai)
9. (Optional) Additional courses depending your personal interest

Final Note

In these strange COVID-19 times, online learning is a great resource for education. Deep Learning is complex and it takes more than completing a couple of courses to truly understand, but both deeplearning.ai and Lazy Programmer provide a great opportunity to start.

“The hardest part of any important task is getting started on it in the first place. Once you actually begin work on a valuable task, you seem to be naturally motivated to continue.” (Brian Tracy)