Essential Differences between Deep Learning and Conventional Neural Network

There has been a resurgence in interest in Neural Network since 2006, because of some breakthroughs known collectively as Deep Learning.

A Deep Learning based solution uses deep neural network with multiple hidden layers. However, the learning techniques are different from conventional neural networks.

Key Differences

Conventional Neural Network

A conventional neural network, uses back propagation of error along with gradient descent to find combination of network weights that minimizes the error. Initial network weights are randomly chosen. The network is trained through an iterative process.

A conventional neural network with multiple hidden layers generally provide poor result i.e large generalization error. the performance gets worse the number of layers go up.

However complex predictions problems require complex function to map input to output. Modeling complex mapping functions require complex network structure i.e. network with multiple hidden layers and many units in each layer.

Deep Learning Network

A network using deep learning (DL) also uses a network with deep architecture. However the similarity with conventional network ends there. A deep learning based solution uses two distinct phases.

In the first phase the network is pre-trained with unsupervised learning algorithms. This step produces a set of network weights. These weights are used as the initial network weights in the second phase, which is supervised learning process just like a conventional neural network.

The two popular techniques for unsupervised pre-training are Restricted Boltzmann Machine and De-noising Auto Encoder. With both techniques, each layer is pre-trained in a greedy layer wise manner.

The pre-training phase can be thought of as a feature reduction technique like Principal Component Analysis. However features extracted have a hierarchy in deep learning. As you go up the layers, you have higher level of abstraction of features. This is akin to neural processing in brain. Our brain learns features at different level of abstractions. Higher level features build upon lower level features.

The supervised learning in the second phase, is very similar to conventional neural network. The key difference is that with DL, the network weights are initialized in the unsupervised pre- training phase as described above. There weights get further modified as we go through the iterations of supervised learning.

In conventional neural network, the initial network weights are random. This difference in approach for initial network weights has a dramatic impact on the performance of DL based neural network. This is the breakthrough that is responsible for the magic in deep learning.

Why Deep Learning Works So Well

It's natural for this question to arise in our mind. Intuitively, it kind of makes sense. In DL we discover hierarchy of features through pre-training. It's believed that our brain also learns hierarchy features with increasing level of abstraction.

However, no rigorous theory exists that can clearly explain why it work so well. It's an interesting and active area of research.There are several explanatory hypotheses. But the jury is still out. I will cover this topic in a separate post.