"Data Science for interview" Series [Blog#7: Deep Learning]

< Previous Episode: Blog#6: Clustering

Goal of this Blog Episode:

To touch the basic of Deep Learning from an interview perspective.

Deep Learning

Definition: Deep Learning, being a sub-field of Machine Learning, is concerned with algorithms called Artificial Neural Networks (ANN) that are inspired by the structure and function of the brain.

Intuition: The term “deep” usually refers to the number of hidden layers in ANN. Traditional neural networks only contain 2-3 hidden layers, while deep neural networks can go up to as many as 150. Deep learning models are usually trained by using large sets of labeled data and neural network architectures that learn features directly from the data without the need for manual feature extraction.

Types of Deep Learning:

• Supervised: The DL model is trained with pre-labeled data which helps the model classifying new inputs. Examples include: Artificial Neural Networks (ANN), Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN).
• Unsupervised: The DL model is trained with un-labeled data and it is expected from the model to label it itself by finding common patterns. Examples include: Self-Organizing Maps (SOM) and Boltzmann Machines.
• Semi-Supervised: There are some DL models which make use of a blend of both supervised and unsupervised approaches to develop a semi-supervised model.

How ‘Deep’ to be called ‘Deep’ Learning?

If the model uses Hierarchical Feature Learning, i.e. identifying lower level features first and then building upon them to identify higher level features (e.g. by using convolution filters), then it is a ‘Deep’ Learning model. If not, then no matter how many layers your model has then it’s not considered a Deep Learning model.

Artificial Neural Networks

Definition: Artificial Neuron Networks (ANN) are computational models inspired by the structure and functions of biological Neural Networks. ANNs attempt to simulate a brain’s network of neurons so a computer may be able to take decisions in a human-like manner.

Intuition: Artificial Neural Networks (ANNs) make use of different mathematical layers of processing to make sense of the information it is fed. Typically, ANN has anywhere from dozens to millions of artificial neurons arranged in a series of hierarchal layers.

Types of Layers in an ANN:

• Input Layer: An input layer is the one where the raw input data is fed which is expected to help in generating useful results. It has several nodes which represent the features involved in the model.
• Hidden Layer: A hidden layer transforms the data received by the preceding input layer after carrying out some mathematical processing. It then feeds the results to the output layer.
• Output Layer: An output layer, usually acting as a classifier, has one or more nodes which render a constructive output after being fed a processed result from the hidden layers.

Terms you should know:

• Perceptron: A Perceptron is a basic single layer neural network. It consists of 4 parts: input layer, weights/bias, net sum and an Activation Function.
• Backpropagation: Backpropagation is an error adjustment technique in neural networks that computes the error at output layer and distributes backwards throughout the network to adjust the error.
• Activation Function: An Activation function serves as a threshold, alternatively called classification. Their role is to make neural networks non-linear. The purpose of an activation function in a Deep Learning context is to ensure that the representation in the input space is mapped to a different space in the output.
• Gradient Descent: Gradient Descent is an optimization algorithm based on a convex function that tweaks its parameters iteratively to minimize a given function to its local minimum. It is simple used to find the values of a function parameters that minimize its cost as much as possible.

Convolutional Neural Networks

Definition: A Convolutional Neural Network (CNN) is a class of Artificial Neural Network primarily used in image recognition and processing that is specifically designed to process pixel data.

Intuition: CNN have their neurons (nodes) arranged more like those of the frontal lobe, the area responsible for processing visual stimuli in humans and other animals. The architecture of a CNN was inspired by the organization of the Visual Cortex.

Why CNN over Feed-Forward Neural Networks: A CNN has the ability to capture the ‘spatial’ and ‘temporal’ dependencies in a pixel data (image) through the use of relevant filters. CNN’s architecture performs a treatment of image dataset due to the reduction in the number of parameters involved and reusability of weights. Shortly, a CNN can be trained to understand the sophistication of an image better.

Types of Layers in a CNN:

• Input Layer: Holds the raw pixelated values of an image.
• Convolutional Layer: Holds a feature map (which has the highlighted aspect of an image) each of which has stored a unique set of features from all the possible permutations.
• RELU Layer: Applies an element-wise activation function which may be used to extract/remove parts of image which may be causing linearity.
• Pooling Layer: It involved extracting and compressing the convolved layers even further to keep different variations of the features.
• Flattening Layer: It involves converting the pooled layers to into 2D columns that would be easier to process by the next fully-connected layers.
• Fully-Connected Layer: A Fully-Connected layer is, usually, an economical way of learning non-linear combinations of the high-level features as represented by the output of the convolutional layer. The Fully-Connected layer is learning a possibly non-linear function in that space.

Recurrent Neural Networks

Definition: A Recurrent Neural Network is a class of Artificial Neural network where connections between nodes form a directed graph along a sequence. This allows RNN to enact a temporal dynamic behavior for a time sequence. RNNs can use their internal state to process sequences of inputs by remembering previous states.

Intuition: We don’t start thinking anything from scratch. Even while reading this sentence your ‘persistent’ thoughts are helping you to comprehend the meaning. Traditional NNs can’t keep track of previous events, so RNNs come in action to address this issue by making use of their looped architecture to remember events. A recurrent neural network can be thought of as multiple copies of the same network, each passing a message to a successor.

Terms you should know:

• Backpropagation through Time: Backpropagation through Time is simply an extension of a regular backpropagation with an addition of being conducted on an unrolled RNN.
• Long-Short Term Memory (LSTM): LSTMs are an extension of RNNs which basically extend their memory with the help of gated cells (input, forget and output gate) which decide whether or not to store/delete information.

Two common issues of RNN:

• Exploding Gradient: Exploding Gradients is a problem when large error gradients accumulate and result in making large gradient updates throughout the network.
• Vanishing Gradient: Vanishing Gradient is a problem when, while adjusting gradient error during backpropagation, the gradient starts to become smaller and smaller as we keep on going backward.

Examples (that you can quote in interviews)

1. Chatbots: Chatbots can be trained on huge text corpuses using RNNs which can then assist humans in customer supports.
2. Colorization of Black & White Images: With enough training, a Deep Learning model can color black and white images.
3. Automatic Game Playing: ANNs can be trained to develop systems that can flawlessly play games.
4. Advertising: Data-driven predictive advertising can be achieved with Deep Learning.

References:

MachineLearningMastery, TowardsDataScience, SearchEnterpriseAI, SuperDataScience, Wikipedia

Next Episode: Blog#8: Python Refresher >