Linear regression in machine learning

D R . S . S H A J U N N I S H A , M C A . , M . P H I L . , M . T E C H . , M B A . , P H . D . ,
A S S I S T A N T P R O F E S S O R & H E A D ,
P G & R E S E A R C H D E P T . O F C O M P U T E R S C I E N C E
S A D A K A T H U L L A H A P P A C O L L E G E ( A U T O N O M O U S ) ,
T I R U N E L V E L I .
S H A J U N N I S H A _ S @ Y A H O O . C O M
+ 9 1 9 9 4 2 0 9 6 2 2 0
MACHINE LEARNING
LINEAR REGRESSION

Agenda
2
• Single Dimension Linear Regression
• Multi Dimension Linear Regression
• Gradient Descent
• Categorical Inputs

What is Linear Regression?
3
• Learning
• A supervised algorithm that learns from a set of training samples.
• Each training sample has one or more input values and a single output
value.
• The algorithm learns the line, plane or hyper-plane that best fits the
training samples.
• Prediction
• Use the learned line, plane or hyper-plane to predict the output value for
any input sample.

Single Dimension Linear
Regression
4

Single Dimension Linear Regression
5
• Single dimension linear regression
has pairs of x and y values as input
training samples.
• It uses these training sample to derive a
line that predicts values of y.
• The training samples are used to
derive the values of a and b that
minimise the error between actual
and predicated values of y.

6
• We want a line that minimises
the error between the Y values
in training samples and the Y
values that the line passes
through.
• Or put another way, we want
the line that “best fits’ the
training samples.
• So we define the error function
for our algorithm so we can
minimise that error.

7
• To determine the value of a that
minimises the error E, we look
for where the partial diﬀerential
of E with respect to a is zero.

8
• To determine the value of b that
for where the partial diﬀerential
of E with respect to b iszero.

9
• By substituting the final
equations from the previous two
slides we derive equations for a
and b that minimise the error

10
• We also define a function which we
can use to score how well derived
line fits.
• A value of 1 indicates a perfect fit.
• A value of 0 indicates a fit that is no
better than simply predicting the
mean of the input y values.
• A negative value indicates a fit that
is even worse than just predicting
the mean of the input y values.

Multi Dimension
Linear Regression
11

Multi Dimension Linear Regression
12
• Each training sample has an x made
up of multiple input values and a
corresponding y with a single
value.
• The inputs can be represented as
an X matrix in which each row
is sample and each column is a
dimension.
• The outputs can be represented as
y matrix in which each row is a
sample.

13
• Our predicated y values are
calculated by multiple the X
matrix by a matrix of weights, w.
• If there are 2 dimension, then this
equation defines plane. If there
are more dimensions then it
defines a hyper-plane.

14
• We want a plane or hyper-plane
that minimises the error between
the y values in training samples
and the y values that the plane or
hyper-plane passes through.
• Or put another way, we want
the plane/hyper-plane that
“best fits’ the training samples.
• So we define the error function
for our algorithm so we can
minimise that error.

15
• To determine the value of w that
for where the diﬀerential of E
with respect to w is zero.
• We use the Matrix Cookbook
to help with the
diﬀerentiation!

16
• We also define a function which we
can use to score how well derived
line fits.
• A value of 1 indicates a perfect fit.
• A value of 0 indicates a fit that is no
better than simply predicting the
mean of the input y values.
• A negative value indicates a fit that
is even worse than just predicting
the mean of the input y values.

17
• In addition to using the X matrix to represent basic features our training data,
we can can also introduce additional dimensions (i.e. columns in our X
matrix) that are derived from those basic feature values.
• If we introduce derived features whose values are powers of basic
features, our multi-dimensional linear regression can then derive
polynomial curves, planes and hyper-planes.

18
• For example, if we have just one
basic feature in each sample of X,
we can include a range of powers
of that value into our X matrix like
this:
• In non-matrix form our
multi- dimensional linear
equation is:
• Inserting the powers of the basic
feature that we have introduced
this becomes a polynomial.

Gradient Descent
19
• Gradient descent is a technique we can use to find the minimum of
arbitrarily complex error functions.
• In gradient descent we pick a random set of weights for our algorithm and
iteratively adjust those weights in the direction of the gradient of the error
with respect to each weight.
• As we iterate, the gradient approaches zero and we approach the
minimum error.
• In machine learning we often use gradient descent with our error function
to find the weights that give the lowest errors.

Gradient Descent
20
• Here is an example with a very simple
function:
• The gradient of this function is given by:
• We choose an random initial value for x and a
learning rate of
0.1 and then start descent.
• On each iteration our x value is decreasing
and the gradient (2x) is converging towards 0.

Gradient Descent
21
• The learning rate is a what is know as a hyper-parameter.
• If the learning rate is too small then convergence may take a very long
time.
• If the learning rate is too large then convergence may never happen because
our iterations bounce from one side of the minima to the other.
• Choosing a suitable value for hyper-parameters is an art so try diﬀerent
values and plot the results until you find suitable values.

Multi Dimension Linear Regression with
Gradient Descent
22
• For multi dimension linear
regression our error function is:
• Diﬀerentiating this with respect to
the weights vector gives:
• We can iteratively reduce the error by
adjusting the weights in the direction
of these gradients.

One-hot Encoding
24
• When some inputs are categories (e.g. gender) rather than numbers (e.g. age) we
need to represent the category values as numbers so they can be used in our
linear regression equations.
• In one-hot encoding we allocate each category value it's own dimension in the
inputs. So, for example, we allocate X1 to Audi, X2 to BMW & X3 to Mercedes.
• For Audi X = [1,0,0]
• For BMW X = [0,1,0])
• For Mercedes X = [0,0,1]

Linear regression in machine learning

Recommended

More Related Content

What's hot (20)

Similar to Linear regression in machine learning (20)

More from Shajun Nisha (18)

Recently uploaded (20)

Linear regression in machine learning