Linear Regression From Scratch: The Model and Cost Function

Introducing the Student Marks dataset

Student Marks Dataset: The data consists of two features including the study time and number of courses. In this blog article we will focus on how the students marks change with respect to the study time.

The Student Marks dataset consists of 3 attributes, two features “study time” and “number of courses”, and a target attribute “Marks”.

In this tutorial we will focus on studying how student marks increase, with respect to the number of hours they put into their studies.

The dataset contain $m=100$ training examples:

$$\begin{array}{lcl} \text{Example } x^{i} & \text{x: "time\_study"} & \text{y: "Marks"} \\ 0 & 4.508 & 19.202 \\ 1 & 0.096 & 7.734 \\ 2 & 3.133 & 13.811 \\ 3 & 7.909 & 53.018 \\ 4 & 7.811 & 55.299 \\ ... & ... & ... \\ 99 & ... & ... \\ \end{array}$$

Scatter plot of the 100 training examples: study time on the x-axis, marks on the y-axis.

What is Linear regression

Linear regression means fitting a straight line to your data.

Training example notation

• $(x^{(i)}, y^{(i)})$: A tuple representing the $i^{th}$ training example. Geometrically, it is the plotted point in the coordinate system, where $x^{(i)}$ represents the input feature (“study_time”) and $y^{(i)}$ represents the target output variable.

• $x^{(i)}$: Geometrically, this is the projection of the $i^{th}$ training point onto the X-axis.

• $y^{(i)}$: Geometrically, this is the projection of the $i^{th}$ training point onto the Y-axis.

• $m$: The total number of training examples.

E.g. the $3^{rd}$ training example in the previous table ($i=2$) is represented by the tuple $(x^{(2)}, y^{(2)}) = (3.133, 13.811)$.

The linear regression model

Fitting a straight line to your data corresponds to finding a model:

$\hat{y} = f_{w,b}(x) = wx + b$

Where:

• $x$ and $y$ correspond to any training example tuple $(x,y)$.
• $\hat{y}$ (pronounced “y-hat”) is the prediction of the model. This is also commonly referred to as the hypothesis, sometimes written as $h(x)$ or $h_{w,b}(x)$.
• $w$ and $b$ are learnable parameters that control the slope of the line and its Y-intercept (intersection point with the Y-axis), respectively.

The model $f_{w,b}(x)$ is called the linear regression model for one variable.

# In NumPy:
import numpy as np

def predict(x, w, b):
    return w * x + b

Before training, a rough initial guess $w=5, b=5$ already traces a line through the data, but it doesn’t fit well yet. Training will find the optimal $w^*$ and $b^*$.

Understanding the slope parameter

$w$ controls the line’s slope. As shown in the figure below, the slope is the tangent of the directed angle from the x-axis to the line.

This is calculated for three cases: a high positive slope $w_{1}$, a lower positive slope $w_{2}$, and a negative slope $w_{3}$.

$$\begin{array}{|c|c|c|c|c|} \hline \textbf{Slope } (w) & \textbf{Angle } (\theta) & (\tan\theta) & \left(\frac{\Delta y}{\Delta x}\right) & \textbf{Value} \\ \hline w_1 & \theta_1 \approx 56.3^\circ & \tan\theta_1 & \frac{3 - 0}{2 - 0} & 1.5 \\ \hline w_2 & \theta_2 = 45^\circ & \tan\theta_2 & \frac{2 - 0}{2 - 0} & 1 \\ \hline w_3 & \theta_3 = -45^\circ & \tan\theta_3 & \frac{-3 - 0}{3 - 0} & -1 \\ \hline \end{array}$$ $$\begin{array}{c} -45^\circ < 45^\circ < 56.3^\circ \\ \theta_3 < \theta_2 < \theta_1 \\ \\ w_3 < w_2 < w_1 \\ -1 < 1 < 1.5 \end{array}$$ $$\boxed{ \begin{array}{l} w = \text{slope} = \tan \theta \\ \theta \uparrow \iff w \uparrow \\ \theta \downarrow \iff w \downarrow \end{array} }$$

So in brief, the slope $w$ controls the direction of the line: when it’s positive, the line points towards the top-right.Whereas when it’s negative, it points down towards the right.

$w$ is calculated from two points on the line, $P_{1}$ and $P_{2}$, by taking the ratio of the change $\Delta y = y2 - y1$ on the Y-axis to the change on the X-axis $\Delta x = x2 - x1$.

Learning the model parameters

Linear regression is a model that learns how to fit a line to the data.

The learning itself needs a supervised learning algorithm that takes the training set consisting of $m$ examples $(x,y)$, and learns the mapping from $x$ to $y$. In our case this is a model $f_{w,b}(x)=wx+b$ that geometrically represents a line.

Learning the model $f_{w,b}(x)$ actually corresponds to learning the coefficients or parameters $w$ and $b$.

Evaluating parameters with an objective function

At each iteration the learning algorithm will try to find the optimal parameters $w$ and $b$ for the model.

To evaluate whether the parameters $w$ and $b$ are actually good or not in each learning iteration, we need to define an objective function.

Given the current parameters $w$ and $b$, the cost function defines how well the current model $f_{w,b}$ is performing on the training set.

In math, this is an optimization problem, because we aim to find the optimal parameters $w$ and $b$ that either maximize or minimize the objective function.

When we try to minimize a function, the function is called a cost function. Whereas when we try to maximize it, the function is called: reward, utility, fitness, depending on the context where it’s used.

In linear regression we use a cost function $J(w,b)$ which measures how inaccurate your linear model $f_{w,b}(x)$ is at predicting target values $y$.

Sum of squared errors

It’s known as SSE, or sum of squared errors. It measures how close the model’s prediction $\hat{y}^{(i)}$ is from the ground truth target value $y^{(i)}$ for each training example $i$ in the training set of size $m$.

That difference is measured by subtracting $y^{(i)}$ from $\hat{y}^{(i)}$ and it’s referred to as the residual $R^{(i)}$ or the error.

$J(w,b)$ is calculated by summing the squared residuals $(R^{(i)})^2$ for each training example $(x^{(i)},y^{(i)})$, and then dividing by the size of the training set $m$ multiplied by 2.

$J(w,b) = \frac{1}{2m} \sum_{i=1}^{m} (R^{(i)})^2$

$R^{(i)} = \hat{y}^{(i)} - y^{(i)}$

$\hat{y}^{(i)} = f_{w,b}(x^{(i)}) = wx^{(i)} + b$

Dividing by $2m$ or $m$ makes no big difference for the learning algorithm, but it simplifies the derivative of the cost function, which we will need in the next article of the series.

With $R^{(i)}$ and $\hat{y}^{(i)}$ substituted in, the cost function expression for linear regression becomes:

$J(w,b) = \frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} + b - y^{(i)} )^2$

# In NumPy:
def compute_cost(x, y, w, b):
    m = len(y)
    return (1 / (2 * m)) * np.sum((w * x + b - y) ** 2)

The figure below summarizes all what we talked about so far.

Conclusion

We now have a model $f$ parametrized by the slope $w$ and the Y-intercept $b$ that geometrically fits a line over the dataset and makes predictions about it.

And also a cost function $J$ that measures how wrong those predictions are.

A natural question that might come to your mind is: how do we actually find the values of $w$ and $b$ that minimize the cost function $J$ to its absolute global minimum?

Well, that’s exactly what we are going to learn in the next article, which is about: Read Part 2: Derivatives, Critical Points, and Convexity of the Linear Regression Cost Function