Simple linear regression is one of the first topics covered in any basic statistics course, whether at university or elsewhere. Generally speaking, linear regression is a fundamental statistical tool because it allows us to study the (linear) relationship , the mathematical link, between two or more variables. In this short article, I will attempt to explain the functioning and interpretation of a simple linear regression model, both on a theoretical and conceptual level, using graphical examples from STATA , a statistical software program.

A short description

In that rigorous statistical world populated by decimal points, linear regression refers to the specific methodology that scientists use to study the linear relationship between two or more variables. In jargon, Y is often used to indicate the dependent variable , that is, the variable that "moves" as a function of another variable, called the independent variable , indicated by X. According to the mathematical definition, in fact, a function is a relationship that exists between two sets, called the domain and the codomain, which associates one and only one element of the codomain to each element of the domain.

Linear regression uses this simple, so to speak, mathematical logic. To be clear, there are roughly two linear regression models: the simple linear regression model and the multiple linear regression model . Now, before describing the difference between the two, let's say that linear regression is composed of a dependent variable ; one or more regressors , or independent variables; a regression coefficient ; and an error term called the residual.

We will explain all these concepts as we progress. Returning to the difference between simple linear regression and multiple linear regression , it will suffice to say that they are distinguished by the number of regressors included in the model. Simple linear regression works with a single regressor , a single independent variable . Multiple linear regression, on the other hand, works with two or more regressors . In this article, we will focus exclusively on the simple linear regression model.

What is the basic idea of ​​simple linear regression?

Simple linear regression is explained by the equation y=β0+β1x, where β0 is the intercept , β1 is the slope of the line , and x is the independent variable . In the figure below, we can observe a first theoretical representation of the regression line.

Figure 1 – Theoretical example of linear regression

One of the main problems we face in building a model of this type is finding the line that best fits the observed points. Let's look at a simple and practical example to better understand this point.

Fig. 2. – Scatter plot Fig. 3 – Scatter plot with linear regression line


In the graph in figure 2 we find on the y-axis our dependent variable, which in this case is the monthly salary. On the x-axis lies the independent variable, or the years of education completed. We want to understand whether monthly salary increases or decreases (and by how much) as years of education increase. When I mentioned the problem of constructing the line that best fits the points, I was referring to the one shown in Figure 3.

Obviously, it's not always possible to build a linear regression model. The first thing to do is to use the scatterplot (Figure 2) to ensure that the points are arranged in a straight line, as in our case. Otherwise, linear regression won't be a good model for our study, but a nonlinear regression, for example, will be . Returning to our topic, the straight line allows us to create the mathematical model on which simple linear regression is based. To do this, it's important to understand how it's constructed and how the parameters we need are calculated.

How do you calculate simple linear regression?

Before I begin, I want to clarify that I won't explain all the formulas in detail, as the goal of this article is to demonstrate the concept and interpretation of the simple linear regression model. For those who would like to delve deeper into this topic, I'll provide some articles to consult. For now, I'll simply point out that to calculate the linear regression line, a mathematical tool called the ordinary least squares method , or OLS , is used.

Using the least squares method, we assign the parameters β0 and β1, the intercept and slope of the line, respectively, the values ​​(b0 and b1) for which the sum of the residual squares is minimal. Conceptually, we need to minimize the sum of the squared distances between the observed values ​​and the line. We need squaring because we don't need negative deviations; the distance must be calculated as an absolute value. In summary, as I propose in this example , which I recommend consulting, we need to find the line that minimizes the sum of the squared deviations.

Figure 4. – Graphical display of residues

In practice, how do we find these values? Fortunately, there are those who have already thought about this long before us and found the answer to the question. These are the formulas for the two parameters needed to calculate the simple linear regression line, where u represents the mean.

Where the numerator is the covariance of x and y, and the denominator is the variance of x. With these two parameters (plus the residuals we'll discuss later), we construct the straight line of the simple linear regression model. In fact, the equation of the regression line is as follows: 

As is evident, to calculate linear regression, it is necessary to calculate the mean. If it is not possible to calculate the mean of the "variables" or data in the distribution, then the regression will not be a good statistical analysis model. To conclude and be more precise, even if we will not discuss the so-called residual analysis in depth, let's say that within the model equation y=b0+b1x we must insert a parameter called the residual . The residuals represent the part of the regression model's prediction error . They are the difference between the theoretical values ​​of the model and the real, observed values. To complete the equation of the straight line, we will then write:

where

How is it interpreted?

Regarding the interpretation of the regression line, we must introduce an index—it is not the only one, but it is one of the most widely used—called the coefficient of determination , denoted by r2. This tool's function is to describe the line's goodness of fit to the observed values. The r2 index has values ​​between 0 and 1, to facilitate its reading and interpretation. It represents the proportion of the dependent variable's variability explained by the regression line. For example, if r2 = 0.5, it means that the regression line explains 50% of the dependent variable's variability, Y.

Fig. 5 – High r-squared value Fig 6. – Low r-squared value

In the practical example proposed above, we observe that r2 = 0.6761, which means that approximately 67% of the income variability is explained by the number of years of education. Now, if, for example, we wanted to predict the value of y given an arbitrary x, having built the linear regression model, we would simply substitute the desired value for x in the equation. 

Fig. 7 – Values ​​taken from our example


If you remember, we mentioned that the regression model, in addition to measuring the dependence of two variables, allows us, through the construction of the model, to predict the values ​​of the independent variable Y, given the values ​​of x. This technique can be summarized as: interpolation ed extrapolation. Interpolation it occurs when we want to predict the value of Y given an x ​​belonging to the range of observed values, in which our relationship is linear. Extrapolation instead, it occurs when x does not belong to that reference interval, it can be larger or smaller. The difference lies in the fact that, by doing extrapolation, the further you move away from the upper/lower end of the range, the more absurd the results risk becoming.

This is because we don't know whether the linear relationship we studied holds for those x values ​​outside the range. Extrapolation should be largely avoided or at most limited to values ​​just outside the reference range. In fact, the regression line was determined using the x values ​​of the relevant range, and there is no way of knowing how the line would change by adding x values ​​outside the relevant range; extrapolations implicitly assume that adding external x values ​​would leave the line essentially unchanged, but this assumption is unverifiable and becomes even less plausible the further the points considered are from the relevant range.

Let's move straight to the practical example. 

Fig 8. – Graphical example of the concept of extrapolation

In the regression model built above, we observe that the linear regression line is represented by the equation: y=320.3408+91.43589x. Now, if we wanted to predict the value of Y (therefore the monthly income) of a person with 16 years of education (variable X), we simply substitute the desired value in place of x, so as to obtain: y=320.3408+(91.43589*16)= 1783.315 . The predicted income for a person with 16 years of education will therefore be €1783.

Conclusion

The concept of linear regression, unlike that of correlation, doesn't simply tell us whether two variables are more or less correlated (positively or negatively). Regression adds a real mathematical model to this information , namely the equation we saw earlier. This is very important because it allows us to produce estimates and predictions that we couldn't make with correlation analysis alone.

Now, it's important to remember that regression and correlation analysis don't necessarily tell us whether variable Y is entirely caused by variable X, since causation (or causality) and correlation are two profoundly different concepts. However, we've already touched on this in this article . Finally, for those who want to delve deeper into the mathematics behind the linear regression model, I highly recommend reading this article .

CORRELATED ARTICLESMORE FROM THE AUTHOR