Introduction
This topic covers simple linear regression, a method that describes how the mean value of one continuous outcome changes with one continuous explanatory variable, and the least squares method, which is the standard way of estimating the line. Students should be able to write the model, explain the meaning of its coefficients, derive the estimates, state the assumptions and interpret the output.
The model
Regression analysis predicts a dependent variable (outcome, Y) from an independent variable (predictor, X). In the simple linear model the relationship is a straight line, and individual observations scatter around it because of random variation. The model is written as:
- a is the intercept: the expected value of Y when X equals 0.
- b is the slope (regression coefficient): the expected change in Y for a one-unit increase in X.
- e is the random error (residual) for each subject, assumed to vary around zero.
The least squares method
The unknown values of a and b are estimated from the sample so that the line lies as close as possible to all points. Closeness is measured by the residual, the difference between the observed value and the value predicted by the line. The least squares method chooses the line that minimises the sum of squared residuals, which avoids positive and negative deviations cancelling out and gives a unique solution:
Setting the partial derivatives of S to zero gives the normal equations, whose solution is:
- x̄ and ȳ are the sample means; s_x and s_y the standard deviations; r the Pearson correlation coefficient.
- The fitted line always passes through the point (x̄, ȳ).
- The sign of b equals the sign of r, so a positive slope means Y tends to rise with X.