You measured two things on each of \(n\) units — an input \(x\) and an output \(y\) — and the points form a cloud. A line is the simplest story you can tell about how \(y\) moves with \(x\). This blueprint is about what that line is really doing, what it promises, and where its promises stop.
Suppose you want to summarize how \(y\) tends to move with \(x\), or to predict \(y\) for a new unit whose \(x\) you'll measure. You could report nothing, or you could report the mean of \(y\) and ignore \(x\) entirely — but that throws away everything you measured about \(x\). The next step up is to assume \(y\) changes at a constant rate as \(x\) moves. That assumption is a straight line. It needs only two numbers: a baseline (where the story starts) and a rate (how much \(y\) shifts per unit of \(x\)). Every more elaborate shape — curves, thresholds, interactions — is a story you have to earn with evidence. The line is the default, the first-order approximation, the thing you fit before you have a reason to fit anything else.
A straight line is the default, not the truth. It's the simplest story — baseline plus constant rate — that you fit before any evidence justifies a more complex shape.
Many lines could be drawn through the cloud. To pick one we need a definition of "error." For each observed point \((x_i, y_i)\), the line's prediction \(\hat{y}_i = b_0 + b_1 x_i\) misses the actual \(y_i\) by a vertical gap called the residual, \(e_i = y_i - \hat{y}_i\). A good line makes the collection of residuals small. We can't just sum them — positives and negatives cancel, so a terrible line through the middle of a vertical scatter could sum to zero. Three properties make squares the conventional choice: squares are always positive (no cancellation), they penalize large misses disproportionately (a miss of 2 costs 4, a miss of 4 costs 16), and they are differentiable — which is what lets calculus produce the closed-form recipes in §03. Minimizing the sum of squared residuals is ordinary least squares (OLS).
Residuals are vertical misses, and OLS squares them so misses can't cancel, big ones hurt more, and the math gives a clean closed form. "Best fit" is always relative to a loss function — squares are a choice, not a law of nature.
Setting the derivative of the squared-error objective to zero yields two equations whose solution is a pair of formulas. To read the slope formula we need two pieces of vocabulary you already feel but haven't pinned down: variance and covariance. Variance is spread made precise — the average squared distance of each \(x\) from its own mean. Covariance is its two-variable cousin: the average product of "how far \(x\) is from its mean" times "how far \(y\) is from its mean." It is positive when \(x\) above its mean tends to travel with \(y\) above its mean, and negative when they move oppositely. The slope is literally their ratio: how much \(y\) and \(x\) move together divided by how much \(x\) moves at all. That is the rate at which \(y\) rises per unit \(x\), exactly what a slope should be.
The slope is co-movement divided by spread: \(\mathrm{Cov}/\mathrm{Var}\). The intercept just parks the line through the centroid \((\bar{x},\bar{y})\). Both come from setting the derivative of squared error to zero.
The total spread of \(y\) around its own mean — the total sum of squares, TSS — can be split into two pieces: the spread the line accounts for (the explained sum of squares, ESS) and the spread left over as residuals (the residual sum of squares, RSS). For each point, the deviation \((y_i-\bar{y})\) decomposes as \((\hat{y}_i-\bar{y})\) (the line's part) plus \((y_i-\hat{y}_i)\) (the residual). Squaring and summing gives \(\text{TSS}=\text{ESS}+\text{RSS}\). \(R^2\) is the fraction of \(y\)'s spread the line explains: \(R^2 = 1 - \text{RSS}/\text{TSS}\). It runs from 0 (the line explains nothing — no better than using \(\bar{y}\) for everyone) to 1 (the line explains everything — every point sits on it). A high \(R^2\) says the line fits this cloud well; it says nothing about causation, and a more flexible model can always push \(R^2\) toward 1 by overfitting, so it is a measure of fit, not of truth.
\(R^2\) is the line's share of \(y\)'s spread. High \(R^2\) means the line summarizes the cloud well — not that the model is right, not that \(x\) causes \(y\), and a wiggly enough model can always force it to 1.
OLS always computes a line — feed it any \(x,y\) and it returns numbers. Whether those numbers mean what you want depends on assumptions about where the data came from. Linearity: the true relationship between \(x\) and \(y\) is straight. If it's curved, the fitted slope is a misleading average and no amount of data fixes that. Independence: observations don't carry information about each other (no clustering, no time series linkage). If it fails, the standard errors in §06 are simply wrong. Homoscedasticity: the spread of the residuals is the same at every \(x\). If it funnels, standard errors are wrong again (though the slope itself is still unbiased). Normality of errors: the residuals are drawn from a normal distribution. This is only needed for the exact \(t\)-tests and confidence intervals in §06; with a decent sample size the central limit theorem lets you mostly ignore it. The first assumption is about the line being the right shape; the other three are about the inference machinery being honest.
Linearity is about the line being the right shape and it fails irrecoverably. Independence and homoscedasticity are about the inference machinery being honest; normality is only needed for exact small-sample \(t\)-tests.
The slope \(b_1\) is an estimate from one sample. Draw a different sample and you'd get a different \(b_1\). That sampling spread has a standard deviation, called the standard error of \(b_1\). Two ingredients set it: how noisy the residuals are (\(\sigma^2\), estimated by \(s^2=\text{RSS}/(n-2)\)), and how much \(x\) actually varies (more spread in \(x\) pins the line's tilt more firmly). Divide \(b_1\) by its SE and you get a \(t\)-statistic: "how many standard errors the estimate sits from zero." Under the null hypothesis that the true slope is zero, it follows a \(t\)-distribution with \(n-2\) degrees of freedom. A large \(|t|\) is unlikely under that null, so we call the slope "statistically significant." A confidence interval brackets the estimate: \(b_1 \pm t^{\!*}\cdot\mathrm{SE}(b_1)\). "Significant" only means we'd be surprised if the true slope were zero — it says nothing about whether the effect is large, whether the model is right, or whether \(x\) causes \(y\).
The slope's standard error shrinks with less residual noise, more spread in \(x\), and more data. "Significant" just means "distinguishable from zero at a chosen threshold" — it is a statement about surprise, not about importance.
Of all the estimators you could cook up for the slope, why use OLS? The Gauss–Markov theorem gives the answer: among all estimators that are linear in \(y\) and unbiased, OLS has the smallest variance. It is the Best Linear Unbiased Estimator — BLUE. The assumptions it needs are linearity, exogeneity (\(E[\varepsilon\mid x]=0\): the errors have mean zero at every \(x\)), homoscedasticity, and no autocorrelation. Notice what is not on that list: normality. Normality is only required for the exact \(t\)-tests in §06; Gauss–Markov holds without it. The theorem is a constrained optimality result — "best" only among estimators that are both linear and unbiased. Drop the unbiasedness constraint and biased estimators (ridge, lasso) can beat OLS on variance, often dramatically, by accepting a little bias to shrink the sampling spread. That trade-off is the door to shrinkage and regularization, which a future blueprint covers.
OLS is the tightest unbiased linear estimator — no normality required for that, only linearity, exogeneity, homoscedasticity, no autocorrelation. "Best" is conditional on staying linear and unbiased; step outside and you trade bias for variance.
Fit simple regression with a binary \(x\) taking values 0 and 1 — say, control vs treated. The math collapses to something familiar: the slope \(b_1\) is exactly the difference between the two group means, and the intercept \(b_0\) is the control mean. In other words, simple regression on a binary predictor is the naive group comparison — the very thing the causal-inference blueprint warns lies (causal-inference §01). The line isn't doing any adjustment; with one predictor there is nothing to adjust for. Adjustment enters when you add more covariates (multiple regression, a future blueprint): each coefficient becomes the effect of its variable holding the others fixed. That is the regression analogue of matching — instead of finding literal twins, you fit a plane and read off a partial slope. Both methods lean on the same unconfoundedness assumption: that the covariates you conditioned on capture every trait driving both the treatment and the outcome. Regression is just a smoother, continuous way to compare like with like.
With a binary predictor, regression's slope is just the group-mean gap — the naive estimate, unadjusted. Adjustment begins only with more covariates, and then regression and matching are two routes to the same unconfoundedness destination.
A straight line is the simplest story two variables can tell — a baseline plus a constant rate — and OLS picks the line that minimizes total squared vertical error, squaring so misses neither cancel nor stay cheap and the math yields a closed form. The slope is co-movement over spread, \(\mathrm{Cov}/\mathrm{Var}\), and the line always passes through the centroid; \(R^2\) is the share of \(y\)'s variance the line explains. The line is only honest under linearity, independence, homoscedasticity, and (for exact inference) normality; the slope's standard error tells you how much it wobbles sample to sample, and "significant" means distinguishable from zero — not important or causal. Gauss–Markov makes OLS the tightest linear unbiased estimator, conditional on staying linear and unbiased. And with a binary predictor, regression's slope is just the group-mean gap — the naive estimate — so adjustment, and the bridge to causal inference, only opens once further covariates enter the model.