A straight line is about to make a medical mistake.

Some questions aren't “how much?” They're “which one?”

“How much will this house sell for?” has a number for an answer, and a straight line through the data — linear regression — handles it beautifully. But look at the questions a hospital, a bank, or an email provider actually asks: Is this tumor malignant? Will this loan default? Is this message spam? The answer isn't a quantity. It's a side: yes or no, this class or that one.

That's classification, and it's one of the most useful things a machine can do. The trick is that we still want a number back — not a flat “yes,” but a probability: “92% likely malignant” tells a doctor far more than a bare label. Logistic regression is the workhorse that turns a row of numbers into exactly that: a calibrated probability of “yes.”

Watch one extreme case break the straight line.

Here's the honest experiment. Take tumors, plot each one by size, and label it benign (0) or malignant (1). Fit two models to the exact same points: a straight line (linear regression) and a logistic sigmoid. On clean data they agree. Then press “Add an extreme case” — a very large, very obviously malignant tumor — and watch what each model does.

The line tilts. That far-out point is technically “more than correct,” and because a line is judged on squared distance, it tips the whole line to get closer to the outlier — which slides its 0.5 crossing to the right. Tumors it used to call malignant now fall below the threshold and get a red ring: newly misclassified, by adding a point everyone agrees on. The sigmoid just flattens toward 1 and ignores the tug.

For an ambitious reader, it's worth naming the three separate things going wrong — because they map onto three real reasons the field moved past the line.

A weighted sum, an S-curve, and a line in the sand.

Logistic regression starts exactly like linear regression: multiply each feature by a weight, add them up, add a bias. Call that sum z. If a bigger z should mean “more likely yes,” then every weight is just how much that feature pushes toward yes.

The one new idea is what you do with z. Instead of reporting it raw, you pass it through the sigmoid — an S-shaped squasher that maps any number, from −∞ to +∞, into the range 0 to 1. A big positive z comes out near 1 (“almost certainly yes”), a big negative one near 0, and z = 0 comes out at exactly 0.5 — dead uncertain.

The decision boundary is just where z = 0

Since the model predicts “yes” whenever the probability clears 0.5, and the probability is 0.5 exactly when z = 0, the frontier between the two decisions is the set of points where w₁x₁ + w₂x₂ + b = 0. That is the equation of a straight line (a flat plane, with more features). This is the decision boundary: cross it and the verdict flips. It's why plain logistic regression always carves the world with a straight edge — the weights can rotate and shift that edge, but never bend it.

Below, two clouds of points and a boundary that begins as a random guess. Press Run and watch the line swing into position while the background — the model's predicted probability everywhere — sharpens from a muddy blur into two confident regions. (How it knows which way to swing is the next section.)

Two things to notice. The boundary is always straight — that's the model's honest limit. And the shading is soft: right next to the line the model is barely 55% sure, far from it the model is 99% sure. Logistic regression never just points; it tells you how strongly it's pointing.

How the line knows which way to move.

“Training” means finding the weights that make the model right. To search for them you need one number that scores how wrong the current weights are — a cost function — and a way to walk it downhill.

The cost: reward honesty, punish confident lies

Before a machine can get better, it needs to know how wrong it is right now, as a single number. The clever part is how logistic regression counts wrongness — it doesn't just tally right and wrong answers, it weighs how confident each one was.

For each example, log loss asks one question — what probability did you put on the answer that actually happened? — and turns it into a penalty. Land near 1 and the penalty is almost nothing. Drift toward 0 and it climbs, gently at first, then straight off a cliff:

That cliff on the bottom row is the whole personality of log loss: being confidently wrong is the cardinal sin. And notice it never lets the model off the hook with a lazy 0.5 either — a permanent coin flip still carries a real penalty, so the model is always pushed to commit to a side when the evidence is there.

Only one of the two terms is ever alive: if y = 1 the cost is −log(p), if y = 0 it's −log(1−p). Averaged over every example, that's the single number the training tries to shrink — the LOG LOSS you saw ticking down in the lab.

The descent: walk downhill in the fog

So we have a wrongness score. Now — how does the model actually lower it, when there are millions of possible weight settings and no map telling it which is best?

That is gradient descent, and the “tilt under your feet” has a name: the gradient, the direction of steepest uphill. Step the opposite way and the cost drops a little; feel, step, repeat. Here's the gift log loss gave us — its landscape has just one valley, no false dips to get trapped in, so every downhill path arrives at the same bottom. It barely matters where you start. And each weight gets nudged in proportion to how much it was to blame for the error — a beautifully simple rule falls out of the math:

Read it in plain English: the update is the error times the input. If the model predicted 0.9 but the truth was 0 (error +0.9), any feature that was large and pushed toward “yes” gets its weight pulled down. The α, the learning rate, is your step size — too small and training crawls, too big and it overshoots the valley and bounces. That loop, run a few hundred times, is every “Step once” you clicked stitched together — and it's the same engine, scaled up, that trains the giant neural networks behind modern AI. The idea that a single error-times-input rule underlies all of it is exactly the kind of thread a 1:1 session can pull all the way through.

A model that memorizes has learned nothing.

A straight boundary can be too rigid — miss an obvious curve in the data (underfitting). The tempting fix is to give the model more power: add squared and cross terms so the boundary can bend. Push that too far and you get the opposite disease. The boundary starts contorting to capture every last point, including the noise — the mislabeled scan, the fluke. That's overfitting: brilliant on the data it studied, useless on anything new.

The tell is the gap between two scores: high accuracy on the training data, low accuracy on data it hasn't seen. The overfit boundary on the right detours just to swallow that one stray point — and that detour is exactly what will misfire on the next patient. The main defenses are worth knowing by name:

Five ideas, one chain.

Everything here links up into a single sentence you can reconstruct the whole model from:

Curious who's behind these lessons? See the proof of work.