B4B33RPZ · Pattern Recognition and Machine Learning

Logistic regression

Lab 6: learning the posterior directly, the loss as a likelihood, and regularisation as a prior

Klára JanouškováFEE CTU, Prague

Warm-up: ML and MAP

A coin lands heads 3 times out of 3. The ML estimate of $\theta = P(\text{heads})$ is…

$\hat\theta_{\text{ML}} = \frac{N_H}{N} = \frac{3}{3} = 1$ . ML trusts the data completely, even 3 flips. Keep this in mind: today ML will want some weights to be infinite.

MAP with a flat (uniform) prior gives…

$\hat\theta_{\text{MAP}} = \arg\max_\theta p(D \mid \theta)\,p(\theta)$ ; a constant $p(\theta)$ does not move the maximum. MAP = ML + "what we believed before". Today: that belief becomes a penalty on the weights.

Part 1

From Bayes to logistic regression

Two classes: which one do we pick?

Will a student pass the test? We know $x$ = hours studied. Classes $k \in \{-1, +1\}$ : fail / pass. With the 0/1 loss (Bayes lab), which class should $q(x)$ say?

↓ 0/1 loss: pick the class with the larger posterior

q(x) = \arg\max_k\, p(k \mid x)

↓ only two classes: compare the two posteriors through their ratio

q(x) = +1 \iff \frac{p(+1 \mid x)}{p(-1 \mid x)} > 1

↓ $\ln$ is increasing, and $\ln 1 = 0$ : ratio $> 1 \iff$ its log $> 0$

q(x) = +1 \iff a(x) = \ln \frac{p(+1 \mid x)}{p(-1 \mid x)} > 0

The log-odds

Odds = "how many to one", as in betting. $p = 0.75$ → odds $\frac{0.75}{0.25} = 3$ (3 : 1 for passing), $a = \ln 3 = 1.1$ .

$a > 0$ : pass more likely; $a < 0$ : fail more likely; $a = 0$ : 50/50.

Have we seen this ratio before?

$\dfrac{p(+1 \mid x)}{p(-1 \mid x)}$ looks like the likelihood ratio $\dfrac{p(x \mid +1)}{p(x \mid -1)}$ from the Bayes and minimax labs. Same thing?

No. Posterior: "class given data". Likelihood: "data given class".

↓ Bayes' rule on top and bottom; $p(x)$ is in both, it cancels

\underbrace{\frac{p(+1 \mid x)}{p(-1 \mid x)}}_{\text{posterior odds}} = \underbrace{\frac{p(x \mid +1)}{p(x \mid -1)}}_{\text{likelihood ratio}} \cdot \underbrace{\frac{p(+1)}{p(-1)}}_{\text{prior odds}}

Equal only when the priors are equal.

A coin: biased ( $P(H) = 0.8$ ) or fair ( $0.5$ )? We see one head. Likelihood ratio (data alone):

0.8 / 0.5 = 1.6

a head is 1.6× more likely with the biased coin

Priors 0.5 / 0.5: posterior odds $= 1.6 \cdot 1$

= 1.6

, so

P(\text{biased} \mid H) = 1.6 / 2.6 = 0.62

Sceptical prior $P(\text{biased}) = 0.1$ : odds $= 1.6 \cdot 0.1 / 0.9$

= 0.18

, so

P(\text{biased} \mid H) = 0.18 / 1.18 = 0.15

What if we learn $a(x)$ directly?

So far: generative (MLE lab)

Estimate $p(x \mid k)$ and $p(k)$ for each class, then Bayes' rule, then $a(x)$ .

A whole density per class, just to get one number per $x$ .

Today: logistic regression

Skip the densities. Assume the log-odds is linear in $x$ and learn its weights:

a(\mathbf x) = w_0 + w_1 x_1 + \dots + w_D x_D = \mathbf w \cdot \mathbf x

with $\mathbf x = (1, x_1, \dots, x_D)$ : the extra 1 carries the bias $w_0$ .

Like a points system: every feature adds or subtracts "evidence for passing"; the weight says how much per unit.

Example:

a(x) = -3 + 1 \cdot x

. At

x = 3

h:

a = 0

, even odds. Each extra hour adds 1 to the log-odds, i.e. multiplies the odds by

e^1 \approx 2.7

.

From log-odds back to a probability

We know $a$ . What is $p = p(+1 \mid x)$ ? (Then $p(-1 \mid x) = 1 - p$ .)

\ln \frac{p}{1 - p} = a

↓ exponentiate both sides

\frac{p}{1 - p} = e^{a} \quad\Rightarrow\quad p = e^{a} - p\,e^{a}

↓ multiplied by $(1 - p)$ above; now move $p\,e^a$ to the left and factor out $p$

p\,(1 + e^{a}) = e^{a} \quad\Rightarrow\quad p = \frac{e^{a}}{1 + e^{a}}

↓ divide top and bottom by $e^{a}$

p(+1 \mid \mathbf x) = \frac{1}{1 + e^{-a}} = \sigma(a)

Example

$a(x) = x - 3$ , $\ \sigma(a) = 1 / (1 + e^{-a})$ :

x = 3

:

\sigma(0) = 1 / (1 + 1) = 0.5

x = 5

:

\sigma(2) = 1 / (1 + e^{-2}) = 0.88

x = 1

:

\sigma(-2) = 1 / (1 + e^{2}) = 0.12

0.88 and 0.12 add up to 1: a hint for the next slide.

One formula for both classes

$p(+1 \mid \mathbf x) = \sigma(a)$ . What is $p(-1 \mid \mathbf x)$ ?

p(-1 \mid \mathbf x) = 1 - \sigma(a) = 1 - \frac{1}{1 + e^{-a}}

↓ common denominator

= \frac{1 + e^{-a} - 1}{1 + e^{-a}} = \frac{e^{-a}}{1 + e^{-a}}

↓ divide top and bottom by $e^{-a}$

= \frac{1}{e^{a} + 1} = \sigma(-a)

With

k \in \{-1, +1\}

both cases are one formula:

p(k \mid \mathbf x) = \sigma(k\, \mathbf w \cdot \mathbf x)

This is why the assignment uses $\pm 1$ and not $\{0, 1\}$ .

Part 2

The sigmoid and its weights

The logistic sigmoid

\sigma(a) = \frac{1}{1 + e^{-a}}

squashes any score $a \in (-\infty, \infty)$ into $(0, 1)$ : a probability
$\sigma(0) = 0.5$ (circle); $\sigma(-a) = 1 - \sigma(a)$
…does it look familiar? $\tanh(a) = 2\,\sigma(2a) - 1$ : the same S, stretched to $(-1, 1)$
both are classic activation functions in neural networks

What do $w_0$ and $w_1$ do?

One feature: $p(+1 \mid x) = \sigma(w_0 + w_1 x)$ . Guess first: what happens when we change $w_0$ ? And $w_1$ ?

$w_0$ shifts the curve left / right: it moves the point where $p = 0.5$ .

$w_1$ sets the steepness: how fast the evidence grows with $x$ . Negative $w_1$ flips the S.

Decision boundary: $p = 0.5 \iff a(x) = 0$ . Where is it in 1D? That is your get_threshold.

Two features: what does the boundary look like?

$a(\mathbf x) = w_1 x_1 + w_2 x_2$ (here $w_0 = 0$ ). Where is $p$ the same?

Equal $p$ ⇔ equal $a$ : parallel straight lines. The boundary $a = 0$ is a line (a plane, a hyperplane in $D$ dimensions).

$\mathbf w$ is perpendicular to them and points to class $+1$ . Longer $\mathbf w$ → lines closer together → a steeper, more confident model.

$w_0$ shifts the line away from the origin.

Quick check

For one student, $\mathbf w \cdot \mathbf x = 0$ . What is $p(\text{pass} \mid \mathbf x)$ ?

$p(+1 \mid \mathbf x) = \sigma(\mathbf w \cdot \mathbf x) = \sigma(0) = \frac{1}{1 + e^0} = 0.5$ : the student sits exactly on the boundary.

We replace $\mathbf w$ by $-\mathbf w$ . What happens?

$\sigma(-\mathbf w \cdot \mathbf x) = 1 - \sigma(\mathbf w \cdot \mathbf x)$ . Same boundary line, but $\mathbf w$ now points to the other class.

Part 3

Which $\mathbf w$ ? The likelihood is the loss

How good is a given $\mathbf w$ ?

Let the model bet on what really happened. A good $\mathbf w$ gives high probability to the true labels. Try $\mathbf w = (-3, 1)$ , i.e. $a = x - 3$ :

student	$x$ (h)	true $k$	$a = x - 3$	$p(k \mid x) = \sigma(k\,a)$
Ann	1	$-1$ fail	$-2$	$\sigma(2) = 0.88$
Ben	4.5	$+1$ pass	$1.5$	$\sigma(1.5) = 0.82$
Cid	3.5	$-1$ fail	$0.5$	$\sigma(-0.5) = 0.38$

All three right at once (independent students): multiply.

L(\mathbf w) = \prod_i p(k_i \mid \mathbf x_i, \mathbf w)

= 0.88 \cdot 0.82 \cdot 0.38 = 0.27

This is the likelihood of $\mathbf w$ , exactly as for the coin last week. ML: pick the $\mathbf w$ with the largest $L$ .

From the likelihood to a loss

L(\mathbf w) = \prod_{i=1}^{N} p(k_i \mid \mathbf x_i, \mathbf w) = \prod_{i=1}^{N} \sigma(k_i\, \mathbf w \cdot \mathbf x_i)

↓ $\ln$ : the product becomes a sum (and no tiny numbers: $0.9^{1000} \approx 10^{-46}$ ). Maximise $\iff$ minimise the negative

\hat{\mathbf w}_{\text{ML}} = \arg\min_{\mathbf w} \sum_{i} -\ln \sigma(k_i\, \mathbf w \cdot \mathbf x_i)

↓ $-\ln \sigma(m) = -\ln \frac{1}{1 + e^{-m}} = \ln(1 + e^{-m})$ ; dividing by $N$ does not move the minimum

E(\mathbf w) = \frac{1}{N} \sum_{i=1}^{N} \ln\!\big(1 + e^{-k_i\, \mathbf w \cdot \mathbf x_i}\big)

the assignment's loss = the (logistic) cross-entropy

Ann, Ben, Cid

$-\ln$ of each bet:

0.13 + 0.20 + 0.97 = 1.30

$-\ln L$ :

-\ln 0.27 = 1.30

: the same

$-\ln p$ = surprise. A confident wrong bet (Cid) costs most.

What does each student cost?

Margin $m = k\, \mathbf w \cdot \mathbf x$ : positive = on the right side of the boundary, large = confident.

$m = -2$ : wrong and confident: 2.13. Far on the wrong side the cost grows like $|m|$ .

$m = 0$ : on the boundary, "50/50": $\ln 2 =$ 0.69.

$m = 2$ : right and confident: 0.13. Never exactly 0: still a small push to be even more sure.

Quick check

Which training example adds the most to $E(\mathbf w)$ ?

Loss $\ln(1 + e^{-m})$ : $m = -3$ gives $\ln(1 + e^{3}) = 3.05$ , $m = 0$ gives $0.69$ , $m = 3$ gives $0.05$ . Wrong and confident costs most.

An example is classified correctly ( $m > 0$ ). Can its loss be larger than 0.5?

$\ln(1 + e^{-m}) > 0.5$ for $m < 0.43$ . Right side but barely: the model was not sure, so it still pays.

Try it: find a good $\mathbf w$ by hand

w_0

(shift)

w_1

(steepness)

Can we solve for the best $\mathbf w$ directly?

For $\mu$ of a Normal we set the derivative to 0 and solved. Can we do that for $E(\mathbf w)$ ?

No closed form: $\mathbf w$ sits inside $N$ sigmoids, the equation $\nabla E = 0$ cannot be solved by hand.

But $E$ is convex: one bowl, no local minima. So just walk downhill.

Gradient descent: in thick fog on a hill, feel the slope under your feet and step the other way.

\mathbf w \leftarrow \mathbf w - \eta\, \nabla E(\mathbf w)

$\eta$ : the step size (learning rate)

Homework

$\nabla E(\mathbf w)$ : derive it with the chain rule through $\ln$ and $\sigma$ (logistic_loss_gradient).

How big should the step be?

too small: safe, but many steps
just right: a few steps to the bottom
too large: jumps over the minimum and can diverge. The assignment adapts $\eta$ (next slide).

Gradient descent in the assignment

# input: x (observations), k (labels), w_init
w = w_init
step_size = 1.0
E, g = compute_E_and_gradient(x, k, w)

while not TERMINATION_CONDITION:
    E_new, g_new = compute_E_and_gradient(
                       x, k, w - step_size * g)
    if E_new < E:          # success
        w -= step_size * g
        g, E = g_new, E_new
        step_size *= 2
    else:
        step_size /= 2
return w

Accept a step only if $E$ decreases. If not, the step was too large (halve it) or we are already at the optimum.

After a success, try a bigger step (double it).

The gradient is normalised by $N$ ( $E$ is a mean), otherwise its size grows with $N$ and you would need smaller steps for more data.

Stop when $\|\mathbf w_{\text{new}} - \mathbf w_{\text{prev}}\| \le \varepsilon$ .

Try it: gradient descent on the students

step size

\eta

adaptive step (assignment rule) standardise

x

(zero mean, unit variance)

Part 4

Why is it called cross-entropy?

Where have we seen $-\ln p$ before?

We already have the loss: for each student, $-\ln$ of the probability the model gave to what really happened.

E(\mathbf w) = \frac{1}{N} \sum_{i} -\ln q(k_i \mid \mathbf x_i), \qquad q(k \mid \mathbf x) = \sigma(k\, \mathbf w \cdot \mathbf x)

Information theory has a name for $-\ln p$ : the surprise of an outcome with probability $p$ .

\text{surprise} = -\ln p

p = 1

: 0, no surprise ·

p = 0.5

: 0.69 ·

p = 0.01

: 4.6, very surprised

So training = making the model as little surprised as possible by the true labels. The rest of the name comes on the next slides.

Cross-entropy: surprise when you believe the wrong thing

In general the truth is a distribution $p$ over the classes, but you bet with the model's belief $q$ . Your average surprise:

H(p, q) = -\sum_k \underbrace{p(k)}_{\text{truth}} \ln \underbrace{q(k)}_{\text{belief}}

↓ one training example: we know its label, so the "truth" puts all its probability on $k_{\text{true}}$ (one-hot): only one term survives

H(p, q) = -\ln q(k_{\text{true}} \mid \mathbf x) = \text{the loss of that example}

Minimising cross-entropy = maximising the likelihood of the true labels. Same thing, two names.

The usual case: two classes, hard labels

Two classes: the truth is $p = (t,\ 1 - t)$ , the model says $q = (\sigma(a),\ 1 - \sigma(a))$ .

↓ write out the sum over the two classes

H(p, q) = -t \ln \sigma(a) - (1 - t) \ln(1 - \sigma(a))

the "binary cross-entropy" of every library

A hard label is one-hot: $t = 1$ (pass) or $t = 0$ (fail). One term vanishes. And $H(p) = 0$ : a one-hot $p$ holds no surprise, so nothing is "cross" yet.

The assignment writes $k \in \{-1, +1\}$ and $\ln(1 + e^{-k a})$ . Same?

↓ $k = +1$ ( $t = 1$ ): only the first term

-\ln \sigma(a) = \ln(1 + e^{-a})

↓ $k = -1$ ( $t = 0$ ): only the second; $1 - \sigma(a) = \sigma(-a)$

-\ln \sigma(-a) = \ln(1 + e^{a})

↓ both are $\ln(1 + e^{-k a})$

Same loss, the labels are just written differently.

Every label is hard. Where does a soft $p$ come from?

Ten students all studied 3 h: 7 passed, 3 failed. Each label is hard. What should the model say at $x = 3$ ?

↓ add up their losses: 7 × $-\ln q$ and 3 × $-\ln(1 - q)$ ; divide by 10

-0.7 \ln q - 0.3 \ln(1 - q) = H(\hat p, q), \quad \hat p = (0.7,\ 0.3)

The best $q$ is the honest one, $q = \hat p$ : 0.61. The overconfident 0.9 is worse than knowing nothing (0.5).

↓ at $q = \hat p$ the cross-entropy is the entropy of $\hat p$ : its own average surprise

H(\hat p) = -\sum_k \hat p(k) \ln \hat p(k) = 0.61

A floor: same $x$ , different outcomes. No model can do better.

KL $= H(\hat p, q) - H(\hat p)$ : the part a better model can remove.

Which weather forecaster is better?

10 days, it rained on 8. Three forecasters say the same number every day:

	says "rain" with	accuracy (cut at 0.5)	cross-entropy $-0.8 \ln q - 0.2 \ln(1 - q)$
Alice	99 %	80 %	0.93
Bob	80 %	80 %	0.50
Cecil	50 %	undecided	0.69

Accuracy cannot tell Alice and Bob apart. Cross-entropy can: it rewards honest probabilities and punishes overconfidence hard.
That is why we train with cross-entropy and not with accuracy (also: accuracy has zero gradient almost everywhere).

Quick check

Your training loss stops at 0.45 and will not go lower, whatever you try. Is it a bug?

If the same $\mathbf x$ comes with different labels, the best possible loss is the entropy of the labels there, not 0. (A too large step makes the loss go up or jump around, not stall.)

A model always says 99 % for the class it picks, and is right 80 % of the time. Compared with a model saying 80 %, its cross-entropy is…

$H(p, q) = -\sum_k p(k) \ln q(k)$ : $-0.8 \ln 0.99 - 0.2 \ln 0.01 = 0.93$ vs. $0.50$ for the honest 80 %.

Part 5

Regularisation is a prior

What if the classes do not overlap?

New class: nobody under 3 h passed, nobody over 3 h failed. What does ML do?

$w_1 = 1$ : everyone on the right side, but unsure.

$w_1 = 3$ : surer, $E$ drops from 0.24 to 0.05.

Steeper is always a bit better: $E \to 0$ only as $w_1 \to \infty$ . ML wants infinite weights: a step function, 100 % sure.

It would bet everything that a 3.1 h student passes. Remember: 3 heads of 3 → $\hat\theta_{\text{ML}} = 1$ .

We want to say: "weights are probably not huge". That is a prior → MAP.

MAP: the data plus what we believed before

Last week, MAP: maximise the posterior instead of the likelihood.

\hat{\mathbf w}_{\text{MAP}} = \arg\max_{\mathbf w}\, p(\mathbf w \mid D)

↓ Bayes' rule; $p(D)$ does not depend on $\mathbf w$ , drop it

= \arg\max_{\mathbf w}\, \textcolor{orange}{p(D \mid \mathbf w)}\, \textcolor{blue}{p(\mathbf w)}

↓ $-\ln$ : the product becomes a sum, max becomes min

= \arg\min_{\mathbf w} \Big[\underbrace{-\ln \textcolor{orange}{p(D \mid \mathbf w)}}_{\text{data loss}} \;\underbrace{-\ln \textcolor{blue}{p(\mathbf w)}}_{\text{penalty}}\Big]

Like a tug of war

Likelihood: "fit the data".

Prior: "stay believable".

MAP: the compromise. The coin: 7 heads of 8 with prior Beta(5, 5) gave $\hat\theta_{\text{MAP}} = \frac{11}{16} = 0.69$ instead of $0.88$ .

Which prior for the weights?

"Weights are probably small." As a distribution for each weight $w$ ?

↓ a Gaussian centred at 0: most of its mass near 0 ( $|w| < 2s$ with 95 %)

\textcolor{blue}{p(w)} \propto \exp\!\Big(-\frac{w^2}{2 s^2}\Big)

↓ take $-\ln$ : a parabola, 0 at $w = 0$ , growing with $|w|$ : a penalty for large weights

-\ln \textcolor{blue}{p(w)} = \frac{w^2}{2 s^2} + \text{const}

↓ independent weights: the product of priors becomes a sum of penalties

-\ln \textcolor{blue}{p(\mathbf w)} = \frac{1}{2 s^2} \sum_j w_j^2 + \text{const} = \frac{\|\mathbf w\|^2}{2 s^2} + \text{const}

Narrow prior (small $s$ ) = steep parabola = strong push towards 0.

Putting it together: the regularised loss

Data loss $-\ln \textcolor{orange}{p(D \mid \mathbf w)} = N\,E(\mathbf w)$ (the sum, not the mean). Add the penalty:

\hat{\mathbf w}_{\text{MAP}} = \arg\min_{\mathbf w} \Big[ N\, \textcolor{orange}{E(\mathbf w)} + \textcolor{blue}{\frac{\|\mathbf w\|^2}{2 s^2}} \Big]

↓ divide by $N$ : the minimum stays where it is

= \arg\min_{\mathbf w} \Big[ \textcolor{orange}{E(\mathbf w)} + \lambda\, \textcolor{blue}{\|\mathbf w\|^2} \Big], \qquad \lambda = \frac{1}{2 s^2 N}

L2 regularisation (weight decay in neural networks) = MAP with a Gaussian prior.

"L2 = weight decay" holds for plain gradient descent; with Adam the two differ, hence AdamW.

How to read it

Narrow prior (small $s$ ): large $\lambda$ , small weights.

More data (large $N$ ): small $\lambda$ : the data outvote the prior, as with the coin.

In practice the bias $w_0$ is usually not penalised: shifting the boundary is not "overconfidence".

L2 or L1?

A Laplace prior $p(w) \propto e^{-|w|/b}$ gives the penalty $\lambda \sum_j |w_j|$ : L1 (lasso).

Picture: the smallest data loss (orange ellipses around ML) that still has a small penalty (blue region).

The L1 diamond has corners on the axes: the solution often lands there, so some weights are exactly 0: feature selection.

L2 shrinks all weights a little, rarely to 0; smooth, easy to optimise.

What it looks like on MNIST

Weight image without regularisation: noisy, every pixel has a weight, values up to about 2. — no regularisation

Weight image with L2: smooth blobs, all weights small, about 0.1 at most. — L2: small and smooth

Weight image with L1: almost all weights exactly 0, a few pixels in the centre non-zero. — L1: a few pixels, the rest 0

The 784 weights of a digit classifier drawn as a 28×28 image (red positive, blue negative, white 0). Mind the different colour scales.

Slide credit: Jonas

Try it: the strength of the prior $\lambda$

data

\lambda

Quick check

$\lambda \to \infty$ (bias not penalised). What does the model predict?

The penalty forces $w_1 \to 0$ , so $p = \sigma(w_0)$ for everyone: the share of passing students. An infinitely narrow prior: the data are ignored.

We collect 10× more students, same prior. The effect of the prior…

$\lambda = \frac{1}{2 s^2 N}$ : 10× more data, 10× smaller $\lambda$ . The data outvote the prior; MAP → ML.

How do we choose $\lambda$ ?

Or the features, or the model: how do we know we chose right?

Not on the training data: the training loss is always lowest at $\lambda = 0$ (like $h \to 0$ for Parzen windows).

Validation: hold out part of the data, train with each

\lambda

, keep the one with the lowest validation error.

The test set is touched once, at the very end, to report the result.

Part 6

Logistic regression in the big picture

Must we cut at $p = 0.5$ ?

The school offers extra tutoring. Tutoring a student who would pass anyway costs 1; missing one who fails costs 4. When do we decide $d =$ "tutor"?

↓ Bayes lab: risk of a decision = expected loss under the posterior $p = p(+1 \mid \mathbf x)$

R(\text{tutor}) = 1 \cdot p, \qquad R(\text{skip}) = 4\,(1 - p)

↓ tutor when its risk is smaller; collect $p$ on the left

p < 4 - 4p \iff 5p < 4 \iff p < 0.8

Logistic regression gives probabilities, so a new loss only moves the cut. No retraining. The perceptron and SVM (next labs) give only a side of the line.

Generative or discriminative?

Discriminative (logistic regression): learns $p(k \mid \mathbf x)$ directly. Only "how to label", not how $\mathbf x$ is produced.

Generative (MLE lab + Bayes): learns $p(\mathbf x \mid k)\,p(k)$ . Can also generate new samples, and gives $p(k \mid \mathbf x)$ via Bayes. Not the other way round.

How many numbers do we need to learn?

Generative: a Gaussian per class

In $D$ dimensions: mean $D$ + covariance (symmetric) $D(D+1)/2$ , two classes, one prior:

2\,(D + D(D+1)/2) + 1 = O(D^2)

MNIST,

D = 784

: 617 009

Logistic regression

one weight per feature + bias:

D + 1 = O(D)

MNIST: 785

Fewer parameters: less data needed to estimate them well.
$D$ too big: slow to evaluate and learn → GPUs, feature selection (AdaBoost later). $N$ too big: use a random minibatch per step → stochastic gradient descent.

Can we read the weights?

Every weight says how much its feature moves the log-odds.

Students: $w_1 = 1.79$ per hour. One more hour multiplies the odds of passing by $e^{1.79} \approx 6$ .

$|w_j|$ large → feature $j$ matters. Only if the features have the same scale (hours vs minutes!): standardise first.

← more interpretableless interpretable →

decision trees

linear regression

logistic regression

random forest

neural network

CNN

More than two classes?

One score per class, $a_k = \mathbf w_k \cdot \mathbf x$ . How to turn $K$ scores into probabilities?

↓ $e^{a}$ makes them positive, dividing by the sum makes them add up to 1

p(k \mid \mathbf x) = \frac{e^{a_k}}{\sum_{j=1}^{K} e^{a_j}} \quad \text{(softmax)}

↓ $K = 2$ : divide top and bottom by $e^{a_1}$

p(1 \mid \mathbf x) = \frac{e^{a_1}}{e^{a_1} + e^{a_2}} = \frac{1}{1 + e^{-(a_1 - a_2)}} = \sigma(a_1 - a_2)

The sigmoid is softmax with two classes; only the difference of scores matters.

Example: 5 classes

scores $(1.3,\ 5.1,\ 2.2,\ 0.7,\ 1.1)$

probabilities

(0.02,\ 0.90,\ 0.05,\ 0.01,\ 0.02)

The largest score wins, but every class keeps some probability. Loss: $-\ln$ of the true class's probability, as before.

Logistic regression is one neuron

Weighted sum, then a sigmoid: the classic artificial neuron.
Add a hidden layer: the network learns its own features $h$ , then does logistic regression on them. Same loss (cross-entropy), same training (gradient descent), L2 = weight decay (with Adam: AdamW).

Only straight boundaries?

Feature lifting: transform $\mathbf x$ with fixed functions (here $x \mapsto (x, x^2)$ ), then the boundary is straight in the new space and curved in the old one. Bonus task: arctan features.
Which functions? Try a set, validate, intuition… Learning them instead: that is deep learning.

Part 7

The assignment

The assignment in one slide

Part 1: letters A vs C

One feature (left half minus right half of the image, standardised), labels $k \in \{-1, +1\}$ .

Loss (given): $E(\mathbf w) = \frac1N \sum \ln(1 + e^{-k\, \mathbf w \cdot \mathbf x})$

You: its gradient, gradient descent with the adaptive step, the 1D threshold, classification.

Part 2: MNIST digits

784 pixels + bias, the same code. Look at the learned weights as an image.

Bonus

Feature lifting: a curved boundary (Prince, ch. 9). Keep $k \in \{-1, +1\}$ .

Logistic regression in five lines

Assume the log-odds is linear: $p(k \mid \mathbf x) = \sigma(k\, \mathbf w \cdot \mathbf x)$ . Boundary: a hyperplane.
ML for the labels = minimise the cross-entropy $E(\mathbf w)$ ("surprise" at the true labels).
No closed form, but convex: gradient descent.
MAP with a Gaussian prior = L2 regularisation: it stops infinite weights. Pick $\lambda$ by validation.
Softmax for $K$ classes; stack neurons and you have a neural network.

More on cross-entropy, and logistic regression today

Cross-entropy, entropy and KL divergence

How much extra surprise do we pay for believing $q$ instead of the truth $p$ ?

D_{\text{KL}}(p \,\|\, q) = \sum_k p(k) \ln \frac{p(k)}{q(k)}

↓ $\ln \frac{p}{q} = \ln p - \ln q$ , split the sum

= \sum_k p(k) \ln p(k) - \sum_k p(k) \ln q(k)

↓ recognise both sums

= -H(p) + H(p, q) \;\ge\; 0

$H(p)$ does not depend on the model, so minimising $H(p, q)$ = minimising $D_{\text{KL}}$ .

Why train with CE and not KL? Same minimum, and with one-hot labels $H(p) = 0$ anyway; CE is simpler and its value does not shift with the label mix of each minibatch.

Soft labels on purpose

Same formula $H(p, q) = -\sum_k p(k) \ln q(k)$ , but now each example's target $p$ is soft on purpose:

Label smoothing (Inception v3, ViTs): target $(1 - \varepsilon,\ \varepsilon)$ instead of $(1, 0)$ .
Knowledge distillation: a small "student" network learns from the teacher's probabilities $q_{\text{teacher}}$ ("this 7 looks a bit like a 1").
Annotator disagreement / multi-label: 3 of 5 annotators say "cat", 2 say "dog" → $p = (0.6, 0.4)$ .

Why smoothing helps

Target $(0.95,\ 0.05)$ : the best $q$ is $0.95$ , not $1$ .

So the weights stay finite: the same cure as the prior on separable data, from the other side.

Szegedy et al., Rethinking the Inception architecture (2016); Hinton et al., Distilling the knowledge in a neural network (2015)

Separable data: where does gradient descent go?

No regularisation, separable classes: $\|\mathbf w\| \to \infty$ . But in which direction?

Gradient descent on the logistic loss converges in direction to the maximum-margin line: the one as far as possible from the nearest points of both classes.

That line is exactly what a (hard-margin) SVM finds on purpose: lab 9.

Slowly (like $1 / \ln t$ ), but it explains part of why huge, unregularised networks still generalise: the optimiser itself prefers "simple", large-margin solutions (implicit bias).

Soudry et al., The implicit bias of gradient descent on separable data, JMLR 2018

Logistic regression inside a vision–language model

SigLIP (behind many current VLMs): one batch of $B$ images and their captions. Every image–text pair $(i, j)$ is one training example of logistic regression:

a_{ij} = t\; \mathbf x_i \cdot \mathbf y_j + b, \qquad k_{ij} = \begin{cases} +1 & i = j \\ -1 & i \ne j \end{cases}

↓ the assignment's loss, summed over all $B^2$ pairs

L = \frac{1}{B} \sum_{i, j} \ln\!\big(1 + e^{-k_{ij}\, a_{ij}}\big)

Only $B$ of the $B^2$ pairs match, so the bias starts at $b = -10$ : the prior log-odds of "this pair matches" are very negative. CLIP uses softmax over each row instead.

Zhai et al., Sigmoid loss for language image pre-training, ICCV 2023; Radford et al., CLIP, 2021

How are big vision models evaluated?

CLIP, DINOv2, … are trained without our labels. How good are their features for a new task?

Linear probe: freeze the network, take its features $\mathbf h(\mathbf x)$ , and train logistic regression on top. Better features → higher probe accuracy.

CLIP's paper: scikit-learn's logistic regression, with the L2 strength $\lambda$ chosen on a validation set. Exactly today's lab, on 768-dim features instead of 1 or 784.

It is the "feature lifting" slide with the lifting learned on millions of images.

Radford et al., CLIP, 2021 (appendix A); Oquab et al., DINOv2, 2023

Language models: the same loss, 100 000 classes

An LLM reads "The cat sat on the" and must predict the next token. How is it trained?

One score per token of the vocabulary (~100k classes), softmax, loss $-\ln q(\text{true next token})$ : multinomial logistic regression on top of a huge network.

↓ the reported number is the exponential of the average cross-entropy

\text{perplexity} = e^{\,\text{CE}}

CE

= 2.3

→ perplexity

e^{2.3} \approx 10

: as unsure as picking uniformly from 10 tokens

"Which answer is better?" Training from preferences

People compare two chatbot answers $A$ and $B$ and pick the better one. How do we learn a score $r$ from that?

↓ Bradley–Terry (1952): the probability of preferring $A$ depends on the score difference

p(A \succ B) = \sigma\big(r(A) - r(B)\big)

↓ maximum likelihood of the human choices, as today

L = -\ln \sigma\big(r(\text{chosen}) - r(\text{rejected})\big) = \ln\!\big(1 + e^{-m}\big)

Our logistic loss with margin $m = r(\text{chosen}) - r(\text{rejected})$ . Reward models for RLHF use exactly this; DPO puts the same $-\ln \sigma(\cdot)$ directly on the language model.

Ouyang et al., InstructGPT, 2022; Rafailov et al., Direct Preference Optimization, 2023

Can we trust the probabilities?

The "cut at 0.8" slide needs honest probabilities: of all students with $p = 0.8$ , about 80 % should pass. Is that true?

Logistic regression: usually close (it is trained with a proper scoring rule, like Bob the forecaster). Modern deep networks: often overconfident, like Alice.

↓ temperature scaling: divide all scores by one number $T$ , fitted by minimising cross-entropy on a validation set

q(k \mid \mathbf x) = \operatorname{softmax}(\mathbf a / T)_k, \qquad T > 1 \text{ softens}

One-parameter logistic regression on top of a network. For an SVM's scores $s$ : Platt scaling, $\sigma(\alpha s + \beta)$ : logistic regression again.

Guo et al., On calibration of modern neural networks, ICML 2017; Platt, Probabilistic outputs for SVMs, 1999

When most examples are easy: focal loss

An object detector scores ~100 000 boxes per image; almost all are background and easy. Cross-entropy sums them all: thousands of tiny losses can drown the few hard ones.

↓ multiply the loss by $(1 - q_t)^\gamma$ , small when the model is already right ( $q_t$ = probability of the true class)

\text{FL} = -(1 - q_t)^{\gamma} \ln q_t, \qquad \gamma = 2

easy,

q_t = 0.9

: CE 0.105 → FL 0.001 (100× less)

hard,

q_t = 0.1

: CE 2.30 → FL 1.87 (almost unchanged)

Lin et al., Focal loss for dense object detection (RetinaNet), ICCV 2017

Resources

Intuition

C. Olah, Visual Information Theory: surprise, entropy, cross-entropy, KL with pictures
3Blue1Brown, Gradient descent, how neural networks learn (video)

Stuck on the assignment

S. Prince, Computer Vision: Models, Learning, and Inference, ch. 9 (logistic regression, the bonus task's lifting), udlbook.github.io/cvbook
C. Bishop, Pattern Recognition and Machine Learning, sec. 4.3

Still used today (optional slides)

SigLIP (sigmoid loss for image–text) · calibration and temperature scaling · focal loss · DPO · AdamW · implicit bias of GD

Logistic regression

Warm-up: ML and MAP

From Bayes to logistic regression

Two classes: which one do we pick?

Have we seen this ratio before?

What if we learn a(x)a(x)a(x) directly?

From log-odds back to a probability

One formula for both classes

The sigmoid and its weights

The logistic sigmoid

What do w0w_0w0​ and w1w_1w1​ do?

Two features: what does the boundary look like?

Quick check

Which w\mathbf ww? The likelihood is the loss

How good is a given w\mathbf ww?

From the likelihood to a loss

What does each student cost?

Quick check

Try it: find a good w\mathbf ww by hand

Can we solve for the best w\mathbf ww directly?

How big should the step be?

Gradient descent in the assignment

Try it: gradient descent on the students

Why is it called cross-entropy?

Where have we seen −ln⁡p-\ln p−lnp before?

Cross-entropy: surprise when you believe the wrong thing

The usual case: two classes, hard labels

Every label is hard. Where does a soft ppp come from?

Which weather forecaster is better?

Quick check

Regularisation is a prior

What if the classes do not overlap?

MAP: the data plus what we believed before

Which prior for the weights?

Putting it together: the regularised loss

L2 or L1?

What it looks like on MNIST

Try it: the strength of the prior λ\lambdaλ

Quick check

How do we choose λ\lambdaλ?

Logistic regression in the big picture

Must we cut at p=0.5p = 0.5p=0.5?

Generative or discriminative?

How many numbers do we need to learn?

Can we read the weights?

More than two classes?

Logistic regression is one neuron

Only straight boundaries?

The assignment

The assignment in one slide

Logistic regression in five lines

Next

More on cross-entropy, and logistic regression today

Cross-entropy, entropy and KL divergence

Soft labels on purpose

Separable data: where does gradient descent go?

Logistic regression inside a vision–language model

How are big vision models evaluated?

Language models: the same loss, 100 000 classes

"Which answer is better?" Training from preferences

Can we trust the probabilities?

When most examples are easy: focal loss

Resources

What if we learn $a(x)$ directly?

What do $w_0$ and $w_1$ do?

Which $\mathbf w$ ? The likelihood is the loss

How good is a given $\mathbf w$ ?

Try it: find a good $\mathbf w$ by hand

Can we solve for the best $\mathbf w$ directly?

Where have we seen $-\ln p$ before?

Every label is hard. Where does a soft $p$ come from?

Try it: the strength of the prior $\lambda$

How do we choose $\lambda$ ?

Must we cut at $p = 0.5$ ?