↓ one flip

↓ the flips are independent: multiply, one factor per flip

↓ the same for any i.i.d. data $D = (x_1, \dots, x_N)$

As a function of $\theta$ (data fixed): the likelihood $\textcolor{orange}{L(\theta)}$.

↓ product rule on the coin likelihood itself, $f = \theta^7$, $g = 1 - \theta$

Zero at $\theta = \frac78$. A product of $N$ factors gives $N$ terms of $N$ factors each. With the log it is a sum of simple terms:

↓ chain rule: outer $\log u$, inner $u = 1 - \theta$, $u' = -1$

Several parameters ($\mu, \sigma^2$): partial derivatives, one at a time, the others held fixed. Treat $\sigma^2$ as one variable $v$: $\frac{d}{dv} \frac1v = -\frac{1}{v^2}$.

General coin: $N_H$ heads, $N_T$ tails, $N = N_H + N_T$. Where can the maximum be?

↓ edges: $L(0) = 0$ if $N_H \ge 1$, $L(1) = 0$ if $N_T \ge 1$; inside $L > 0$. So the maximum is inside $(0, 1)$, where the slope is 0. (H H H: $N_T = 0$, maximum at the edge $\theta = 1$.)

↓ take the log: products become sums, and log is increasing, so the argmax does not move

↓ derivative: $(\log\theta)' = \frac1\theta$, $(\log(1 - \theta))' = -\frac{1}{1 - \theta}$; set it to 0

↓ multiply by $\theta(1 - \theta)$

↓ divide by $N$

↓ Bayes' rule, as in lab 2 with $H = \theta$, $E = D$

↓ $p(D)$ does not depend on $\theta$: it does not move the argmax

↓ take the log

Same data (7 H, 1 T). Prior: $\textcolor{blue}{p(\theta)} \propto \theta^{a-1}(1-\theta)^{b-1}$ (a Beta$(a, b)$ density); here $a = b = 5$: "most coins are roughly fair".

↓ likelihood times prior: the exponents add up

↓ same shape as the likelihood, with $N_H + a - 1$ heads and $N_T + b - 1$ tails: same derivation as before

The prior acts like $a - 1 = 4$ extra heads and $b - 1 = 4$ extra tails.

↓ the expectation you know: $\mathbb E[f(z)] = \int p(z)\, f(z)\, dz$, with $z = \theta$ and $f(\theta) = p(x \mid \theta)$

↓ coin: $p(H \mid \theta) = \theta$, so we need the mean of the posterior

↓ the posterior is Beta$(N_H + a,\ N_T + b)$; the mean of Beta$(\alpha, \beta)$ is $\frac{\alpha}{\alpha + \beta}$ (derived on the next slide)

Posterior Beta$(\alpha, \beta)$, for our coin $\alpha = N_H + a = 12$, $\beta = N_T + b = 6$:

↓ definition of the mean; $\theta \cdot \theta^{\alpha - 1} = \theta^{\alpha}$

↓ the integral is the same normaliser, with $\alpha + 1$ instead of $\alpha$

↓ for whole numbers $B(\alpha, \beta) = \frac{(\alpha - 1)!\,(\beta - 1)!}{(\alpha + \beta - 1)!}$; most factorials cancel

Sketch $L(\theta)$ first: what happens for $\theta < 60$? For $\theta \ge 60$?

↓ if $\theta < 60$, tank 60 could not exist: one factor is 0

↓ otherwise every factor is $\frac1\theta$; $k = 4$ tanks, $m = \max_i x_i = 60$

↓ $\theta^{-k}$ decreases: take the smallest allowed $\theta$ (the edge case from the recap: no slope 0 anywhere)

The largest captured number can never exceed $\theta$: $\hat\theta_{\text{ML}} \le \theta$ always, so on average it is too small (biased).

↓ the $k$ numbers split $1..m$ into gaps of average size $\frac{m}{k} - 1$; assume one more gap above $m$

This estimator is unbiased (on average exactly $\theta$) and has the smallest variance among unbiased ones.

Prior: no idea, except "at most 1000 tanks": $\textcolor{blue}{p(\theta)} = \frac{1}{1000}$ for $\theta = 1, \dots, 1000$.

↓ Bayes' rule; a flat prior is a constant, so the posterior has the shape of the likelihood

$Z = \sum_{t = m}^{1000} t^{-k}$ makes it sum to 1.

↓ summarise the posterior by its middle (median) and its mean

And an honest range: $P(\theta \le 161 \mid D) = 0.95$. ML gives one number; Bayes also says how sure we are.

Plug in $\hat\theta_{\text{ML}} = 60$: "the next tank has a number from 1 to 60".

Is that believable after seeing only 4 tanks?

↓ Bayes: average the uniform $p(x \mid \theta)$ over the posterior, as for the coin

The ML plug-in is overconfident: it rules out every tank it has not seen. Same story as face 4 of the die and $\sigma^2_{\text{ML}}$ for 5 samples.

↓ log-likelihood, plus a Lagrange multiplier $\lambda$ times the constraint

↓ derivative with respect to one $p_k$: only one term of each sum depends on it

↓ derivative with respect to $\lambda$ gives back the constraint; plug in $p_k$

↓ put $\lambda = N$ back

↓ same shape as the likelihood, with $\alpha_k - 1$ instead of $N_k$ ("pseudo-counts")

↓ MAP: maximize likelihood × prior, same steps as ML (you derive it in the assignment)

↓ Bayes: the posterior is Dir$(N_k + \alpha_k)$; average over it

An estimator is a recipe: data in, number out. New data set, new number. Imagine repeating the experiment many times.

↓ average the estimate over all possible data sets $D$ of size $N$

Bias 0: unbiased, right on average.

$\hat\mu$ sits in the middle of the samples, closer to them than the true $\mu$, so the spread around it is too small. Divide by $N - 1$: unbiased.

What we care about: how far the estimate is from the truth, on average.

↓ expand the square around the average estimate $\mathbb E[\hat\theta]$; the cross term averages to 0

$\sigma^2$ from $N = 5$ Gaussian samples, $S = \sum_i (x_i - \hat\mu)^2$:

Conjugate prior: normal-inverse-gamma $\textcolor{blue}{\text{NIG}(\mu, \sigma^2 \mid \mu_0, \nu, \alpha, \beta)}$ (formula in the notebook). Read its parameters as imaginary data:

• $\mu$ was estimated from $\nu$ samples with mean $\mu_0$
• $\sigma^2$ from $2\alpha$ samples with sum of squared deviations $2\beta$
• peak of the prior: $\mu = \mu_0$, $\sigma^2 = \frac{\beta}{\alpha + 1.5}$

↓ MAP (you derive these, photo in the submission)