The bayesian setting §

As always in baysian statistics, one exploits bayes theorem.

The task is to find the probability of $P(x,y)$, given a number of (finite) observations.

The max likelihood method MLE §

This is a frequentist approach (it assumes flat prior). Estimation of a pdf based on a number of finite observations (data) of a random variable $X$.

$D=\{x_i{\}}_{i=1,\dots ,M}$ is our data, sampled from an unknown pdf $P$.

The likelihood of the data $D$ given a parameter $\lambda$ for our family of distribution is

$$P(D|\lambda )=\prod _{i=1}^M{P}_{\lambda }(x_i)$$

since the data is i.i.d. We now use the standard trick

$$=e^{{\sum }_{i=1}^M\log P_{\lambda }(x_i)}=:e^{\mathcal{L}(\lambda |D)}$$

ovviamente massimizzare la verosomiglianza equivale a massimizzare la funzione $\mathcal{L}$ appena introdotta. Ora riscaliamo di un fattore $\frac{1}{M}$ e aggiungiamo una costante, l’entropia empirica della vera distribuzione

$$L(\lambda |D)=-\frac{1}{M}\sum _{i=1}^M\log \frac{P(x_i)}{P_{\lambda }(x_i)}-H_M[P]$$

è apparsa proprio la divergenza KL empirica. In conclusione massimizzare la verosomiglianza è equivalente a minimizzare la Kullback-Leibler divergence empirica.

$$\underset{\lambda }{\max }L(\lambda |D)=\underset{\lambda }{\min }{D}_M(P\Vert P_{\lambda })$$

Bayesian Learning MAP §

Again, suppo we have a set of noisy data $D$, generated by an unknown source $S(x)=f_w(x)+\text{ noise}$.

Here we also need a prior distribution for the weights $P(w)$, this reflects prior knowledge/prejudice.

The goal is to find the posterior distribution of the weights $P(w|D)$, that is our new knowledge about the weights given data $D$.

The posterior is computed using Bayes theorem:

$$P(w|D)=\frac{P(D|w)P(w)}{P(D)}$$

but we don’t know the true data distribution (this is infact our goal), but since

$$P(D)=\int P(w\cap D)dw$$

using this we get

$$P(w|D)=\frac{P(D|w)P(w)}{\int P(D|w^{\prime })P(w^{\prime })d{w}^{\prime }}$$

Remark Given a family of pdf and a parameters, we can compute $P(D|w)$.

We can schematize the procedure in 3 steps:

1. Definitions define the parametrized model $f_w(x)$ and the prior distribution $P(w)$.
2. Model translation Convert the model in a standard probabilistic form, that is specify the liklihood of finding $y$ upon input $x$, given parameters $w$, i.e. $P(y|x,w)$.
3. Posterior distribution Compute the data likelihood $P(D|w)={\prod }_{i}P(y_i|x_i,w)$ as a function of $w$

Remarks

• No need for cross-validation
• Returns an error measure (a distribution of $w$)
• This gives an error on the estimate
• Traditional learning via gradient-descent is recoverd.

Example Let’s define a family of determinstic models $y(x)=\tanh (wx)$ with just one parameter $w\in \mathbb{R}$. Let’s choose a normal prior $w\sim N(0,1)$. We have only one data point $D=(1,-1/2)$.

Convert the model in probabilistic form:

$$P(y|x,w)=\delta (y-\tanh (wx))$$

compute posterior

$$P(w|D)=\frac{\delta (-1/2-\tanh (w))e^{-w^2/2}}{\int d{w}^{\prime }\delta (-1/2-\tanh (w^{\prime }))e^{-w^{\prime 2}/2}}$$

carring out the computations, one gets

$$=\delta (w-\tanh (-1/2))$$

so the posterior is a delta, and the value of $w$ that maximize it (the only possible choice) is $w=-\tanh (1/2)$.

Link with traditional learning §

In traditional statistical learning we minimize the empirical cost plus some regularization to ensure no overfitting

$$\frac{dw}{dt}=-\eta {\nabla }_w\left[E_M(w)+\lambda R(w)\right]$$

where the empirical cost is

$$E_M(w):=\frac{1}{M}\sum _{i=1}^{M}dist(y_i,f_w(x_i))$$

On the other hand in bayesian learning, the most probable weights are found minimazing

$$w_b=argmi{n}_w\ln P(w|D{)}^{-1}=argmi{n}_w\left[-\ln P(w)-\sum _{i=1}^M\ln P(y_i|x_i,w)+C\right]$$

where $C$ are just some constant terms. If we assume that our data is perturbed by a noise with distribution $p$ then

$$P(y|x,w)=p(y-f_w(x))$$

Let’s see what we are trying to minimize

$$\frac{1}{M}S(w,D)=-\frac{1}{M}\sum _{i=1}^M\ln p(y_i-f_w(x_i))-\frac{1}{M}\ln P(w)$$

we have added $1/M$ just to see clearly the relationship with classical learning. Then if we choose a noise distribution $p(z)=e^{-dist(y_{,}{f}_w(x))}$ and the weights prior as $P(w)=e^{-M\lambda R(w)}$ we are back in the setting of classical learning.

Remark this suggest to choose $\lambda \sim M^{-1}$.

Gradient descent per minimizzare D_KL §

Il primo algoritmo che viene in mente è il metodo del gradiente, fissiamo un learning rate $\eta >0$, ed aggiorniamo il vettore dei parametri $\lambda$ come

$$\lambda (t+\eta )=\lambda (t)-\eta {\nabla }_{\lambda }{D}_M(P\Vert P_{\lambda })$$

per $\eta$ abbastanza piccoli la divergenza non può crescere, infatti

$$\frac{d\lambda }{dt}=-{\nabla }_{\lambda }{D}_M(P\Vert P_{\lambda })$$

da questo segue

$$\frac{d}{dt}{D}_M(P\Vert P_{\lambda })={\nabla }_{\lambda }{D}_M(P\Vert P_{\lambda })\frac{d\lambda }{dt}=-({\nabla }_{\lambda }{D}_M(P\Vert P_{\lambda }){)}^2\le 0$$

Remark $P$ is unknown, but it doesn’t appear in the gradient of the KL divergence:

$${\nabla }_{\lambda }{D}_M(P\Vert P_{\lambda })=-\frac{1}{M}\sum _{i=1}^M\frac{{\nabla }_{\lambda }P(x_i)}{P_{\lambda }(x_i)}$$

The maximum entropy principle §

Idea: to find the unknown pdf, find $P$ such that it maximazies $H[P]$ given information (contrains) of mean values of some function $⟨f(x)⟩$.

$$P^{\ast }={\text{argmax}}_{P}H[P]\text{ subject to }⟨f_i(x)⟩=f_i)$$

to solve this problem we can use Lagrange multipliers.