Asymptotic Computations of MLEs¶
We apply the Fundamental Theorem of MLEs to a few computations
Exercise
The relative entropy of the empirical distribution and distributions on a bit.
Recall the classical coordinates on probability distributions on \(\{0,1\}\):
\[\begin{split}\begin{align*} \mathrm{Prob}(\{0,1\}) & \overset{\simeq} \longrightarrow \mathbb{R} \\ \rho &\longmapsto \rho(0) = p \end{align*}\end{split}\]
given data \(X \in \{0,1\}^{times N}\) of size \(N\), drawn from a distribtion \(\rho\) and a distribution \(\rho_\theta\):
\[\begin{split}\begin{align*} \mathcal{D}(\rho_X || \rho_\theta) &\approx \mathcal{D}(\rho || \rho_\theta) \\ &= - \bigl(\rho \log\rho_\theta + (1-\rho)\log(1 - \rho_\theta) \bigl) + \mathcal{S}(\rho) \end{align*}\end{split}\]
where the last term is the entropy of the empirical distribution, and does not depend on \(\rho_\theta\).
Example
We now compute the standard devation of \(\hat{\rho}_\theta\), using the computation above and the fundamental theorem. As we are in dimension 1, the hessian is just a second derivative:
\[\begin{split}\begin{align*} \sigma_\theta &= \partial^2_{p_\theta} \mathcal{D}(\rho || \rho_\theta) \lvert_{\rho_\theta = \rho} \\ &= \bigl( \frac{1}{p} + \frac{1}{1-p} \bigl)^{-1} \\ &= p(1-p) \end{align*}\end{split}\]
This recovers a result which is commonly justified using the central limit theorem. I prefer this derivation.
Exercise
The relative entropy of the empirical distribution and multinomial distribution
Example
distributions on a finite set compute the hessian find normal coordinates of the hessian say something interesting
Exercise
Recall the standard coordinates on the space of normal distributions on \(\mathbb{R}\) in terms of it’s first two cumulants: the expected value and squared standard deviation.
\[\begin{align*} \mathcal{D}(\rho_0 || \rho_1) &= \frac{1}{2} \Bigl( \log \bigl( \frac{}{} \bigl) \end{align*}\]
Here, we are working in units in which:
\[\mu_0 = 0, \sigma_0 = 1\]
Example
normal distribution with fixed standard distibution compute the hessian find normal coordinates of the hessian say something interesting
Exercise
The relative entropy of the empirical distribution and
Example
general normal distributions compute the hessian find normal coordinates of the hessian say something interesting
Exercise
The relative entropy of the empirical distribution and
Example
exponential distribution compute the hessian find normal coordinates of the hessian say something interesting
Exercise
The relative entropy of the empirical distribution and
Example
poisson distribution compute the hessian find normal coordinates of the hessian say something interesting
Note
these are spherical and hyperbolic metrics