Asymptotic Computations of MLEs
-------------------------------

We apply the Fundamental Theorem of MLEs to a few computations

.. admonition:: Exercise

   The relative entropy of the empirical distribution and  distributions on a
   bit.

   Recall the classical coordinates on probability distributions on
   :math:`\{0,1\}`:

      .. math::
         \begin{align*}
         \mathrm{Prob}(\{0,1\}) & \overset{\simeq} \longrightarrow \mathbb{R} \\
         \rho &\longmapsto \rho(0) = p
         \end{align*}

   given data :math:`X \in \{0,1\}^{times N}` of size :math:`N`, drawn from a
   distribtion :math:`\rho` and a distribution :math:`\rho_\theta`:

      .. math::
         \begin{align*}
         \mathcal{D}(\rho_X || \rho_\theta) &\approx 
         \mathcal{D}(\rho || \rho_\theta) \\
         &= - \bigl(\rho \log\rho_\theta + (1-\rho)\log(1 - \rho_\theta) \bigl)
         + \mathcal{S}(\rho)
         \end{align*}
         
   where the last term is the entropy of the empirical distribution, and 
   does not depend on :math:`\rho_\theta`.
   
.. admonition:: Example
   
   We now compute the standard devation of :math:`\hat{\rho}_\theta`, using
   the computation above and the fundamental theorem. As we are in dimension
   1, the hessian is just a second derivative:
   
      .. math::
         \begin{align*}
         \sigma_\theta &= 
         \partial^2_{p_\theta} \mathcal{D}(\rho || \rho_\theta)
         \lvert_{\rho_\theta = \rho} \\
         &= \bigl( \frac{1}{p} + \frac{1}{1-p} \bigl)^{-1} \\
         &= p(1-p)
         \end{align*}
         
   This recovers a result which is commonly justified using the central limit
   theorem. I prefer this derivation.

.. admonition:: Exercise

   The relative entropy of the empirical distribution and multinomial
   distribution
   
.. admonition:: Example
   
   distributions on a finite set
   compute the hessian
   find normal coordinates of the hessian
   say something interesting

.. admonition:: Exercise

   Recall the standard coordinates on the space of normal distributions
   on :math:`\mathbb{R}` in terms of it's first two cumulants: the 
   expected value and squared standard deviation.
   
      .. math::
          \begin{align*}
             \mathcal{D}(\rho_0 || \rho_1) &= \frac{1}{2} \Bigl(
             \log \bigl( \frac{}{}  \bigl)
          \end{align*}
      
   Here, we are working in units in which:
   
      .. math::
         \mu_0 = 0, \sigma_0 = 1

.. admonition:: Example

   normal distribution with fixed standard distibution
   compute the hessian
   find normal coordinates of the hessian
   say something interesting

.. admonition:: Exercise

   The relative entropy of the empirical distribution and  

.. admonition:: Example
   
   general normal distributions
   compute the hessian
   find normal coordinates of the hessian
   say something interesting

.. admonition:: Exercise

   The relative entropy of the empirical distribution and  

.. admonition:: Example
   
   exponential distribution
   compute the hessian
   find normal coordinates of the hessian
   say something interesting

.. admonition:: Exercise

   The relative entropy of the empirical distribution and  

.. admonition:: Example
   
   poisson distribution
   compute the hessian
   find normal coordinates of the hessian
   say something interesting

.. note::
   
   these are spherical and hyperbolic metrics