Univariate

Preliminaries

Consider the case of observing $N$ independent samples from a $\text{Normal}(\mu ,{\sigma }^2)$ distribution. Suppose we know the true value of ${\sigma }^2$, and we are interested in determining the posterior distribution of $\mu$. It is conventional to place a conjugate normal prior on $\mu$. Our model is:

$$\begin{array}{rcl}{y}_i& \stackrel{iid}{\sim }& \text{Normal}(\mu ,{\sigma }^2)\\ \mu & \sim & \text{Normal}({\mu }_0,{\tau }^2)\end{array}$$

In Bayesian statistics, the posterior is proportional to the likelihood times the prior. Because all observations are independent, our likelihood is the product of $N$ Normal density functions: one for each $y_i$. The prior then provides another Normal density function term. After simplifying and dropping terms that are not functions of $\mu$, we end up with a posterior distribution for $\mu$ proportional to

$$\exp \{-\frac{1}{2{\sigma }^2}\sum _{i=1}^N(y_i-\mu {)}^2\}\exp \{-\frac{1}{2{\tau }^2}(\mu -{\mu }_0{)}^2\}$$

The first term is the likelihood, and the second term is the prior. Because we desire a posterior that is a simple function of $\mu$, we need to gather all the terms that include $\mu$ together; as the posterior is written above, $\mu$ is scattered across $N+1$ terms. Completing the square is the trick that will allow us to gather all the $\mu$ terms into one.

The first step is to expand the squares containing $\mu$. This yields

$$\exp \{-\frac{1}{2{\sigma }^2}\sum _{i=1}^N(y_i^2-2\mu y_i-{\mu }^2)\}\exp \{-\frac{1}{2{\tau }^2}({\mu }^2-2\mu {\mu }_0+{\mu }_0^2)\}$$

We now distribute the sum, yielding

$$\exp \{-\frac{1}{2{\sigma }^2}(\sum _{i=1}^N{y}_i^2-2\mu \sum _{i=1}^N{y}_i+N{\mu }^2)\}\exp \{-\frac{1}{2{\tau }^2}({\mu }^2-2\mu {\mu }_0+{\mu }_0^2)\}$$

There are several terms above that do not involve $\mu$; these are highlighted in red. When we drop those terms, combine all remaining terms within one exponent, and then focus only on what is in the exponent, what remains is

$$-\frac{1}{2}(\frac{N}{{\sigma }^2}{\mu }^2+\frac{1}{{\tau }^2}{\mu }^2-2\mu \frac{\sum _{i=1}^N{y}_i}{{\sigma }^2}-2\mu \frac{1}{{\tau }^2}{\mu }_0)$$

We can combine the ${\mu }^2$ terms together, and the $\mu$ terms together:

$$-\frac{1}{2}((\frac{N}{{\sigma }^2}+\frac{1}{{\tau }^2}){\mu }^2-2(\frac{\sum _{i=1}^N{y}_i}{{\sigma }^2}+\frac{1}{{\tau }^2}{\mu }_0)\mu )$$

Finally, for ease of notation, we can use the fact that $\sum _{i=1}^N{y}_i=N\overline{y}$:

$$-\frac{1}{2}((\frac{N}{{\sigma }^2}+\frac{1}{{\tau }^2}){\mu }^2-2(\frac{N}{{\sigma }^2}\overline{y}+\frac{1}{{\tau }^2}{\mu }_0)\mu )$$

After all of this algebraic simplification, inside the parentheses what we have looks something like $a{x}^2-2bx$. We can now “complete the square” to obtain something of the form $(x-c{)}^2$.

Competing the square

Our first step is to make the notation easier to follow. Let $a=\frac{N}{{\sigma }^2}+\frac{1}{{\tau }^2}$ and $b=\frac{N}{{\sigma }^2}\overline{y}+\frac{1}{{\tau }^2}{\mu }_0$. Using the new, simplified notation, we have

$$-\frac{1}{2}(a{\mu }^2-2b\mu )$$

We can move the coefficient $a$ on $\mu$ outside the parentheses:

$$-\frac{a}{2}({\mu }^2-2\frac{b}{a}\mu )$$

We now add and subtract the same value inside the parentheses. This doesn’t change the value at all, since the terms sum to 0:

$$-\frac{a}{2}({\mu }^2-2\frac{b}{a}\mu +\frac{b^2}{a^2}-\frac{b^2}{a^2})$$

In fact, neither term is a function of $\mu$, so we can simply drop the term colored in red.

$$-\frac{a}{2}({\mu }^2-2\frac{b}{a}\mu +\frac{b^2}{a^2})$$

The terms within the parentheses are of the form $x^2-2xc+c^2$, which, from the rules learned in algebra, can be simplified to $(x-c{)}^2$. Applying this to our terms, we obtain:

$$-\frac{a}{2}{(\mu -\frac{b}{a})}^2$$

We have thus completed the square. We are not done, however: this was only the portion of the posterior distribution that was in the exponent. Replacing the terms in the exponent yields:

$$\exp \{-\frac{a}{2}{(\mu -\frac{b}{a})}^2\}$$

By completing the square, we have revealed that the posterior distribution of $\mu$ has the form of a normal distribution with a mean of $b/a$ and a variance of $1/a$, or

$$\mu \mid y\sim \text{Normal}({\mu }_n,{\sigma }_n^2)$$ where

$$\begin{array}{rcl}{\sigma }_n^2=\frac{1}{a}& =& {(\frac{N}{{\sigma }^2}+\frac{1}{{\tau }^2})}^{-1},\\ {\mu }_n=\frac{b}{a}& =& {\sigma }_n^2(\frac{N}{{\sigma }^2}\overline{y}+\frac{1}{{\tau }^2}{\mu }_0).\end{array}$$